HTTP GET In Python

Question

I have a bash script that downloads a lot of files from a URL and saves them to my Computer like:

curl 'http://website.com/shop.php?store_id=[66-900]&page=[7-444]&PHPSESSID=phpsessid' -O

(and a bunch of headers set, the typical request you get when you copy a request as curl in firefox)

Tried to rewrite the script in python but turns out it is very hard (complete newbie) like:

import requests

cookies = {
    'PHPSESSID': '<phpsessid here>',
}

headers = {
    '<bunch of header stuff here>'
}

params = (
    ('store_id', '[66-900]'),
    ('page', '[7-444]'),
    ('PHPSESSID', '<phpsessid here>'),
)

response = requests.get('http://website.com/shop.php', headers=headers, params=params, cookies=cookies)

But it just doesn't do anything (curl command downloading everything as commanded) I realised I may need to write my downloaded stuff to a file but am completly lost since I am not handling one file but hundreds of files and as seen from the curl command above.

Completly lost what I am missing or need to do to replicate a curl request like that in python, ANY help appreciated!

requests may get something but it will not save it in file automatically. You should check what you have in response.content and write code which write it in file. — furas
– furas, Commented Nov 7, 2021 at 16:32
it seems you should write it as for-loop and use single value in store_id instead of [66-900] and the same with page — furas
– furas, Commented Nov 7, 2021 at 16:34

balrundev · Accepted Answer · 2021-11-07 16:51:58Z

1

You can't use a range as you can do it when using curl.

As @furas noted, you can use a loop, because curl sends requests with all parameter combinations from the ranges.

Function response.get returns an object of class requests.models.Response, so you should use json() method or properties content (returns bytes) and text (returns string).

answered Nov 7, 2021 at 16:51

balrundev

3811 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

furas · Accepted Answer · 2021-11-07 17:02:33Z

1

You have to use for-loops instead of [66-900], [7-444]

for store_id in range(66, 990+1):
    for page in range(7, 444+1):

        params = {
           'store_id': store_id,
           'page': page,
           'PHPSESSID': '<phpsessid here>',
        }

        response = requests.get(...)

        with open(f"{store_id}-{page}", "wb") as fh:
             fh.write(response.content)

edited Nov 7, 2021 at 17:02

answered Nov 7, 2021 at 16:38

furas

149k12 gold badges121 silver badges171 bronze badges

Collectives™ on Stack Overflow

HTTP GET In Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related