Looping a PYTHON script

Question

I need some help with Python loop techniques. After a few days of searching, I give up...

System: Windows ( Anaconda )

Idea: "I created an HTML parser script, but due to missing knowledge and experience with Python scripts, it appears that I need to run it on every page. I can't fix it, that's why I decided to loop this script and make it run 100 times for 100 pages"....but as a result, I can't find the right way to do so...

My script

import requests
import pandas as pd
import urllib.parse
import urllib.request
import re
import os
import sys



 url = "*******************/store/index.php"

 querystring ={"id":"***","act":"search","***":"***","country":"",
 "state":"*","city":"","zip":"","type":"","base":"","PAGENUM":"2"}

 headers = {
 'Host': "www.*****",
 'Connection': "keep-alive",
 'Upgrade-Insecure-Requests': "1",
 'User-Agent': "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 
 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36",'Accept':"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
'Referer': "h************/store/index.php?id=********************&pagenum=2",
'Accept-Encoding': "gzip, deflate",
'Accept-Language': "en-US,en;q=0.9",
'Cookie': "php_session_id_real=**********; cookname=**********; cook******",
'cache-control': "no-cache",
'Postman-Token': "**************************"
}

 response = requests.request("GET", url, headers=headers,params=querystring)
 df_list = pd.read_html(response.text)
 df = df_list[-1]

 print(df)

All I need to change is PAGENUM querystring ( ex: &pagenum=2,3,10,50,etc ... )

Is it possible to run this python script X times, and each time change the value of pagenum = pagenum + 1 ??

Hope for your advice!

Cheers

Have you searched "for loops in python"? Should only take a little research to figure out how to do this. — user2263572
– user2263572, Commented Nov 30, 2018 at 15:11

seralouk · Accepted Answer · 2018-11-30 17:05:19Z

1

Use a for and iterate to a list containing all the desired values. Next, use str to store the value in the dictionary.

Do this:

import requests
import pandas as pd
import urllib.parse
import urllib.request
import re
import os
import sys

pagenums=[2,3,10,50]
#or pagenums = np.range(1,101)

for page in pagenums:
    querystring ={"id":"***","act":"search","***":"***","country":"",
                  "state":"*","city":"","zip":"","type":"","base":"","PAGENUM":str(page)}
    #......
    #..... # more code here

    #headers = {....}

For each iteration, the value of PAGENUM key, will updated.

edited Nov 30, 2018 at 17:05

answered Nov 30, 2018 at 15:13

seralouk

33.6k10 gold badges127 silver badges141 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Shubham Sharma · Accepted Answer · 2018-11-30 15:17:58Z

You need to apply for loop which runs 100 times and takes all your pages. I hope below code works fine.

import requests
import pandas as pd
import urllib.parse
import urllib.request
import re
import os
import sys
import numpy as np


 url = "*******************/store/index.php"
 pagenums = np.arange(0,100)
 for i in pagenums:
     querystring ={"id":"***","act":"search","***":"***","country":"",
     "state":"*","city":"","zip":"","type":"","base":"","PAGENUM":str(i)}

     headers = {
     'Host': "www.*****",
     'Connection': "keep-alive",
     'Upgrade-Insecure-Requests': "1",
     'User-Agent': "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 
     (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36",'Accept':"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    'Referer': "h************/store/index.php?id=********************&pagenum=2",
    'Accept-Encoding': "gzip, deflate",
    'Accept-Language': "en-US,en;q=0.9",
    'Cookie': "php_session_id_real=**********; cookname=**********; cook******",
    'cache-control': "no-cache",
    'Postman-Token': "**************************"
    }

     response = requests.request("GET", url, headers=headers,params=querystring)
     df_list = pd.read_html(response.text)
     df = df_list[-1]

        enter code here

     print(df)

Collectives™ on Stack Overflow

Looping a PYTHON script

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related