Need help in Python for JSON data scraping

Question

I am using the below code for scraping data from a website. but I am facing key error: 0

Kindly tell me the problems in my code.

Original JSON response from the webpage: https://www.demo.com/api/user_details/22

Response:

{"user_details":{"user_id":"22","username":"Test","user_email":"[email protected]"}}

I wanna scrape the username, user_id and user_email.

What I have tried:

import json
import requests
import datetime

#data outputs to a CSV file in the current directory
csv_output = open("test.csv", "w")

end_page = 5;

#scan through pages 1 to end_page for data, 20 results per page
for page in range(1,end_page+1):
    r = requests.get('https://www.demo.com/api/user_details/' + str(page))
    data = r.json()
    for index in range(len(data["user_details"])):

        csv_output.write("\"%s\",%s\n" % (data["user_details"][index]["user_id"].encode('ascii', 'ignore'))), 

        data["user_details"][index]["user_id"]

csv_output.close()

AKS · Accepted Answer · 2017-03-01 06:12:10Z

1

data["user_details"] is a dict and not a list and you are getting the error because you are trying to access the values using an index:

data["user_details"][index] ....

You can get the entries by accessing specific keys from the dict:

user_id = data["user_details"]['user_id']
username = data["user_details"]['username']
user_email = data["user_details"]['user_email']

answered Mar 1, 2017 at 6:12

AKS

20k3 gold badges47 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

porthunt · Accepted Answer · 2017-03-01 06:13:36Z

0

Exactly what AKS asnwered, but I really recommend you to use a framework called Scrapy tocreate crawlers. Much easier. :)

answered Mar 1, 2017 at 6:13

porthunt

5042 gold badges5 silver badges13 bronze badges

1 Comment

porthunt Over a year ago

Now I'm using it more than ever. The way to go. :)

masnun · Accepted Answer · 2017-03-01 06:18:49Z

0

{"user_details":{"user_id":"22","username":"Test","user_email":"[email protected]"}}

User details is a dictionary here. On the other hand, index is an integer coming from the range call. The first value would be 0. Your code tries to load data["user_details"][0]. But there is no key 0 in that dictionary.

To iterate over a dictionary, you can call the items method which would give you a tuple with (key, value) pair.

d = {"user_id":"22","username":"Test","user_email":"[email protected]"}
for k,v in d.items():
    print("Key: {}, Value: {}".format(k,v))

edited Mar 1, 2017 at 6:18

answered Mar 1, 2017 at 6:13

masnun

12k4 gold badges44 silver badges53 bronze badges

1 Comment

Manoj Nath Over a year ago

My code is not looping through the request. it is just showing the data in the 'd' variable.

Collectives™ on Stack Overflow

Need help in Python for JSON data scraping

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related