Read JSON file in python: ValueError

Question

I have a .txt file with 70+k json object obtained by pulling data from twitter and dumping into file using:

with open("followers.txt", 'a') as f:
     for follower in limit_handled(tweepy.Cursor(api.followers, screen_name=account_name).pages()):
         for user_obj in follower:
             json.dump(user_obj._json, f)  
             f.write("\n")

When I try to read this in python using the code below:

import json
with open('followers.txt') as json_data:
     follower_data = json.load(json_data)

I get error:

ValueError: Extra data: line 2 column 1 - line 2801 column 1 (char 1489 - 8679498)

It worked when I read a test file with one json object copied from the original file using the same code above. Once I add a second json object to this file then using the same code above gives the error:

ValueError: Extra data: line 2 column 1 - line 2 column 2376 (char 1489 - 3864)

How do I read the file with more than one json object?

You're not writing it correctly, so you cannot read it correctly. — cs95
– cs95, Commented Jul 19, 2017 at 6:17

cs95 · Accepted Answer · 2017-07-19 06:19:31Z

2

The issue comes when you write your JSON. You must write a single JSON object, so you may also load a single JSON object. Currently, you are writing multiple separate objects, causing the error.

Modify your write code a bit:

json_data = []
with open("followers.txt", 'a') as f:
     for follower in limit_handled(tweepy.Cursor(api.followers, screen_name=account_name).pages()):
         for user_obj in follower:
             json_data.append(user_obj._json)             

     # outside the loops
     json.dump(json_data, f)

Now, when reading, your existing code should work. You'll get a list of dictionaries.

answered Jul 19, 2017 at 6:19

cs95

406k106 gold badges745 silver badges798 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

T-Jay Over a year ago

Thanks for pointing this out @cᴏʟᴅsᴘᴇᴇᴅ. I have updated my write code.

cs95 Over a year ago

@T-Jay if you used my answer then you should accept mine man ;p Just kidding.

Shai · Accepted Answer · 2017-07-19 06:24:00Z

1

Of course it is best to treat the problem from the root: write a single json and read it, as suggested by COLDSPEED.
However, if you already wrote multiple json objects into a single file, you can try the following code to use the already created file:

import json
follower_data = []  # a list of all objects
with open('followers.txt') as json_data:
  for line in json_data:
    follower_data.append( json.loads(line) )

Assuming you did not indent your json objects when you wrote them to 'flowers.txt', then each line in the file is a json object that can be parsed independantly.

answered Jul 19, 2017 at 6:24

Shai

115k39 gold badges259 silver badges398 bronze badges

1 Comment

T-Jay Over a year ago

thanks, this works perfectly to load data already written in multiple json objects.

Collectives™ on Stack Overflow

Read JSON file in python: ValueError

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related