How to iterate through JSON objects (results of twitter api search) in Python and save to a CSV file?

Question

AIM: need to find out how to parse data from api search below into a CSV file. The search returns results in the following format:

[(u'Bertille Maciag', 10), (u'Peter Prior', 5), (u'Chris OverPar Duguid', 4), (u
'Selby Dhliwayo', 4), (u'FakeBitch!', 4), (u'Django Unchianed UK', 4), (u'Padrai
g Lynch ', 4), (u'Jessica Gunn', 4), (u'harvey.', 4), (u'Wowphotography', 3)]

I'm a newbie to python and any help would be greatly appreciated

import twitter, json, operator
#Construct Twitter API object
searchApi = twitter.Twitter(domain="search.twitter.com")
#Get trends
query = "#snow"
tweeters = dict()

for i in range(1,16):
    response = searchApi.search(q=query, rpp=100, page=i)
    tweets = response['results']
    for item in tweets:
        tweet = json.loads(json.dumps(item))
        user = tweet['from_user_name']
        #print user
        if user in tweeters:
            # print "list already contains", user
            tweeters[user] += 1
        else:
           tweeters[user] = 1

sorted_tweeters = sorted(tweeters.iteritems(), key=operator.itemgetter(1),     reverse=True)  

print len(tweeters)
print tweeters

print sorted_tweeters[0:10]
print 'Done!'

Why do you dump and load the item object instead of just loading? — jdi
– jdi, Commented Jan 20, 2013 at 18:34

tacaswell · Accepted Answer · 2013-01-20 21:11:24Z

2

It looks like you have all the hard bits working, and are just missing the 'save to csv' part.

import collections
import twitter, json, operator
#Construct Twitter API object
searchApi = twitter.Twitter(domain="search.twitter.com")
#Get trends
query = "#snow"
tweeters = collections.defaultdict(lambda: 0)

for i in range(1,16):
    response = searchApi.search(q=query, rpp=100, page=i)
    tweets = response['results']
    for item in tweets:
        user = item['from_user_name']
        #print user
        tweeters[user] += 1

sorted_tweeters = sorted(tweeters.iteritems(), key=operator.itemgetter(1), reverse=True)  

str_fmt = u'%s\u200E, %d \n'
with open('test_so.csv','w') as f_out:
    for twiters in sorted_tweeters:
        f_out.write((str_fmt % twiters).encode('utf8'))

You need the 'u' on the format string and encode because you have non-ascii charachters in the user names. u'\200E is the ltr marker so that the csv file will look right with rtl language user names.

I also cleaned up the iteration code a bit, by using a defaultdict you don't need check if a key exists, if it does not exist, the generator function is called and it's value is returned (in this case 0).

item is already a dict, there is no need to convert it to a json string and then back to a dict

edited Jan 20, 2013 at 21:11

answered Jan 20, 2013 at 18:32

tacaswell

88.2k23 gold badges222 silver badges203 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

stewiestudent Over a year ago

Hi tcaswell,thanks for your fast reply.placed the code between tweeters = sorted(tweeters.iteriitems().....and...print len(tweeters). Received the following error message C:\Users\lenovo>python C:\Users\lenovo\workspace\test3\ass3_working.py Traceback (most recent call last): from .api import Twitter, TwitterError, TwitterHTTPError, TwitterResponse File "C:\Python27\lib\site-packages\twitter-1.9.0-py2.7.egg\twitter\api.py", l ine 20, in <module> import json File "C:\Users\lenovo\workspace\test3\json.py", line 12, in <module> AttributeError: 'module' object has no attribute 'JSONEncoder'

tacaswell Over a year ago

@masterkilla Fixed a typo tweeters -> sorted_tweeters I really have no idea what code you actually ran and can make no sense of the error messages.

pipe-devnull · Accepted Answer · 2013-01-20 18:29:31Z

1

Have you looked at the python CSV module? using your output:

import csv, os

x = [(u'Bertille Maciag', 10), (u'Peter Prior', 5), (u'Chris OverPar Duguid', 4), (u'Selby Dhliwayo', 4), (u'FakeBitch!', 4), (u'Django Unchianed UK', 4), (u'Padraig Lynch ', 4), (u'Jessica Gunn', 4), (u'harvey.', 4), (u'Wowphotography', 3)]

f = open("/tmp/yourfile.csv", "w")
writer = csv.writer(f, quoting=csv.QUOTE_ALL)
for i in x:
    writer.writerow(i)

f.close()

answered Jan 20, 2013 at 18:29

pipe-devnull

3422 silver badges5 bronze badges

Collectives™ on Stack Overflow

How to iterate through JSON objects (results of twitter api search) in Python and save to a CSV file?

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related