I'm trying to learn how to scrape tweets using python. I'm trying to use the following code, but I keep getting the error. I'm not sure how to rectify it.
def fetch_tweets(query, count = 50):
api = connect() # Gets the tweepy API object
tweets = [] # Empty list that stores all the tweets
try:
fetched_data = api.search_tweets(q = query + ' -filter:retweets', count = count)
for tweet in fetched_data:
txt = tweet.text
clean_txt = cleanText(txt) # Cleans the tweet
stem_txt = TextBlob(stem(clean_txt)) # Stems the tweet
sent = sentiment(stem_txt) # Gets the sentiment from the tweet
tweets.append((txt, clean_txt, sent))
return tweets
except tweepy.TweepyException as e:
print("Error: "+ str(e))
exit(1)
tweets = fetch_tweets(query = 'Birdman', count = 200)
# Converting the list into a pandas Dataframe
df = pd.DataFrame(tweets, columns= ['tweets', 'clean_tweets','sentiment'])
# Dropping the duplicate values just in case there are some tweets that are copied and then stores the data in a csv file
df = df.drop_duplicates(subset='clean_tweets')
df.to_csv('data.csv', index= False)
ptweets = df[df['sentiment'] == 'positive']
p_perc = 100 * len(ptweets)/len(tweets)
ntweets = df[df['sentiment'] == 'negative']
n_perc = 100 * len(ntweets)/len(tweets)
print(f'Positive tweets {p_perc} %')
print(f'Neutral tweets {100 - p_perc - n_perc} %')
print(f'Negative tweets {n_perc} %')
I keep getting the following error
TypeError: The `text` argument passed to `__init__(text)` must be a string, not <class 'NoneType'>
And this is the stem(text) function, which is where the problem seems to occur:
def stem(text):
porter = PorterStemmer()
token_words = word_tokenize(text)
stem_sentence = []
for word in token_words:
stem_sentence.append(porter.stem(word))
return " ".join(stem_sentence)
I saw this response in a different place but since i'm new to coding, I wasn't sure how to use it?
df['data'].apply(lambda x: sentiment(' '.join(x)))
TextBlob(stem(clean_txt)). The value ofstem(clean_txt)isNone, so you're callingTextBlob(None). It can be the result ofclean_textorstem. You haven't pasted theclean_txtfunction. To have a good answer here you also need to paste the value offetched_data. Please read "How to create a Minimal, Reproducible Example".print()to see what you have in variables. And if you sometimes getsNonethen you should useif not stem(clean_txt): continueto skip rest offor-loop when you getNone