0

I'm trying to learn how to scrape tweets using python. I'm trying to use the following code, but I keep getting the error. I'm not sure how to rectify it.

def fetch_tweets(query, count = 50):
    api = connect() # Gets the tweepy API object
    tweets = [] # Empty list that stores all the tweets
    try:
        fetched_data = api.search_tweets(q = query + ' -filter:retweets', count = count)
        for tweet in fetched_data:
            txt = tweet.text
            clean_txt = cleanText(txt) # Cleans the tweet
            stem_txt = TextBlob(stem(clean_txt)) # Stems the tweet
            sent = sentiment(stem_txt) # Gets the sentiment from the tweet
            tweets.append((txt, clean_txt, sent))
        return tweets
    except tweepy.TweepyException as e:
        print("Error: "+ str(e))
        exit(1)
tweets = fetch_tweets(query = 'Birdman', count = 200)
# Converting the list into a pandas Dataframe
df = pd.DataFrame(tweets, columns= ['tweets', 'clean_tweets','sentiment'])

# Dropping the duplicate values just in case there are some tweets that are copied and then stores the data in a csv file
df = df.drop_duplicates(subset='clean_tweets')
df.to_csv('data.csv', index= False)

ptweets = df[df['sentiment'] == 'positive']
p_perc = 100 * len(ptweets)/len(tweets)
ntweets = df[df['sentiment'] == 'negative']
n_perc = 100 * len(ntweets)/len(tweets)
print(f'Positive tweets {p_perc} %')
print(f'Neutral tweets {100 - p_perc - n_perc} %')
print(f'Negative tweets {n_perc} %')

I keep getting the following error

TypeError: The `text` argument passed to `__init__(text)` must be a string, not <class 'NoneType'>

And this is the stem(text) function, which is where the problem seems to occur:

def stem(text):
    porter = PorterStemmer()
    token_words = word_tokenize(text)
    stem_sentence = []
    for word in token_words:
        stem_sentence.append(porter.stem(word))
        return " ".join(stem_sentence)

I saw this response in a different place but since i'm new to coding, I wasn't sure how to use it?

df['data'].apply(lambda x: sentiment(' '.join(x)))
2
  • Is this your code or did you get it somewhere else? Do not paste just the last line of the error, but the full traceback. It seems the issue happens when caling TextBlob(stem(clean_txt)). The value of stem(clean_txt) is None, so you're calling TextBlob(None). It can be the result of clean_text or stem. You haven't pasted the clean_txt function. To have a good answer here you also need to paste the value of fetched_data. Please read "How to create a Minimal, Reproducible Example". Commented Apr 4, 2022 at 1:45
  • first you could use print() to see what you have in variables. And if you sometimes gets None then you should use if not stem(clean_txt): continue to skip rest of for-loop when you get None Commented Apr 4, 2022 at 7:01

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.