Does twitter streaming stops if no tweet for longer duration

python
tweepy

#1

I have a python script that streams tweets into a csv file. I have provided the runtime parameter of 48600 seconds which is equal to 13 hrs, this dictates for how long the tweets have to be streamed into that csv. It was running fine for the specified duration until yesterday, but today the script ran for 7.5 hrs only and it stopped streaming afterwards. I believe, there was no tweet about the topic i was streaming for a certain duration and hence the disconnect happened. So even if, people started tweeting about the topic in question at later point, the script didn’t stream those tweets to csv file. Hence, i had to restart the script in another instance and let the script stream the tweets into another csv file.

But i am not sure if that was the case. Below is the script that i used, please advise what could have happened.

runtime = 18000

class listener(StreamListener):

def on_data(self,data):
    data1 = json.loads(data)
    time = data1["created_at"]
    tweet1 = BeautifulSoup(tweet, "lxml").get_text() 
    url = "https://twitter.com/{}/status/{}".format(data1["user"]["screen_name"], data1["id_str"])
    file = open('MARCH_DATA.csv', 'a')
    csv_writer = csv.writer(file)
    csv_writer.writerow([time, tweet1, url])
    file.close()
           
auth = OAuthHandler(consumer_key,consumer_secret) 
auth.set_access_token(access_token,access_token_secret)
twitterStream = Stream(auth, listener())
twitterStream.filter(track=["MTA"], async = True)
time.sleep(runtime)
twitterStream.disconnect()

Thanks


#2

Hi @nakul_3184 - yes, it is possible: if the rate of Tweets being read off of the stream drops suddenly, the connection will be closed. Other possible reasons include server-side errors, excessive client-side lag and network hiccups.

More information on disconnects is available in our documentation, under the Disconnections section: https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/connecting The Stalls section on that page also outlines some strategies for keep alive.


#3

Hey @Aurelia

I am running this script over a web platform, on an internet server. I really doubt that there could be a server/network hiccup issue or the read-rate of the stream has fallen. Also, as we speak, i was running my script which was streaming tweets successfully until half an hour ago and it got disconnected. Could you please advise, what change to make in my script so that the connection is kept alive and this disconnected is avoided?

Thanks and Regards
Nakul


#5

Do the strategies on the page that is listed in the post above yours not help you to make changes? Check out the advice in the Stalls and Reconnecting sections. You’ll need to implement some error handling to check for disconnections, and then reconnect somehow.