I’m working on a small portfolio project in Node.js trying to use the filtered stream endpoint https://api.twitter.com/2/tweets/search/stream to pull a minimal number (5) of tweets per hour about “2020”. All the script does is stream the tweets and then close the connection. However, I continue to get the below error on occasion and I am a bit at a loss as to why.
function readStream(stream) {
return new Promise((resolve, reject) => {
let saved = [];
stream
.on('data', function (data) {
console.log('Debug | readStream data: ', data); // Error Message outputted here
const json = JSON.parse(data);
saved.push(json.data);
if (saved.length > 4) {
stream.destroy();
}
})
.on('close', () => {
resolve(saved);
})
.on('error', (error) => reject(error))
.on('timeout', (error) => reject(error));
});
}
{
title: 'ConnectionException',
detail: 'This stream is currently at the maximum allowed connection limit.',
connection_issue: 'TooManyConnections',
type: 'https://api.twitter.com/2/problems/streaming-connection'
}
The first thing I did was investigate and confirm that I was well below the rate limits specified for the V2 Filtered Stream API. After researching the error it occurred to me that perhaps this might be an external bug of some type and didn’t want to keep searching my code if that was the case. Does that “too many connections” apply to Twitter’s connection capacity? Or am I still doing something wrong trying to connect? The problem is very intermittent which is why it’s been so hard to track down and solve.
pull a minimal number (5) of tweets per hour about “2020”
The streaming API is designed for long running (permanently open) connections, so connecting to stream a few tweets and disconnecting as not a good use of this - try using Recent search quick start | Docs | Twitter Developer Platform instead.
You’re probably getting the too many connections errors because you’re connecting and reconnecting too frequently, or the previous connection hasn’t disconnected yet and you’re trying to connect again.
If you want to use the stream, it’s best to open it, start streaming tweets forever, and only reconnect when there is an error, saving tweets somewhere where you can sample whatever smaller number you need per hour or whatever later.
2 Likes
The streaming API is designed for long running (permanently open) connections, so connecting to stream a few tweets and disconnecting as not a good use of this…
Yeah this is a suggestion I came across after I started to investigate the problem. If I would have known before hand I wouldn’t have structured the database this way. Alas - comically - by the time I figured it out in a matter of a day I had burned through my entire monthly camp. I did manage to save down a small dev database that I can continue to develop the project from but I will rewrite the piece later to get a more significant sample size.
2 Likes
Yeah that monthly cap isn’t ideal - very easy to burn through. I hope they change it when the endpoints come out of “preview”.
