Changes to the follow stream

[TweetsFromAaron]

Hello,

We are using the follow parameter as a part of the stream API (https://dev.twitter.com/streaming/overview/request-parameters#follow).

At 9:45 EST both our staging and production machines (running different keys) stopped ingesting. It appears that we can ingest items fine if we drop the number of accounts we follow (from 1948 to 1657). However neither the code or the data has not changed in these environments.

Do you know if anything changed with the Stream API?

Thanks,
Aaron.

[andypiper]

I’m not aware of a change. The API should enable the ability to follow up to 5000 user IDs, but this may be restricted by the length of the URL parameters. Can you confirm whether you are using a POST or a GET request on the filter endpoint?

Both GET and POST requests are supported, but GET requests with too many parameters may cause the request to be rejected for excessive URL length. Use a POST request to avoid long URLs.

[TweetsFromAaron]

Hey @andypiper we’re definitely using POST requests via the twitter gem https://github.com/sferik/twitter.

Digging through the gem, the request is passed in the format https://stream.twitter.com:443/1.1/statuses/filter.json?follow=82916196,594257142,121077487,356952302,580269538,191332758,76497791,143505435,28743774,275145990,1074073514,156713922,19567338,928634394,377007897,789808844,23642374,115403866,21424966,16252118,388916221,2258440815,21119272,18560607,96815634,80163916,24109979,164076105,105017346,21777465,22502646,583215014,1155205856,87044439,281669945,761559356777398272,53103297,35583547,50778210,27315969,1012558242,374645472,14550149,256593427,53285540,28753082,41116918,120019718,28397181,36973175,46677036,194148680,339096771,3003688642,2699057586,227126529,157072539,4832916069,25324851,448114771,336136128,18734310,46968377,26762221,155798084,327017143,45977746,8824902,69180124,20737672,188781567,71344809,19911952,121509578,26746279,1419177625,538416452,28573278,325846450,73542155,364460082,21120759,188027699,23955472,50684862,371292415,3305543211,3148111567,53167708,38968665,14401521,22073642,369723148,64378435,40358743,38140624,1112091415,799359972,844823689,15707738,72716600,282607945,41156772,61771891,47713256,53197137,....

[andypiper]

That’s weird - I just tried this using twurl (which uses the twitter gem as its core) and it does not seem to append URL parameters to the endpoint value - it sends it as POST body data.

$ twurl -H stream.twitter.com -A "Accept-encoding: none" -t -d "follow=54931584"  -X POST "/1.1/statuses/filter.json"

opening connection to stream.twitter.com:443...
opened
starting SSL for stream.twitter.com:443...
SSL established
<- "POST /1.1/statuses/filter.json HTTP/1.1\r\nAccept-Encoding: none\r\nAccept: */*\r\nUser-Agent: OAuth gem v0.4.7\r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization: OAuth oauth_consumer_key=\"xxxxx\", oauth_nonce=\"xxxxx\", oauth_signature=\"xxxxx%2FUIxo%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1501510767\", oauth_token=\"xxxxx\", oauth_version=\"1.0\"\r\nConnection: close\r\nHost: stream.twitter.com\r\nContent-Length:15\r\n\r\n"
<- "follow=54931584"
-> "HTTP/1.1 200 OK\r\n"

[TweetsFromAaron]

Hey @andypiper, looking at the Gemspec and Gemfile for the twurl (https://github.com/twitter/twurl/blob/master/twurl.gemspec and https://github.com/twitter/twurl/blob/master/Gemfile), I don’t see any mention of the twitter gem. Maybe it is no longer a dependency?

Based on what you are saying though, the args should be passed as the POST body data rather than as a part of the URL. Is that the way we should be doing it? We try than instead if that is the preferred way of doing things.

Also, to be clear, we hadn’t changed anything on our end with this. This just stopped working on Friday.

Thanks

[andypiper]

Hmm - ok that may be my misunderstanding - I know @sferik had a hand in both so I may have conflated that information, to be fair! Sorry!

For long queries the POST body is definitely preferred.

I hear ya on the unexpected change - I’m really not sure why things would have caused an error like this, so I can only apologise and admit my lack of knowledge here! Fingers crossed that a change to POST body might help on this.

[KatyaObyedkova]

Hi Aaron,
I ran into the same problem last Friday. I use sferik’s Twitter gem which definitely uses Post requests. I have around 3200 ids and now the code works for only up to 1623 ids. What is even more strange is that in May I had a similar situation. I had to reduce the number of ids from 3400 to around 3200. It was a quick fix at the time so I’ve not found out what the real problem was. Now I really need to find a solution. The limit is 5000 so there has to be a way to make this work. If you find a solution I would really appreciate if you could let me know and I will do the same. Thanks, Katya

[TweetsFromAaron]

Hi @KatyaObyedkova, our workaround for now is to fork the process so that we can have two streams running with 1000 ids on separate threads. I was looking into sferik's gem and I’m hoping to change it. However, I am having trouble getting the underlying header library simple_oauth to be able to generate a successful signature for the post. simple_oauth also hasn’t been touched for a few years and I was wondering if it is even maintained.