Getting too much 502 after migrating to Twitter API 1.1

eksigundem · 2012-09-18T21:06:52.000Z

When I changed my codes from Twitter REST API 1.0 to 1.1,
I started to get too much 502 errors when I make requests.
For example:
GET - 1.1/users/lookup

Nearly one third of my requests gets a 502 Error.
Do I miss something? Is there anything that I should consider?

Thank you in advance.

episod · 2012-09-18T21:13:00.000Z

Can you provide some more details?

How many user IDs or screen names are you looking up at a time?
What’s the rate at which you’re issuing requests?
How many requests have you attempted in the measured time?
How many of those returned 502s?
Are your requests divided between access tokens or issued from the same one?

In regards to comparing to version 1 of the same API:

Are you comparing with the same set of IDs?
Are you using authentication? If so, are your requests divided between access tokens or issued from the same one?

Thanks!

eksigundem · 2012-09-18T21:41:46.000Z

Q:How many user IDs or screen names are you looking up at a time?
A: It depends to the user’s popularity. I lookup for minimum 10, maximum 300 ids. I lookup them by dividing into chunks of 100 IDs at a time. When I lookup for 220 user’s data, I request 3 times.
A: When I make a 3 time request I get 1 times HTTP 502 and 2 times HTTP 200. Not all of them requests turns 502.

Q:What’s the rate at which you’re issuing requests?
Q:How many requests have you attempted in the measured time?
Q:How many of those returned 502s?
A:I couldn’t make a detailed statistics yet. I had strange data for some days, after 1.0 to 1.1 change but I couldn’t understand. I just understood the cause is users lookup. I cannot say a number yet.

Q:Are your requests divided between access tokens or issued from the same one?
A:I use each time the user’s own tokens to lookup. So every different user use different tokens. But tokens are same between the 100-sized chuncks.

Q:Are you comparing with the same set of IDs?
A:IDs are different but the only different thing inside my code is changing “1/users/lookup” to “1.1/users/lookup”

Q:Are you using authentication? If so, are your requests divided between access tokens or issued from the same one?
A:I use authentication because 1.1 dictates to have access tokens for every request.

eksigundem · 2012-09-18T21:47:00.000Z

I just saw this: "You are strongly encouraged to use a POST for larger requests."
Inside this: https://dev.twitter.com/docs/api/1.1/get/users/lookup

Should I use POST in this case?

episod · 2012-09-18T21:53:56.000Z

Thanks for all the info.

I would definitely use POST in your use case. I would also recommend lowering the amount you ask for in each request, especially in the cases where you’re getting a 502 and asking for 100 at a time – you can get much more predictable performance that way usually.

Meanwhile, I’ll investigate with the engineers on why this method theoretically may be throwing more 502s than in v1’s implementation.

eksigundem · 2012-09-18T21:57:36.000Z

Thank you @episod for help.

eksigundem · 2012-09-19T10:35:54.000Z

Say it I have 300 IDs.
I split it to three sets of A,B and C.

Which one is better solution??

Fetch data for A, if it gives a 502 error, then re-request same set, if it gives HTTP 200 code then change cursor to fetch B, and then C.
Split 300 IDs into 6 sets and fetch data for each 50 users?

It seems like seconds solution is more time consuming. More time will be passed to finish task, but in first solution it will be finished earlier. But it seems like I need to be cautious to not to enter to infinite loop, if my application gets 502 error for set A, over and over.