403 Forbidden through API for every follow

[ruzdic]

Twitter says we are hitting supposed “rate limiting” but this is not the same rate limiting described in Twitter’s documentation. According to the documentation, a “429 Too Many Requests” is the response Twitter’s API is supposed to be returning when a rate limit is reached. This is not the HTTP response our engineers are receiving, rather, they are receiving a “403 Forbidden”, which contains no rate limit headers indicating that we have hit a rate limit. This would indicate that we are not hitting a “rate limit” as detailed in Twitter’s rate limiting documentation.According to Twitter’s documentation, the API call for friendships/create is not rate limited. (https://dev.twitter.com/docs/api/1.1/post/friendships/create). We are more than aware of which API calls are rate limited, and which aren’t. We also have measures in place to stop users from breaching these limits (on rate limited requests), and when these limits are reached, we internally block the user or account manager from even attempting to make further requests.

Twitter Operation was no help.

[andypiper]

First of all, apologies for the inconvenience and confusion here. Let’s see if I can help.

According to Twitter’s documentation, the API call for friendships/create is not rate limited. (https://dev.twitter.com/docs/api/1.1/post/friendships/create)

Actually the POST friendships/create endpoint is rate-limited. You can also use the application rate limit status endpoint for help here.

You’re referencing follow in the title of this post. Can you be clear about which endpoint(s) you are trying to call, and how often?

[ruzdic]

Hi,

Thanks for the reply.

To answer your specific question, the call we are making is POST friendships/create.

Now to clear up some confusion: Twitter has just updated the documentation for that method (friendships/create) declaring that it is in fact Rate limited. This however was not the case a week or so ago (before the documentation site had the facelift - which is very nice by the way). Previously (and we are only talking weeks), the friendships/create documentation specified that there was no rate limiting (it literally had “Rate Limited : No” in the documentation for that call). So, the quote I gave you no longer applies, due to Twitter changing this in the last week or so. Might I add, that I have been chasing this issue up with support for a few weeks, and during the inception of the conversation, the documentation had not been updated. I even have a reply from Twitter support staff stating that friendships/create is not rate limited, which confirms that this was previously the case, and the documentation only changed recently.

Regardless, let’s put that aside for a moment, as it isn’t relevant now. Let’s focus on what is true now.

You have now informed us that the friendships/create endpoint is rate limited. We have no problem with that. The problem we have is, the list of Rate limited requests supplied here: https://dev.twitter.com/rest/public/rate-limits does not define a rate limit for friendships/create. Nor does making a call to https://dev.twitter.com/rest/reference/get/application/rate_limit_status - as per your suggestion - supply a limit for friendships/create. I tested it, and this is what was returned for the friendships resource family:

“friendships”: {
"/friendships/outgoing": {
“limit”: 15,
“remaining”: 15,
“reset”: 1410441483
},
"/friendships/no_retweets/ids": {
“limit”: 15,
“remaining”: 15,
“reset”: 1410441483
},
"/friendships/lookup": {
“limit”: 15,
“remaining”: 15,
“reset”: 1410441483
},
"/friendships/incoming": {
“limit”: 15,
“remaining”: 15,
“reset”: 1410441483
},
"/friendships/show": {
“limit”: 180,
“remaining”: 180,
“reset”: 1410441483
}
}

As you can see rather clearly, friendships/create is not specified in the API call you suggested. And nor is it specified in the list of Rate limited endpoints as described earlier.

We have tried our best to be responsible in our provisioning of API access to our users. And for all the endpoints that have specified limits, we have implemented limiting on our users so they cannot breach these limits. Importantly, we can do this because the limits are explicitly documented by Twitter. There is no grey area. If a user reaches their limit, we cut them off. However, for friendships/create, we can’t get a straight answer from anyone as to what the limit is.

To make matters worse, when our users’ made lots of calls (lots meaning 250 per day at absolute most) to friendships/create when we went live, we did not receive the standard 429 declaring that we had hit any formal rate limit. Instead, we received a 403 with no headers telling us anything (let’s call this ‘pseudo rate limiting’ for the sake of clarity from here on). And, once we got ‘pseudo rate limited’, all of our users where rate limited, including new users that signed up while we were in this ‘pseudo rate limited’ state, even though these new users had never made a request through our application. Further evidence that this was not standard rate limiting, was that the rate limiting was removed after around 24 hours, rather than after the standard 15 minute or 180 minute windows.

FYI, I am not calling it ‘pseudo rate limiting’ to be facetious, rather, I am calling it that because it does not fit the profile of a proper 429 type of rate limit response which Twitter documents quite thoroughly. It’s just a 403, with a JSON blob that had something along the lines of ‘You cannot follow any users at this time’ (this is not verbatim).

So, the questions that need to be answered are:

1. Given the documentation now says the friendships/create method is rate limited, and these limits are not documented anywhere, how do we proceed to rectify this and stop our application from being ‘pseudo rate limited’ (or even stop our application from being rate limited at all)? If you would kindly tell us what these formal limits are, or where we can find them, we would easily be in a position to limit our users from breaching any limits.

2. Can you tell us how one or more users breaching this nebulous limit would cause other users to also be ‘pseudo rate limited’, given the API request to POST friendships/create is made under the authentication context of a User, and not the Application alone, meaning only the naughty users would be limited, rather than every user being rate limited, even ones that have just signed up and haven’t made a request.

I apologize for the lengthy response, but your reply was the first ‘non-canned response’ we have received from Twitter, and we are really hoping you can explain to us what is going on, and how we can fix it. As a social media agency, we have invested substantial amounts of capital into this project, and we can’t afford to just turn it off due to some possible misunderstandings, most likely on our part.

Regards, Emir

[andypiper]

Thanks for the question! We’re looking into the questions you’ve asked here, and will let you know when we’ve got more information to share.

[andypiper]

Hey Emir.

So I’m going to start with two apologies - first of all, apologies that this hasn’t been made clear enough, this is something we have taken on board and will look at ways to clarify on the Developer documentation site; secondly, apologies for my misleading statements about using the application/rate_limit_status endpoint when you were trying to use a POST / write API, which is not a valid use case (per below).

There are two ways in which rate limits apply - to application actions, and to user actions. There’s a section in the rate limiting docs headed “GET and POST Request Limits” which implies the difference, but this hasn’t made it clear enough for users and developers like yourself.

• GET / read API endpoints are rate limited at an application action level. These are currently explained in the pages you are familiar with, https://dev.twitter.com/rest/public/rate-limits and https://dev.twitter.com/rest/public/rate-limiting

• POST / create endpoints are feature limited on a per-user basis. For instance, a user can post 2400 tweets per day, broken down into hourly intervals. These are documented on the Support pages. Since these limits vary and may be more fluid for operational reasons, there is no equivalent list of POST endpoints with their rate limits listed out.

The HTTP status codes for these calls vary between 403 and 429, as you have found.

In the specific case of what you are wanting to call, the relevant limit is 1,000 follows per day, but there are some ratios that come into play based on how many followers a user has, so again, this is not something that is explicitly listed and you should take care to follow “reasonable” approaches within your app.

I hope that this helps you to understand the way this works, and your questions have already caused us to re-think the way this information is presented in the future, so thank you very much for your patience, attention, and curiosity!

[ruzdic]

Andy,

No need to apologise, I appreciate your help in this. We have taken your comments on board and have made some modifications to the application and asked about 20 people (friends, colleagues and customers) to test the application. We made sure the only following and unfollowing activity which they were doing was to come from our application and not to use any other application. We also asked them not to do too many follows and to limit their follow requests to less than 250 per day.
Despite all this, our application is still getting blocked with 403 errors. After the application was blocked we did a few tests. Firstly we asked a 5 of our users to use Twitter Website and to continue preforming the same actions as they were on our application. These users have reported that no errors were ever received. We then asked 5 other users to logon and start using Managed Flitter (another application very similar to ours) and preform the same actions they were doing on our app and they reported that no errors were received.
As it stands today our app is suspended and I am baffled as to why our application would be suspended. We have followed every documented limit to the letter.