Missing header x-rate-limit-remaining in response while request is performed sucessfully

[abexloss]

Dear Twitter developers,

We are using REST API v1.1 for retrieving information from various feeds:

• search/tweets
• statuses/user_timeline
• lists/statuses

Lately, we quite often receive REST API responses without the header x-rate-limit-remaining . This is a problem for us as we are using this value to define the caching strategy of the response received: caching TTL increases as value of x-rate-limit-remaining decreases. But without value x-rate-limit-remaining, we can’t really adapt the caching strategy.

Could you please explain why the x-rate-limit-remaining is sometimes missing, and what can be done to force it or make sure it’s always present ?

Please find some examples of REST API requests / responses:

• Search Tweet

Request:

[method] => GET
[query] => Array
    (
        [count] => 12
        [q] => #NNanotech #EU_KETs #EU_NANO #EU_MATERIALS #EU_ #PRODUCTION #H2020_EU #NMP_EU #FoF_EU #EeB_EU #SPIRE_EU #cPPP_EU +exclude:retweets
        [result_type] => mixed
        [count_initial] => 3
    )
[path] => /1.1/search/tweets.json

Response:

[0] => HTTP/1.0 200 Connection established
[1] => HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-length: 596
content-type: application/json;charset=utf-8
date: Tue, 13 Jan 2015 06:59:48 UTC
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Tue, 13 Jan 2015 06:59:48 GMT
pragma: no-cache
server: tsa_b
set-cookie: guest_id=v1%3A142113238848598898; Domain=.twitter.com; Path=/; Expires=Thu, 12-Jan-2017 06:59:48 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-access-level: read
x-connection-hash: 8c2551e68a3ddeb6ee37ea544a836dd5
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 64
x-transaction: 8a72976bbd79a5e1
x-twitter-response-tags: BouncerCompliant
x-xss-protection: 1; mode=block
[2] => {"statuses":[],...

• Statuses User Timeline

Request:

[method] => GET
[query] => Array
(
    [count] => 15
    [screen_name] => EU_Justice
    [exclude_replies] =>
    [include_rts] => 1
    [trim_user] =>
    [count_initial] => 15
)
[path] => /1.1/statuses/user_timeline.json

Response:

[0] => HTTP/1.0 200 Connection established
[1] => HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-length: 62304
content-type: application/json;charset=utf-8
date: Tue, 13 Jan 2015 06:59:58 UTC
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Tue, 13 Jan 2015 06:59:58 GMT
pragma: no-cache
server: tsa_b
set-cookie: lang=en
set-cookie: guest_id=v1%3A142113239869937350; Domain=.twitter.com; Path=/; Expires=Thu, 12-Jan-2017 06:59:58 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-access-level: read
x-connection-hash: 9732bb3e768bc26e49b4b2b907bc7eef
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 83
x-transaction: 4833f9a19740ff52
x-twitter-response-tags: BouncerCompliant
x-xss-protection: 1; mode=block
[2] => [{"created_at":"Mon Jan...

• List Statuses

Request:

[method] => GET
[query] => Array
    (
        [count] => 20
        [owner_screen_name] => EU_Commission
        [slug] => ec-spokespersons
        [include_rts] =>
        [include_entities] => 1
        [count_initial] => 5
    )
[path] => /1.1/lists/statuses.json

Response:

[0] => HTTP/1.0 200 Connection established
[1] => HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-length: 25278
content-type: application/json;charset=utf-8
date: Tue, 13 Jan 2015 07:08:52 UTC
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Tue, 13 Jan 2015 07:08:52 GMT
pragma: no-cache
server: tsa_b
set-cookie: lang=en
set-cookie: guest_id=v1%3A142113293205894832; Domain=.twitter.com; Path=/; Expires=Thu, 12-Jan-2017 07:08:52 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-access-level: read
x-connection-hash: dfd498df0387b3dad4dd7a340893a43d
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 76
x-transaction: 713852434e946048
x-twitter-response-tags: BouncerCompliant
x-xss-protection: 1; mode=block
[2] => [{"created_at":"Tue Jan...

The following topic x_ratelimit_remaining is not appearing in OAuth response header has been checked but unfortunately does not provide an answer.

Thanks a lot and best regards,

Alexandre

[matevzg]

This is happening to us too.

As it seems all three headers are missing, x-rate-limit-remaining, x-rate-limit-total and x-rate-limit-reset. This is happening sporadically, and sometimes goes away in an hour, sometimes sticks for 12 hours.

Any one else seeing this situation?

[matevzg]

Oh, as per [1], API docs say:

Lastly, there may be times in which the rate limit values that we return are inconsistent, or cases where no headers are returned at all. Perhaps memcache has been reset, or one memcache was busy so the system spoke to a different instance: the values may be inconsistent now and again. We will make a best effort to maintain consistency, but we will err toward giving an application extra calls if there is an inconsistency.

This has been happening quite often now, and the periods keep getting longer. To mitigate this situation we might be changing our server side rate limiting strategy in a way where we will just keep max x-rate-limit-remaining value, when there is no response headers present.

[1] https://dev.twitter.com/rest/public/rate-limiting

[abexloss]

Thanks a lot matevzg for this information and the link https://dev.twitter.com/rest/public/rate-limiting, this is really useful. We read the documentation you mentioned but missed it as we really focused on § HTTP Headers and Response Codes.

Well, in that case, it looks that Twitter is aware of this fact and have no other immediate plan than maintaining consistency on best effort. Good to know, so on our side we’ll have to refine the caching strategy when no x-rate-limit-remaining.

Anyway, thanks a lot for highlighting the statement from the documentation.

Best regards,

Alexandre