Search/tweets is occasionally returning with bad content-length header

[lphilps]

Starting around 5:30pm Eastern Time yesterday (Nov 7, 2016) we started to see an uncharacteristically high number of returns from calls to the search/tweets endpoint that contained no data. The vast majority of calls to that endpoint are continuing to returning data as expected, but the errors persisted so I investigated.

It turns out that in every case in which we’ve seen a failure, the http headers returned by the call contain 2 “content-length” length headers. The first contains a very plausible numeric value for the size of the content, but the second is either:

content-length: macaw_search
or
content-length: api.twitter.com

This causes libcurl, which is what we’re using to fetch data, to think the size of the body is 0 and so we get no data.

Is anybody else seeing this?

Here’s a full set of headers included in a response we received earlier today (note the 2 consecutive content-length headers:

HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-disposition: attachment; filename=json.json
content-encoding: gzip
content-length: 48348
content-length: macaw_search
content-type: application/json;charset=utf-8
date: Tue, 08 Nov 2016 16:16:34 GMT
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Tue, 08 Nov 2016 16:16:34 GMT
pragma: no-cache
server: tsa_b
set-cookie: lang=en; Path=/
set-cookie: guest_id=v1%3A147862179430166399; Domain=.twitter.com; Path=/; Expires=Thu, 08-Nov-2018 16:16:34 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-access-level: read
x-connection-hash: eb39594668daf971a3e40dfe87b3f3e5
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-rate-limit-limit: 450
x-rate-limit-remaining: 365
x-rate-limit-reset: 1478622136
x-response-time: 272
x-transaction: 0040d4b100742a6c
x-twitter-response-tags: BouncerCompliant
x-xss-protection: 1; mode=block

[andypiper]

Sounds really weird. Anyone else able to reproduce?

[When2plus2is5]

I’ve been noticing similar problems, though I haven’t had the time to investigate. I’m now logging for non-numeric content-length values, and will report if I find anything.

[markunsworth]

Also seeing the same issue, but its not just affecting the content-length header. I have received “macaw_search, 92” in the x-rate-limit-remaining header. We’re also seeing random data in the x-rate-limit-reset and x-rate-limit-limit headers - (we don’t read any of the others).

Examples of data we have received (from my error logs) - all in different requests to the search API, there’s no obvious pattern, but it’s pretty frequent:

x-rate-limit-remaining: 'macaw_search, 92’
x-rate-limit-remaining: '51, api.twitter.com’
x-rate-limit-remaining: 'api.twitter.com’
x-rate-limit-reset: '1478663054, api.twitter.com’
x-rate-limit-reset: '008a65fe00d3124b, 1478650828’
x-rate-limit-reset: '1478658420, 1478658393’
x-rate-limit-reset: '1478660684, api.twitter.com’
x-rate-limit-limit: ‘009c664f00eef814’

[andypiper]

OK thanks - this gives me a little more to go on. I’ll do some digging internally, but I’m a little busy right now so this is not going to be a high priority. Apologies to those of you encountering errors in these areas.

[lphilps]

I’m still seeing this too. I will re-iterate that the vast majority of the API calls return with correct header, but I’m still consistently seeing problem responses frequently.

As @markunsworth pointed out above, I’ve now also detected errors in other headers. Here are some snippets of problem headers that came in overnight:

---

content-encoding: gzip
content-encoding: macaw_search
content-length: 67529
content-length: api.twitter.com

content-disposition: api.twitter.com
content-encoding: 00644bdb0068deb5
content-length: 43
content-length: 37
content-length: 46093
content-type: text/html; charset=ISO-8859-1

content-encoding: macaw_search
content-length: 59076
content-length: api.twitter.com

It is also really surprising that one of the header sets above included “charset=ISO-8859-1”, as my understanding is that the twitter api “always” returned utf-8 data?

I do get notified by my s/w whenever an unparseable response is returned from a Twitter API call and in the past that happened, maybe, once every couple of months or so - so infrequently that I never bothered to look into it. That changed abruptly at 5:30pm Eastern Time on Nov 7 and since then these errors have been occurring frequently.

I’ve been capturing the “full response - including headers” (like the one I pasted into the first post on this thread) for unparsable responses on a couple of my servers since about 10am ET yesterday (Nov 8). @andypiper if you want me to package these up and send them along, just ask.

[andypiper]

Thanks for the additional context! All good information. I can’t promise that we can address this immediately, but given information like date of change in behaviour etc, it may be possible to identify a deploy on our side that affected the headers.

[When2plus2is5]

I am also seeing it.

This is coming from the statuses/user_timeline endpoint:

HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-disposition: attachment; filename=json.json
content-encoding: gzip
content-length: 93211
content-length: timelines_api

[markunsworth]

The first time I see errors for this in our logs is November 3 00:34:30.670 UTC, the distribution of errors is very low up until the 8th November at 12:00 UTC at which point the increase is very noticeable. However it’s difficult for me to tell if this increase in errors was just down to an increase in the number of requests to the search API as we don’t store that information in a readily accessible format.

[When2plus2is5]

Here are some more.

content-encoding: gzip
content-length: 103654
content-length: timelines_api

---

content-encoding: gzip
content-length: 72681
content-length: timelines_api

---

content-encoding: 00e15ced0001a855
content-encoding: gzip
content-length: 200 OK
content-length: 64345

---

content-encoding: gzip
content-length: 87297
content-length: timelines_api

---

content-encoding: gzip
content-length: 1; mode=block
content-length: 105
content-length: 27150
content-length: 1444
content-type: text/javascript; charset=utf-8
content-type: text/html;charset=utf-8
…
set-cookie: _twitter_sess={redacted}; Path=/; Domain=.twitter.com; Secure; HTTPOnly
set-cookie: guest_id={redacted}; Domain=.twitter.com; Path=/; Expires=Sat, 10-Nov-2018 03:20:21 UTC

---

content-encoding: gzip
content-length: 200 OK
content-length: 35897
content-type: application/json;charset=utf-8
content-type: 1; mode=block
…
last-modified: Thu, 10 Nov 2016 03:57:28 GMT
last-modified: 1478750636
…
set-cookie: 1500
set-cookie: api.twitter.com
set-cookie: guest_id={redacted}; Domain=.twitter.com; Path=/; Expires=Sat, 10-Nov-2018 03:57:28 UTC

---

content-encoding: gzip
content-encoding: api.twitter.com
content-length: 38401
content-length: timelines_api

[johnspurlock]

Just saw this myself for the first time on 2016-11-09T19:42:31.888Z. Response to a search api request. First time I remember seeing “macaw_search”. Also note the weird content-encoding and unexpected location.

"headers": {
  "content-encoding": "api.twitter.com",
  "content-length": "229",
  "date": "Wed, 09 Nov 2016 19:42:31 GMT",
  "last-modified": "Wed, 09 Nov 2016 19:42:31 GMT",
  "location": "https://platform.twitter.com/jot.html",
  "server": "tsa_b",
  "status": "macaw_search",
  "strict-transport-security": "max-age=631138519",
  "x-connection-hash": "3b3ba3fbdbdf2852aff80afa7a5451e1",
  "x-rate-limit-limit": "450"
}

[andypiper]

OK thanks for all these reports. I’m trying to track down the correct folks to help me look into it!

[When2plus2is5]

I’m trying to limit what I show to different examples, and not just repeat examples of essentially the same issue.

Here’s an interesting one.

cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0 <-- Not Included!
content-length: 31
content-length: 00f01cf000530739
content-length: 43212
content-type: text/html; charset=ISO-8859-1
content-type: application/json;charset=utf-8
…
x-adtype: clear
…
x-failurl: https://ads.mopub.com/m/ad?v=8&udid=ifa:00000000-0000-0000-0000-000000000000&id=47bf0f3adf094486a5fc61abda26cf84&nv=4.5.1&q=segments:tmaze,segments:ijeyw,z_sdkversion:4.7.2,z_impression_id:00066783-3255-4844-89FD-3AF5CAAD08BE,z_slot_name:MOB_WWF2_I12,z_nexage:false,z_min_version:4.1.5&o=p&sc=3.0&z=-0500&mr=1&dnt=1&ct=3&av=4.152&cn=T-Mobile&iso=us&mnc=260&mcc=310&dn=iPhone8%2C2&w=1242&h=2208&bundle=com.newtoyinc.NewWordsWithFriendsFree&request_id=7e6a7428a076483da592bbc55cb41c37&fail=1&fail=1&fail=1&fail=1&fail=1&fail=1&fail=1&fail=1&fail=1&fail=1&fail=1&exclude=000b04c5229d448986caf0455eeeb1e8&exclude=0d5c0c8e03444b919cf503873786f4b5&exclude=200a99ffd0924f16959d39fc8e95e197&exclude=234578434a6f433c9e5932a81a940c63&exclude=2e2409d0db214da6beae0fa47f33fb02&exclude=409c4e48a1e0469091d27cb3e5abf820&exclude=67d373833ed740318140d7344a94ae0d&exclude=abe261d31d22443f8d060424f09e7a8f&exclude=c4ce4d16ac614056a1529753017c7da8&exclude=cf70e23d61234c12abd0317d10eff18e&exclude=d9fb983bfa9c4a26915216042290ce90&exclude=e24046af6416470b8192150d59f50049&fail=1
x-orientation: p
…
x-rate-limit-limit: api.twitter.com
…
x-refreshtime: 60
…
x-xss-protection: timelines_api

I’ve never seen x_failurl before, and have no other examples of many of the other headers in this example. I have 3 recorded instances of x-refreshtime, but the other 2 are 30.

[andypiper]

Curious where this one is coming from since it includes a mopub URL? What API endpoint is this?

[When2plus2is5]

All of the snippets I’ve been providing are coming from statuses/user_timeline

[lphilps]

I’ve also seen a mopub url in a header. This response was returned from a call to search/tweets and in addition to the multiple, and bogus, content-length value(s) also includes a mopub url in an x-imptracker header:

HTTP/1.1 200 OK
content-disposition: attachment; filename=json.json
content-encoding: gzip
content-length: 297
content-length: 48
content-type: application/json;charset=utf-8
date: Wed, 09 Nov 2016 00:59:27 GMT
last-modified: Wed, 09 Nov 2016 00:59:27 GMT
server: tsa_b
set-cookie: lang=en; Path=/
set-cookie: guest_id=v1%3A147865316707649398; Domain=.twitter.com; Path=/; Expires=Fri, 09-Nov-2018 00:59:27 UTC
status: 200 OK
strict-transport-security: max-age=631138519
x-access-level: read
x-connection-hash: 38f55d862d190b61eecf58e87012bf2d
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-fulladtype: millennial_full
x-imptracker: http://ads.mopub.com/m/imp?appid=&cid=6c48c6ddcf7b4d9f81eddf73578b4a26&city=Kingston&ckv=2&country_code=JM&cppck=CA21B&dev=SAMSUNG-SM-G900A&exclude_adgroups=99aeee72a9b64692916bfc78988515bf%2Ce93cd4a42301407280b36cff2e6b0a67&id=a9f8d1629b8111e281c11231392559e4&is_mraid=0&os=Android&osv=5.0.0&req=45411efff7d844678483d6c38d8a8adf&reqt=1478653166.0&rev=0.000200&udid=ifa%3Adbc39fef-a9c9-439f-a46b-b2e94f9a82de&video_type=
x-nativeparams: {“adHeight”:480,“adUnitID”:“133415”,“adWidth”:320}
x-orientation: p
x-rate-limit-limit: 450
x-rate-limit-remaining: 270
x-rate-limit-reset: 1478653325
x-response-time: 39
x-scrollable: 0012fb9e001702c9
x-xss-protection: 1; mode=block

[lukeasrodgers]

We are also experiencing the same issues above. One additional datapoint I haven’t seen mentioned yet: in some responses, the x-rate-limit-remaining and x-rate-limit-reset headers are present (though they contain junk data), but the x-rate-limit-limit header is totally absent.

[andypiper]

Thank you all for these data points. Working on a couple of other tasks but we have this on our radar.

[lphilps]

Well, the good news is that I haven’t detected an API response with a bad/duplicate content length header since 22:37UTC on Friday. Has anybody else detected any problems since that time?

[When2plus2is5]

I was just about to report the same thing.