Retweets with comments starting with mentions are missing from user streams

[dekisu]

(edit: changed thread title to reflect the actual cause for this issue, see later posts for details)

Hi! I have this very rare and hard to reproduce bug with user streams in which some (but not all) of the recently introduced “retweets with comment” aka “quoted status” just don’t seem to show up in the user’s stream.

This was first reported to our bug tracker three weeks ago and blamed it on a temporary issue with the server, but it’s still happening.

These tweets don’t happen very often, most of them just show up just fine, and it’s hard to notice that something is missing when most tweets are mostly independent pieces of information.

While it might be a problem with my client (which does deduplication of retweets), I’m 99% sure this isn’t the case.

Example tweet I found today affected by this - nothing special about it, and everything looks fine from the REST api.

Web client (most recent tweet first), third tweet is the missing one

My client (most recent tweet last)

The missing tweet should appear right before 23:38 <_0xabad1dea> [de] (the [de] being the two char hex ID that my client uses to refer to those tweets, also helps confirm there’s no client side filtering), timestamp should be 23:36

[dekisu]

I believe i’ve found a pattern for this. It’s when quoting tweets that start with a mention.

Definitely a problem with the streaming api, and they do show up in the REST api. Also not a problem with my client - i used the “t” gem to test it this, it has the same problem.

From my test account I retweeted-with-comment’d the following thing adding “FUNNY TWEET” as a comment.

Justin Bieber (justinbieber)

Then waited for the tweet to appear in the stream, right after “beep boop” below.

[dx@bolsa ~]$ t stream timeline  -c
ID,Posted at,Screen name,Text
[...]
597094942775844864,2015-05-09 17:45:02 +0000,everyunicode,羪
59709583534xxxxxxx,2015-05-09 17:48:35 +0000,[unrelated user],[unrelated tweet]
597097534566326273,2015-05-09 17:55:20 +0000,[test account name here],can anyone hear this
597097644541005825,2015-05-09 17:55:46 +0000,[test account name here],beep boop
59709808391xxxxxxx,2015-05-09 17:57:31 +0000,[unrelated user],[unrelated tweet]
^C%

Then ran the same command again to make it fetch recent tweets from the REST api, and there it is.

[dx@bolsa ~]$ t stream timeline  -c
ID,Posted at,Screen name,Text
[...]
597094942775844864,2015-05-09 17:45:02 +0000,everyunicode,羪
59709583534xxxxxxx,2015-05-09 17:48:35 +0000,[unrelated user],[unrelated tweet]
597097534566326273,2015-05-09 17:55:20 +0000,[test account name here],can anyone hear this
597097644541005825,2015-05-09 17:55:46 +0000,[test account name here],beep boop
597098027946516480,2015-05-09 17:57:18 +0000,[test account name here],FUNNY TWEET https://t.co/RDkgcHyWfe
59709808391xxxxxxx,2015-05-09 17:57:31 +0000,[unrelated user],[unrelated tweet]

Other quoted tweets which don’t start with mentions don’t have this problem.

[dekisu]

Did some more tests, now with just curl + signatures for the userstream generated with OAuth Tool.

And, uh, either it’s fixed, or my theory that it’s related to the fact that they started with mentions is wrong. Now I kinda regret not testing more than two times. I repeated it 12 times, all of them showed up, both in the curl stream and my client

I think I’m just going to keep looking for more of those like i did before (comparing stream results with twitter web contents) for a few more days until I get bored of wasting time with this.