Sitestream missing many DMs across multiple users

[Scott_Sparky]

Many of our clients have reported DMs missing in the middle of conversations. DMs before and after the missing ones come in just fine. Tweets are coming in through Sitestream without any issue, so the scope of this report is entirely related to Direct Messaging. Our logs show everything else entering our system, but it is as if these DMs were never sent out via Sitestream in the first place. There are no special characters or anything out of the ordinary in the missing messages, either.

This issue was first discovered on July 29, by one of our own employees on their personal account. Shortly thereafter, another employee noticed a DM in the middle of a DM conversation that never arrived, whereas it showed up in native Twitter without issue. Today, July 31st, we have had reports about this issue affecting multiple separate Twitter accounts.

Our questions are:

• Are there any rate limits within Sitestream for DM?
  We’ve been told no in other places, but I would like to confirm

• What information would you need from us to assist in investigation?
  We have some examples of the full conversations in native, with the missing messages highlighted

• Have any other users reported this issue with Sitestream during this week?
  We’ve been unable to locate any other open bug tickets related to this issue, per bug reporting etiquette

Again, we will be monitoring this bug report for communication, and greatly appreciate any response. We are quite certain that this is occurring with more customers than have actually reported it to us. We will be able to provide any non-sensitive information needed to assist, so please let us help you help us. Thank you in advance!

[pi_kumar]

@Scott_Sparky can you tell us some of the times at which these events occurred and also the time zones in which they happened? It will help us to investigate it on our end.

[Scott_Sparky]

Thank you for your response!

Unfortunately, as the missing messages did not arrive in our system, we only have the timestamps for the DMs that did arrive. Viewing the missing DMs in native Twitter does not reveal timestamps, but we have one where the user was able to locate the timestamp for a missing DM in Tweetdeck.

For this affected conversation we have the timestamp, the following timestamps are for messages that came through on Sitestream with no issue. All timestamps are EST (Eastern time). The messages have sensitive information, though:

Jul 30 - 9:00am
Jul 31 - 2:25am
Jul 31 - 8:50am

Here is the timestamp for the message that did not arrive. The message is not sensitive, so I
have included it:

Jul 31 - 11:40am - It’s not going to work the phone i have has a different frequency

Another affected conversation, I have gathered the timestamps of multiple DMs that came through in quick succession from a customer, with missing ones in between. These timestamps are from our logs and are UTC timezone, and I have included the message from one that is not sensitive:

Jul 30 - missing DM
Jul 30 - 3:46:49 am - received
Jul 30 - missing DM
Jul 30 - 3:47:22 am - received - "of possible fruad."
Jul 30 - missing DM
Jul 30 - 3:48:27 am - received
Jul 30 - missing DM
Jul 30 - 3:50:22 am - received

Is it possible to initiate a private conversation for this issue, so I can discuss more specific information that I am able to provide? I appreciate your cooperation in this matter.

[Scott_Sparky]

Just to let everyone else who may be reading this report, the issue has been resolved by the Sitestream team, and is no longer occurring. Huge thank you to pi_kumar for the assistance!

[rowanseymour]

Hi. We may be having the same issue with UserStreams. Our app (http://app.rapidpro.io) uses user streams to fetch incoming DMs from users. A couple of days ago a user reported that we were missing some DMs they had sent. I’ve been able to replicate this pretty consistently by sending 10 DMs from my own account and our app will typically only get 8-9 of them.

We’re using twitterhbc/twitter4j and not seeing anything in our logs when a DM goes missing. You can see our UserStream code here.

Curiously, on a different instance of the app with a different Twitter API key - the messages com through.

Please advise on what information we can provide to help track the source of this problem.

[Scott_Sparky]

Hey Twitter,

We are again experiencing this issue, albeit more severely. This time, we have Tweets going missing as well. Here are a couple examples of Sitestream missing Tweets from just this morning:

TO - FROM - TIMESTAMP (PST) - MESSAGE
@asklyft @mcclure_sf 10:27am
@asklyft @mcclure_sf 10:42am “Better now, thank you. On hold now.”
@tmobilehelp @hullowkitty 10:50am “@TMobileHelp (test hi guys!)”

We are currently working on getting timestamps for missing DMs, which as explained earlier in the original ticket, are a bit more difficult to determine. If you need to contact us via DM or private message, we are more than happy to work with you. We understand that Piyush is unavailable for the next two weeks, and are fine with another agent. Pi mentioned that there was an internal ticket related to this issue, if that is any assistance to you.

Thank you in advance, and we look forward to your reply.

[happycamper]

Thanks for the details here. We are looking into this for you.

[Scott_Sparky]

Thank you very much for your prompt response.

We are monitoring this thread for responses, so if you require any further information to assist in your investigation, please respond here or send a DM and we will gather what you need!

[happycamper]

Hi Scott, could you let me know if you were able to pull these these missing DM’s through the REST API to retrieve? If so, are these greater than 140 characters?

[Scott_Sparky]

Hey HC,

These are entirely through Sitestream (push), not the REST API (pull). The missing messages simply aren’t being pushed by Sitestream. Also, this is not just related to DM messages anymore, as @mentions are also included.

Are you asking us to attempt to pull messages through REST as a troubleshooting step?

Many of the messages, including the ones in all of our examples in this thread, are way less than 140 characters.

[happycamper]

Hi Scott,

Thanks for clarifying and bringing this to attention. Yes, using REST API would be a data retrieval option if needed. We found and resolved a timing inconsistency which appears to have fixed this issue. Let us know if you have additional questions or occurrences.

[enewcomer]

We are still seeing this problem (I work with @rowanseymour above).

Here’s a specific case of a message that never arrived:

From: JaimeArchM
To: UreportMexico
Message: "No"
Sent: 12:03pm (GMT -5)

[karen_colpitts]

We are also still seeing this issue

[BasJS]

Same here

[enewcomer]

Is anybody else still seeing this issue? We are still seeing randomly missing DMs from our UserStream. Here’s an example DM message that never showed up in the stream for @textitin recently:

To: @textitin
From: @enewcomer
Message: 'Industry’
Date: Tuesday, December 15, 2015 @ 7:30am Pacific

Any idea what’s up?

[coletg]

@enewcomer We’re seeing the same issue again as well.

[theSarawr]

This seems to be happening again. Can you confirm whether the same issue has cropped up?

Example
From: @mtgdan
To: @chegghelp
Message: “Here is the order number”
(redacted from message for privacy: pic.twitter.com URL containing the order #)
Sent: 3:52pm Pacific