[Bug?] Mentions not listed as Mentions


#1

The Following two tweets mention the account @RefugeesRPKoeln:
/susiekju/status/743114201321852928
/RaikMeinshausen/status/743353104431620097

But if you search for Mentions of this account, they are missing:
https://twitter.com/search?f=tweets&vertical=default&q=%40RefugeesRPKoeln&src=typd

If I request the mentions of this account via the api route
/1.1/statuses/mentions_timeline.json
they are missing

If I request specifically one of the missing tweets with
/1.1/statuses/show.json?id=743353104431620097

The mentioned account RefugeesRPKoeln is not listed in user_mentions

Why is that? Is it a Bug? Some sort of Spam filtering?


#2

I’m not certain. I’d imagine possibly spam filtering, since these do not seem to be promoted Tweets.


#3

Are you, or a[nother] developer still looking into this?
Would be rather important for us

Thanks


#4

What is the specific issue here? Is this a search API problem?


#5

The specific issue is that we have customers who expect to have a list of ALL tweets mentioning them.

They send us the tweets I specified above and complained that they are missing from our listing.
But it is currently impossible for us to fetch them via the appropriate API route (/1.1/statuses/mentions_timeline.json) because they are not listed there.

Now we either have to logically explain to them why these specific tweets cannot be fetched.
Or this is a bug of the Twitter API itself and needs to be fixed so that our software works as expected again.


#6

Is this an issue relating to the Twitter apps and website, or the API?


#7

I’m not sure how to answer your question.

I already told you that we are experiencing the issue with the API (specifically the mentions_timeline route) and I also said in the first post that I can reproduce the issue using the advanced search of the twitter.com website.

Would you mind forwarding this to an appropriate developer?


#8

Do the affected users have the statuses missing if they log into the Twitter website and view the mentions page?


#9

There are a few reasons why the specific Tweets might not be showing up in the mentions endpoint - are the originating users muted or blocked? that would prevent the Tweets from being in the mentions timeline.

When I retrieve either of those specific Tweets manually, I can see the accounts all listed in user_mentions, so I’m surprised you’re not seeing that.

Unfortunately there’s no guarantee that the public API will always return every item of data since the indices are optimised for performance and recency.


#10

Do the affected users have the statuses missing if they log into the Twitter website and view the mentions page?

Asked the customer, they say that they can’t see them as Notifications.

are the originating users muted or blocked? that would prevent the Tweets from being in the mentions timeline.

Customer says they are not blocked.

Even if the were blocked, that would not explain why I can’t see them when searching for them without being logged in here:
https://twitter.com/search?f=tweets&vertical=default&q=%40RefugeesRPKoeln&src=typd

When I retrieve either of those specific Tweets manually, I can see the accounts all listed in user_mentions, so I’m surprised you’re not seeing that.

I used the customers authentication data and was not able to see them.

Unfortunately there’s no guarantee that the public API will always return every item of data since the indices are optimised for performance and recency.

So it just possible that sometimes tweets are not listed in the API?

Why is the customer unable to see them in the notifications then when he says the users are not blocked?
Can you check whether that is true? We don’t have the credentials of the customer account to validate it.

Thanks for looking into it


#11

Correct - for a variety of reasons, Tweets may not be in the public API. Performance, datacenter cache inconsistencies, and short-term search indexing are often some of these reasons.

If you need higher fidelity then the commercial Gnip solution may be more appropriate. It doesn’t allow access to mentions timelines, rather a set of streaming rules that you could use to track things more clearly.

What’s interesting here is that the Tweets are missing from (web) search so they may have been temporarily blacklisted or marked as spam - NB the search API uses a different index to the web search.

I’m surprised that you say the actual user_mentions objects are missing in the Tweet object if this is pulled using the user’s auth token - I’ve not heard of that before and I’d expect Tweets to be rendered consistently between accounts.

In terms of notifications, you might also want to look at this response, since some of the same things might apply to the mentions timeline.

I realise that this is non-deterministic and that’s frustrating as a developer, but there’s not many other avenues I can think of to suggest.


#12

Correct - for a variety of reasons, Tweets may not be in the public API. Performance, datacenter cache inconsistencies, and short-term search indexing are often some of these reasons.

Does the new Premium API guarantee that all mentions are returned?


#13

You’ll see all available data that matches your query within the timeframe specified.


#14

Thank you for the fast reply, but this is a rather important issue to us – so I want to make sure.

We are working (among others) with the German Police, providing them an Interface where they gather all Mentions of their Twitter account. While no other customer is reporting missing Mentions/Tweets, it keeps happening for the Police accounts and we suspect that it does not have technical reasons, as you suggested, but rather some form of censorship.

For the German Police it is of high interest to get all Tweets – no matter how inflammatory – as especially those Tweets are the ones the Police has to react as fast as possible to prevent possible harm. Unfortunately we are not able to get the Budget for the Gnip Enterprise solution you suggested and the new Premium APIs would be a welcome solution.

Are you sure that if we buy and implement the Premium API all mentions will be available to us?

Because, looking at the features/advantages of the Premium APIs it seems to have mostly to do with how many requests we can make / how many / how detailed the results will be. It does mention to bring “reliability” but does that really mean what we hope it means?


#15

We can’t comment on specific use cases in the forums, but encourage you to apply and our team will be in touch as part of the review process. You don’t need to pay up front, if your access is approved you will have the ability to test the capabilities using the free Sandbox tier without choosing to pay for access.