Discrepancy in followers_count

[dankimche]

Hi Andy,

Thank you very much for your reply!

This is actually very strange because I made this request once per hour on last 7 days, and had only 3 times the right result above more than 150.

Also, I searched for “twitter pumaargentina stats” on google, and one of the result was https://www.socialbakers.com/statistics/twitter/profiles/detail/177254406-pumaargentina
As you can see on this page, they seem to have the same issue I have: about 84110 followers.

Is there a chance that you are always hitting an accurate server/data center and I am almost always hitting a non accurate server?

Perhaps this is due to server attribution rules?

Thank you,
Dan

[dankimche]

Hi @andypiper

Any thoughts on this?

Thank you,
Dan

[andypiper]

All I can say right now is that I’ve checked again today and I’m seeing consistency between API calls, and the web profile (84633 at the time of typing this). I have no idea when that other website may have cached the value that it is showing, but it is demonstrably incorrect / out of sync since you can follow the link directly from that page to the profile, and see 84633 is the current figure. Are you continuing to see a consistent ratio of differences?

[dankimche]

Hi @andypiper

Again, thank you for your answer.

Unfortunately, I still see the discrepancy, please see an example of result I got just few minutes ago: https://jpst.it/RrSR

As you can see, I get 84092 followers, and the status object contains an object with date “Mon Jan 09 17:38:54 +0000 2017”. On this date, there were 84600+ followers.

My server is on Amazon AWS in california, I tried to release/take a new instance to get a new IP address, but had same result.

Please, tell me how I can provide more information.

Thank you,
Dan

[andypiper]

In your original post you indicated that the results were inconsistent on an 80/20 proportion of calls. Is that still the case or are you seeing an incorrect value in 100% of cases?

[dankimche]

Hi @andypiper

I am now seeing it on 100% of the cases.

I tried to reproduce the issue on apigee, without success, anyway, I have seen two differencies in the header that can help debug:

on apigee, I can see:
Server: tsa_b
x-access-level: read-write-directmessages

on my requests:
server: tsa_a
x-access-level: read

I tried many times, and still can’t have another server or the correct number of followers.

Thank you,
Dan

[andypiper]

That’s definitely odd and does sound routing-related. It may take me a couple of days to check further as I just boarded a flight. Out of interest does anything change if you modify your app key access permissions? I would not expect it to, but I’m just curious.

[dankimche]

Hi @andypiper

I tried with read-write-directmessages, tried with another app, tried with an app from another user, tried /users/lookup instead of /users/show and always had the same issue.

Thank you,
Dan

[andypiper]

Got it - thanks for the additional information. I’ll try to dig in on this in the next couple of days.

[PaulCroubalian]

I’ve been seeing it as well. Curiousity. . . are large hosting providers able to take Twitter “snapshots”? From what I see, accuracy is about 36hours behind.

[andypiper]

There’s a chance you might be hitting some CDN somewhere but I don’t think that’s systemic to Twitter specifically.

Are you able to provide additional data (specific accounts, percentage accuracy / frequency or is this 100% inconsistent, etc)? Thanks.

[dankimche]

Hi @andypiper

Actually, I made a test, and result is really weird!

I made a script that called once per second for 500 times /users/show with user_id ‘177254406’, and that print the number of followers. I tailed the output, and, as number of followers didn’t move, I went to the account and followed it myself, thinking that result won’t change as it was not correct (84086 instead of 84630) but it actually gained one follower (see in result file around try number 87).

To be sure it was due to my action, I then unfollow, follow again, then asked my coworkers to follow then unfollow the account, and their action were always reflected on the number of followers, even if it was wrong.

Also, in rare cases (3 over 500), I got the right number!

Please, see here the result file: https://jpst.it/Rxjb

Thank you,
Dan

[aartiles24]

We are seeing the same behaviour here too for both followers_count and friends_count. We monitor those values for our customers and we are seeing drops like this

Currently https://twitter.com/milosburgershop has 6687
Unfortunally the behaviour is random so I’m not able to reproduce it.

[andypiper]

I see 9425 via the API and via the web for that account. If you can provide API call responses and headers that reflect an inaccurate number, that would be helpful.

I am continuing to investigate whether there could be any caching that might affect this.

[dankimche]

Hi @andypiper

Please, see here https://jpst.it/ROnY all hourly nb of followers I get from the API since Jan 10th.

It seems that the number of followers started to increase progressively on Saturday January 14th at 2AM UTC until 4PM where it reached the real number of followers. From this date, it seems that I always get the real number of followers from the API.

I made again a 500 request loop and always get the correct number of followers, so it seems that the problem is fixed now.

Could you please tell me if you know what happened? and/or if there was a software update or hardware modification on Saturday 14th around 2AM UTC?

Anyway, thank you!
Dan

[dankimche]

also, I always see the correct number of followers/following for @milosburgershop

[andypiper]

I think this matches my original diagnosis - eventual consistency and caching being an issue. I’m not aware of any specific change on that date, but it could be that the job that was syncing the data came to life around that time. I’ll admit that I’m surprised that it would have taken a period of a week or more before this all settled out for you, however - usually I would not expect things to take this long.

[dankimche]

Hi @andypiper

Thank you for your answer.

Actually, I’m almost sure that this was not a cache issue as during my test (post number 14 of this discussion), I realized that, when me and my colleagues followed the account, we have seen the incorrect number of followers increase, and when we unfollowed, we have seen it decrease… my test was a loop of 500 API requests with a 1 second sleep between each request, and we made these follow/unfollow during the 500 seconds timeframe and saw the results changing live (on the second we clicked, the number of followers did change).

a caching issue should not take in account these live changes.

Please, tell me if I’m wrong or if I missed something.

Thank you,
Dan

[andypiper]

Twitter’s backend systems run on a distributed Apache Mesos cluster of decomposed services. There are often cases where data sets will need to resynchronise between instances and data centers.

While I realise that this discrepancy took place over an unusually long period of time, I’ve already seen evidence in the calls you listed earlier in the thread that some instances were reporting different values to others, so I think this is extremely likely to be the background here. You should also take into account that as we continue to clean up spammy accounts or users are suspended, there will occasionally be a sharper drop than unusual in follower or following numbers, and those kind of larger changes may take longer to be reflected on the API than at other times.

[dankimche]

Hi @andypiper

Ok, this makes sense!

Thank you very much for taking time to answer my questions!
Dan