Incremental searches using since_id misses tweets

[EngagorTom]

At Engagor we created an incremental crawler that performs search request using the since_id parameter. Each time a tweet is found we bump the since_id. So we only fetch tweets that are more recent since the one we have seen. Lately we have been missing out on some tweets. After some investigation we found the following

We did a search request at Tuesday, March 10, 2015, 07:02:27 with the following params

{"q":"@nmbs","count":100,"result_type":"recent","since_id":"575187765559713792"}

We got back a tweet with id 575189806499635201 (posted Tuesday, March 10, 2015, 07:01:51). However this request should also have returned tweet with id 575189802754129920 (posted Tuesday, March 10, 2015, 07:01:50). If we rerun the original request now, we do get the tweet we missed…

Is it possible that in some cases newer tweets are returned before older ones? If so, is there a reliable way to build an incremental crawler?

[EngagorTom]

We found 4 consecutive tweets mentioning @nmbs with the following ids:

$id1 = “575187765559713792”;
$id2 = “575189802754129920”;
$id3 = “575189806499635201”;
$id4 = “575190091309662208”;

When we specify id1 as since_id and id4 as max_id, we only get back one tweet (i.e. the one with id3). However, if we specify id3 as the max_id, we also get back the tweet with id2.

[m_k_kane]

We are seeing the same problem. I can consistently reproduce. Here is an example of three searches, one without max_id or since_id, one with max_id, one with max_id and since_id. Note that max_id and since_id are set such that all three searches should return the same results

+--------------------+---------------------------+---------------------------------+-----------------------------------+
| tweet ids          | {'count': 100,            | {'count': 100,                  | {'count': 100,                    |
|                    |  'q': 'cigref',           |  'max_id': 575635858893717505L, |  'max_id': 575635858893717505L,   |
|                    |  'result_type': 'recent'} |  'q': 'cigref',                 |  'q': 'cigref',                   |
|                    |                           |  'result_type': 'recent'}       |  'result_type': 'recent',         |
|                    |                           |                                 |  'since_id': 574497480571748352L} |
+--------------------+---------------------------+---------------------------------+-----------------------------------+
| 575635858893717504 | x                         |                                 |                                   |
| 575624089689268224 | x                         | x                               | x                                 |
| 575623042484858880 | x                         | x                               | x                                 |
| 575622718613250048 | x                         | x                               | x                                 |
| 575621979132166144 | x                         | x                               | x                                 |
| 575621529565806592 | x                         | x                               | x                                 |
| 575621322404945920 | x                         | x                               | x                                 |
| 575601313154744321 | x                         | x                               |                                   |
| 575598608017416192 | x                         | x                               |                                   |
| 575598094940831744 | x                         | x                               |                                   |
| 575597231480422401 | x                         | x                               |                                   |
| 575596453114048512 | x                         | x                               |                                   |
| 575590441611034625 | x                         | x                               |                                   |
| 575589618168512513 | x                         | x                               |                                   |
| 575581690061545472 | x                         | x                               |                                   |
| 575576594309210112 | x                         | x                               |                                   |
| 575575450497454080 | x                         | x                               |                                   |
| 575574090272141312 | x                         | x                               |                                   |
| 575571913747136513 | x                         | x                               |                                   |
| 575557630262968320 | x                         | x                               |                                   |
| 575546718407757824 | x                         | x                               |                                   |
| 575537440720621568 | x                         | x                               |                                   |
| 575446528761925633 | x                         | x                               |                                   |
| 575439538463580161 | x                         | x                               |                                   |
| 575379364533039104 | x                         | x                               |                                   |
| 575366762906128384 | x                         | x                               |                                   |
| 575365734261485570 | x                         | x                               |                                   |
| 575365288457338880 | x                         | x                               |                                   |
| 575364893102186496 | x                         | x                               |                                   |
| 575364541925711872 | x                         | x                               |                                   |
| 575364116975611905 | x                         | x                               |                                   |
| 575363398591995904 | x                         | x                               |                                   |
| 575362746700730368 | x                         | x                               |                                   |
| 575350642660765696 | x                         | x                               |                                   |
| 575339863194624001 | x                         | x                               |                                   |
| 575339364324143105 | x                         | x                               |                                   |
| 575329197440286721 | x                         | x                               |                                   |
| 575295344696815616 | x                         | x                               |                                   |
| 575287634714574848 | x                         | x                               |                                   |
| 575285850688323584 | x                         | x                               |                                   |
| 575285038977253377 | x                         | x                               |                                   |
| 575274209074659330 | x                         | x                               |                                   |
| 575273705443606528 | x                         | x                               |                                   |
| 575273574275112960 | x                         | x                               |                                   |
| 575263133150871552 | x                         | x                               |                                   |
| 575260279040315392 | x                         | x                               |                                   |
| 575260183166869504 | x                         | x                               |                                   |
| 575260056402464768 | x                         | x                               |                                   |
| 575195392020774913 | x                         | x                               |                                   |
| 574991911930437632 | x                         | x                               |                                   |
| 574896310421168128 | x                         | x                               |                                   |
| 574870836370411521 | x                         | x                               |                                   |
| 574854991799681024 | x                         | x                               |                                   |
| 574854461195165696 | x                         | x                               |                                   |
| 574841464687058944 | x                         | x                               |                                   |
| 574841333115981824 | x                         | x                               |                                   |
| 574658008006983682 | x                         | x                               |                                   |
| 574608203893231617 | x                         | x                               |                                   |
| 574591014264049664 | x                         | x                               |                                   |
| 574575401348956164 | x                         | x                               |                                   |
| 574571987235921920 | x                         | x                               |                                   |
| 574518527987687425 | x                         | x                               |                                   |
| 574508636665430016 | x                         | x                               |                                   |
| 574508057570443264 | x                         | x                               |                                   |
| 574507694339547136 | x                         | x                               |                                   |
| 574506646703108096 | x                         | x                               |                                   |
| 574506393098657792 | x                         | x                               |                                   |
| 574499406130626560 | x                         |                                 |                                   |
| 574498892932317184 | x                         |                                 |                                   |
| 574497480571748353 | x                         |                                 |                                   |
+--------------------+---------------------------+---------------------------------+-----------------------------------+

Edit: That looks a bit crap sorry. Here it is as a gist https://gist.github.com/michaelkane/ab3a762ef1bcf6a78c00

[rchoi]

Sorry for the delay here. Our eng team is looking into it; I’ll reply here when I have more information.

[rchoi]

Hi everyone,

We rolled out a fix yesterday. Wanted to get your input if your use cases are now working. Please let me know!

[rchoi]

This topic was automatically closed after 6 hours. New replies are no longer allowed.