Duplicate tweets

[amr777505]

while pulling tweets from twitter I see that a lot of tweets are duplicates, although they do have different IDs, maybe people are sharing tweets is there a way to eliminate these duplicates without just comparing all the results with each other ??

[andypiper]

There’s nothing in the API that would help with this, no - you would need to do that filtering in your code.

[Tazeg]

I suppose those are retweets. A RT is a new tweet, so a new ID, but the content is the same as the original, except geolocated data, and normally with “RT” added at the beginning of the text. You may filter your search to avoid retweeted tweets with : " -filter:retweets".