Sandbox 30 days search returns error

[LeoUrushi]

I’m using python to make search queries. I’ve made successful calls using standard search. When I switch to Premium/ Sandbox/ 30-Day option, it will return an error message.
Instructions in developer(dot)twitter(dot)com seems to suggest the standard search uses “q”, whereas premium searches use “query”. And “query” seems to be the only mandatory parameter. I kept it simple whilst I test the proof of concept.

Standard search

TWITTER_URL = ‘https://api.twitter.com/1.1/search/tweets.json’

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

keyword = ‘dogs’

url = twurl.augment(TWITTER_URL,
{‘q’: keyword})

connection = urllib.request.urlopen(url, context=ctx)
data = connection.read().decode()
js = json.loads(data)
print(json.dumps(js, indent=2))

Premium/ 30-Day option

TWITTER_URL = ‘https://api.twitter.com/1.1/tweets/search/30day/prod.json’

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

keyword = ‘dogs’

url = twurl.augment(TWITTER_URL,
{‘query’: keyword})

connection = urllib.request.urlopen(url, context=ctx)
data = connection.read().decode()
js = json.loads(data)
print(json.dumps(js, indent=2))

Error message from the 30-Day option

connection = urllib.request.urlopen(url, context=ctx)
… …

raise HTTPError(req.full_url, code, msg, hdrs, fp)

urllib.error.HTTPError: HTTP Error 422: Unprocessable Entity

Screenshot below:

[IgorBrigadir]

There are more differences between standard and premium call than just the parameter and url, Premium for example, supports App only authentication, and has a completely different set of query operators.

GitHub - geduldig/TwitterAPI: Minimal python wrapper for Twitter's REST and Streaming APIs is a good library that supports both

Here’s a working example:

https://gist.github.com/igorbrigadir/abeba05058fc01dd3923354a4f69c9c2

[LeoUrushi]

Thank you for your help.

I’ve played around with this. And I managed to get API response using this library (software? Package?).

I was really hoping to find a native solution that only uses the actual Twitter. But it looks I will have to use this third party library at my skill level.

Thanks.

Is there a concept similar to cursoring in this search API?

Cursoring

I find that the 30-day search result often returns only the results from the most recent date in the range.
For example, I set the date range from Jan 1st to Jan the 5th and set the max results at 100. I only see the tweets from the Jan 4th. (Weirdly, it’s usually the day BEFORE the final date set in the range)

Is there a way to keep looping the search until I find ALL the tweets similar to the cursor?

I’ve noticed that there is a section called “next” towards the very end of the API response. Is that the equivalent of cursor?

[IgorBrigadir]

Yes, paging in Premium is with the next token. Remember that each “page” uses up a request.

If you want an easier way, GitHub - twitterdev/search-tweets-python: Python client for the Twitter 'search Tweets' and 'count Tweets' endpoints (v2/Labs/premium/enterprise). Now supports Twitter API v2 /recent and /all search endpoints. can be run as a CLI without modifying python - specifying all the date ranges / max results etc. This handles the next tokens for you - but note the setting on limits - so you don’t use up all your requests with a query that doesn’t set them. This tool however doesn’t support standard search, only premium.