I’ve done a bit more digging on this here, and discovered that the problems seem to be in connecting to the Twitter streaming API from AWS EU infrastructure, and seemingly nowhere else.
I wanted to try to remove as many mitigating factors as possible, so started playing around with connecting using curl. I’m seeing the same issues connecting to the API simply using curl and not passing any oAuth credentials at all - so that should rule out any problems being caused by the account being used for authentication. This was also being done from an entirely fresh instance brought up in the AWS EU region in Ireland - so anything to do with IP restrictions shouldn’t be affecting these tests either.
When trying to connect direct with curl the response we’re expecting is to be told that we’re unauthorized - ie. a response containing:
Problem accessing ‘/1.1/statuses/filter.json’. Reason:
Unauthorized
We knocked up a quick script (copied below) to test how often we see this dropped connection behaviour and to compare it to trying to create the same connections from servers located elsewhere. Trying to connect from a box in the AWS EU region we saw 52 empty replies and 48 successful connections in 100 attempts. Doing the same thing from a box in the AWS US East Coast facility, and also direct from a laptop over a pretty awful ADSL line from here in London, both of those scripts managed to connect successfully 100/100 times.
I’m also seeing similar behaviour in this testing with curl, in that requests which have a very small payload - ie. just a handful of track terms, seem to connect successfully. However the failures occur once there are more terms. I’m unsure so far whether the frequency of the empty responses increases with term count, or whether there is some cutoff above which the failed connections occur.
#!/bin/bash
SLEEP_TIME=30
TERMS='disillusionment,gram,masculines,flounders,skullduggery,dressage,workplace,snare,colloquia,Magsaysay,effeminacy,Episcopalian,gargles,Erika,intermediate,Tartar,bugging,nonresidents,hooter,Balaton,shirks,glaucoma,commissariat,Transcaucasia,ornithology,adulterer,hagglers,Zionisms,regulator,Hooke,inseminating,ahoy,diplomat,perishables,Lilliputian,schlock,toe,archipelago,thou,readjustments,profusions,ship,motherless,denouement,Silva,Tolyatti,declension,clergyman,zany,killjoy,wittier,dominance,fixation,prefabricates,Alvarado,campaigners,medal,reactivated,Ruth,Jeremiah,lurk,furnishings,mild,rebound,beltway,effigy,antiabortion,feels,sturdiness,stuffing,scrubbing,Bourbon,hemorrhoids,mop,Capablanca,mobiles,globules,centerfold,clef,Na,rostrum,Helene,deaconess,maelstroms,Puccini,vivacity,ion,councils,snoozing,Lela,floozy,Cotopaxi,Finnish,trolled,scalpel,Swanson,reforms,nuisance,biospheres,triennial,counts,Laundromat,Shanghai,ladybugs,disciplining,ratting,undemonstrative,deposing,merit,cooperating,vagrancy,ovum,edification,Emile,ye,Caribbeans,abreast,obsessions,drones,oxide,ornament,brightly,Passovers,hood,operand,yeast,wiring,Mahabharata,tiffs,defaulters,sparse,Moloch,vision,pyromaniac,improvident,attest,imagining,Atacama,paramedical,tribune,aquaplanes,conditioners,resurgences,hydrocarbon,hedonistic,shoelace,oversight,purge,Lithuania,provides,snipe,truant,quintets,litigation,prudential,hygienically,stimuli,single,oversells,armlet,polisher,jellyfish,stored,gill,carton,wilderness,scone,internists,multitudinous,Dawes,agronomy,voluntaries,limestone,slug,weapon,inform,pleat,boodle,twin,gazebos,repays,Shankara,washbowls,seismologists,grudge,wearier,basting,Buick,manageability,boatmen,castaway,Albuquerque,misapprehends,reserving,cannonade,rapine,castigation,Chattanooga,chained,Timur,whimsical,joyous,reassures,captions,Justinian,bloodstream,commissioner,rhombi,globally,immunity,Keven,Cinderella,swish,Scandinavians,weeknight,pricier,acacias,cochleae,fever,perfectionist,ascension,armrest,transshipped,hardiness,accusing,ramifying,blitz,Verde,uncle,ranting,lipstick,slur,illegitimate,tenderizing,Nikolayev,civilities,tared,caw,unhorsing,roadway,misrule,attitude,universality,simulcast,clubbing,Wac,cunninger,preexisting,afforests,awesome,substantiating,slacker,thumbtack,wheelwright,comparability,countenances,anemones,purism,bibles,Lowenbrau,threnodies,locutions,sweats,retold,devilry,rulers,conductors,Grey,Saskatchewan,albatross,alkaloids,species,drugstore,obliteration,grizzly,crazes,entertainers,maligns,championship,deconstructions,statesmanship,Austerlitz,pathogens,reverential,scampers,Friday,taffeta,recklessly,proclaimed,putative,fuddle,oxymoron,fueling,dither,undisciplined,corrosion,civilization,anticipations,engraving,carjacking,expostulated,rubier,sequel,inpatient,acrimony,ceases,heliport,Berliner,orangutangs,overhauls,pounded,hinds,Fronde,bedstead,demilitarizing,fretfulness,Yiddish,plutocracy,hobby,super,finitely,residual,Bastille,cactus,sax,tactlessly,dustier,Connolly,stargazers,nerdier,pups,heeling,goodliest,windpipe,Algiers,Poirot,pediment,Cindy,melodrama,colloquy,diarist,tequilas,Downy,naps,subservience,catalogue,filed,fatty,folklore,portioned,Hohokam,legging,Tallinn,deliriously,rightmost,motorbiked,miscarries,inner,outshined,metacarpi,agitating,interviewers,foes,Kalashnikov,sylph,bottles,Verizon,adverted,scrambled,woodwind,demoralized,beaux,insidiousness,enthronement,mismatched,improbability,vigilant,Antipas,Nankings,redraws,jauntiness,Len,synchronous,Emacs,moldering,Hovhaness,enclose,foreman,weirdness,dermatitis,admixtures,originality,Irtish,rosette,Mb,Barbra,stockrooms,confusingly,misrepresent’
CURL_CMD="curl --max-time 120 --connect-timeout 60 --request ‘POST’ https://stream.twitter.com/1.1/statuses/filter.json --data ‘track=$TERMS’"
let CONNECT_COUNT=0
let FAILURE_COUNT=0
for i in $(seq 1 100)
do
echo "Connecting attempt $i"
response=$($CURL_CMD 2>&1)
if [[ “$response” == *Unauthorized* ]]
then
echo "Connected"
let CONNECT_COUNT++
elif [[ “$response” == *Empty* ]]
then
echo "Timeout"
let FAILURE_COUNT++
else
echo "Unrecognised response"
echo $response
fi
sleep $SLEEP_TIME
echo "Failures: $FAILURE_COUNT. Connections: $CONNECT_COUNT"
done
echo “FINAL: Failures: $FAILURE_COUNT. Connections: $CONNECT_COUNT”
