Streaming API's Filter


#1

Hi all,

the filter function ( statuses/filter) of streaming API seems to return very little(or not at all) tweets on some key words.

if i put “track=food” or track=twitter, it returns a lot of tweets.

but if i put in some real keyword like spc12 or john, it returns nothing. I v also made some tweets on the keywords i try to search, but nothing show up, i didnt turn on the “protect my tweets” option in my settings.

Could someone explain to me if i have done something wrong here?

thanks so much


#2

any one?


#3

Can you share how you’re formulating the request? I see plenty of data for keywords like john (as there are a number of people tweeting the word “john” in real time) but spc12 is a bit rarer of a term and it would be expected to not get a steady stream of Tweets.


#4

One thing to be aware of is that some HTTP clients buffer data, so you may not see messages right away until the buffer grows to a certain size. This isn’t a problem for high-volume streams, but for certain low-volume queries it can be problematic. It manifests itself with gzip often - as an example, here’s a way to fix a Java implementation: https://dev.twitter.com/docs/streaming-apis/processing#Gzip_and_Java


#5

this is my code:

public static void Streaming(string method, string hashAlgorithm, string url, string consumerKey, string consumerSecret, HttpSessionStateBase Session)
{
while (true)
{
try
{
url = OAuth.GetUrl(method, hashAlgorithm, url, consumerKey, consumerSecret);
var req = HttpWebRequest.Create(url);
if (method != null) req.Method = method;
List result = new List();
using (var response = req.GetResponse())
using (var stream = response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
while (true)
{
var line = reader.ReadLine();
result.Add(line);
Session.Remove(“Tweets”);
Session[“Tweets”] = result;
}
}
}
catch (Exception ex)
{
Session.Remove(“Tweets”);
Session[“Tweets”] = ex.Message;
Thread.Sleep(5000);
}
}
}

public static string GetUrl(string method, string hashAlgorithm, string url, string consumerKey, string consumerSecret)
{
method = method ?? “GET”;
hashAlgorithm = hashAlgorithm ?? HMACSHA1;

        string timestamp = Math.Floor((DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalSeconds).ToString();
        string nonce = GetRandomString(10);
         
        int index = url.IndexOf('?');
        string querystring;
        if (index == -1) {
            querystring = string.Empty;
        } else {
            querystring = url.Substring(index + 1);
            url = url.Substring(0, index);
        }

        //parse the querystring into a dictionary, rather than NameValueCollection, so it's easier to manipulate
        Dictionary<string, string> query = querystring.Split('&').Select(x => {
            int i = x.IndexOf('=');
            if (i == -1) return new[] { x, null };
            else return new[] { x.Substring(0, i), Uri.UnescapeDataString(x.Substring(i + 1)) };
        }).ToDictionary(x => x[0], y => y[1]);

        //add the oauth stuffs
        query.Add("oauth_consumer_key", consumerKey);
        query.Add("include_entities", "true");
        query.Add("oauth_nonce", nonce);
        query.Add("oauth_signature_method", "HMAC-SHA1");
        query.Add("oauth_timestamp", timestamp);
        query.Add("oauth_token", "27111111-y97mIqoSxxxxixxxxxxxx");
        query.Add("oauth_version", "1.0");
        query.Add("track", "spc12");

        //put the querystring back together in alphabetical order
        querystring = string.Join("&", query.OrderBy(x => x.Key).Select(x => string.Concat(x.Key, (x.Value == null ? string.Empty : "=" + x.Value.UrlEncode()))).ToArray());

        string data = string.Concat(method.ToUpper(), "&", url.UrlEncode(), "&", querystring.Substring(1).UrlEncode());

        string sig;
        using (var hasher = GetHashAglorithm(hashAlgorithm)) {
            hasher.Key = (consumerSecret + "&NxxxxxxxxxxxxxwmAUfM").GetBytes();
            sig = hasher.ComputeHash(data.GetBytes()).ToBase64();
        }

        return string.Concat(url, "?", querystring, "&oauth_signature=", sig.UrlEncode());
    }

#6

any one?


#7

This is a lot of code to look through. It looks like StreamReader is what you’re using to read off of the HTTP connection - make sure this isn’t buffered by default.