Hi,
I recently have error on gzipped sitestream after added users. I noticed this issue on last Thursday or Friday(JST).
The situation is…
- Open a sitestream with “Accept-Encoding: deflate, gzip” and get control_uri. When I remove Accept-Encoding header from request header, this error never happened.
- Add users by the control stream API. This action make a block contain the added user’s friends ids on sitestream. But this make the HTTP body inflater loss the end of gzipped block or sometimes raise Zlib::DataError. Thus my program fails to continue process the stream.
I think this issue is added user dependent because some users make this error but not seen on another added users. And further more, the users who made error make sometimes no error.
Log:
C: log from control stream API
S: log from sitestream
On Normal:
C: Add: USER-A
C: HTTP: 200
S: chunk tail: "\xFE\a\x00\x00\xFF\xFF"
S: chunk size: 653
S: for USER-A_id, [“for_user”, “message”] # USER-A’s friends
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: alive
On Error:
C: Add: USER-B
S: chunk tail: "Q7x\n\xDE["
S: chunk size: 4096 # must come the next or last block.
C: HTTP: 200
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
… last forever …
Alway getting same chunk size(4096 bytes) and the chunk ends with the same “Q7x\n\xDE[” on error. The error chunk has a differrent tail if the other user is added as you expect. For example, added USER-C
C: Add: USER-C
C: HTTP: 200
S: chunk tail: "j\x9A]\xED%\xAB"
S: chunk size: 4096 # size was the same as USER-B
Interestingly sometimes make no error even if added the same USER-B.
C: Add: USER-B
S: chunk tail: "\xFC\x0F\x00\x00\xFF\xFF"
S: chunk size: 494
S: chunk tail: "\xF9\x03\x00\x00\xFF\xFF"
S: chunk size: 2073
S: chunk tail: "\xE3\x1F\x00\x00\xFF\xFF"
S: chunk size: 1588
S: for USER-B_id, [“for_user”, “message”] # USER-B’s friends
C: HTTP: 200
My application is written by Ruby and the simplified test script for sitestream is here.
Json_regexp = Regexp.new("^.*?\r\n\r\n")
zstream = Zlib::Inflate.new(Zlib::MAX_WBITS + 32)
consumer = OAuth::Consumer.new(C_token, C_secret)
access_token = OAuth::AccessToken.new(consumer, A_token, A_secret)
request_url = "https://sitestream.twitter.com/1.1/site.json"
request_url += “?follow=#{init_users}”
uri = URI.parse(request_url)
request = Net::HTTP::Get.new(uri.request_uri)
no gzip, no error
request[“Accept-Encoding”] = “deflate, gzip”
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true if uri.scheme == ‘https’
request.oauth!(http, consumer, access_token)
http.start.request(request){|response|
case response
when Net::HTTPSuccess
buffer = ""
response.read_body{|chunk|
next if chunk == “”
print "S: chunk tail: #{chunk[-6 .. -1].inspect}\n"
print "S: chunk size: #{chunk.size}\n"
if response["content-encoding"] == "gzip"
zstream << chunk
buffer += zstream.flush_next_out
else
buffer += chunk
end
while json = buffer.slice!(Json_regexp)
#print "S: JSON: #{json.inspect}\n"
json.sub!(/\r\n\r\n$/, "")
if json == ""
print "S: alive\n"
else
message = JSON.parse(json)
if message.has_key?("for_user")
user_id = message["for_user"]
print "S: for #{user_id}, #{message.keys.inspect}\n"
else
print "S: message: #{message.inspect}\n"
if message.has_key?("control") and message["control"].has_key?("control_uri")
control_uri = message["control"]["control_uri"]
end
end
end
end
}
end
}
Thank you.
