Adding a user makes gzipped sitestream broken


#1

Hi,

I recently have error on gzipped sitestream after added users. I noticed this issue on last Thursday or Friday(JST).

The situation is…

  1. Open a sitestream with “Accept-Encoding: deflate, gzip” and get control_uri. When I remove Accept-Encoding header from request header, this error never happened.
  2. Add users by the control stream API. This action make a block contain the added user’s friends ids on sitestream. But this make the HTTP body inflater loss the end of gzipped block or sometimes raise Zlib::DataError. Thus my program fails to continue process the stream.

I think this issue is added user dependent because some users make this error but not seen on another added users. And further more, the users who made error make sometimes no error.

Log:
C: log from control stream API
S: log from sitestream

On Normal:
C: Add: USER-A
C: HTTP: 200
S: chunk tail: "\xFE\a\x00\x00\xFF\xFF"
S: chunk size: 653
S: for USER-A_id, [“for_user”, “message”] # USER-A’s friends
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: alive

On Error:
C: Add: USER-B
S: chunk tail: "Q7x\n\xDE["
S: chunk size: 4096 # must come the next or last block.
C: HTTP: 200
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
S: chunk tail: "\x02\x00\x00\x00\xFF\xFF"
S: chunk size: 8
… last forever …

Alway getting same chunk size(4096 bytes) and the chunk ends with the same “Q7x\n\xDE[” on error. The error chunk has a differrent tail if the other user is added as you expect. For example, added USER-C

C: Add: USER-C
C: HTTP: 200
S: chunk tail: "j\x9A]\xED%\xAB"
S: chunk size: 4096 # size was the same as USER-B

Interestingly sometimes make no error even if added the same USER-B.
C: Add: USER-B
S: chunk tail: "\xFC\x0F\x00\x00\xFF\xFF"
S: chunk size: 494
S: chunk tail: "\xF9\x03\x00\x00\xFF\xFF"
S: chunk size: 2073
S: chunk tail: "\xE3\x1F\x00\x00\xFF\xFF"
S: chunk size: 1588
S: for USER-B_id, [“for_user”, “message”] # USER-B’s friends
C: HTTP: 200

My application is written by Ruby and the simplified test script for sitestream is here.

Json_regexp = Regexp.new("^.*?\r\n\r\n")
zstream = Zlib::Inflate.new(Zlib::MAX_WBITS + 32)

consumer = OAuth::Consumer.new(C_token, C_secret)
access_token = OAuth::AccessToken.new(consumer, A_token, A_secret)

request_url = "https://sitestream.twitter.com/1.1/site.json"
request_url += “?follow=#{init_users}”

uri = URI.parse(request_url)
request = Net::HTTP::Get.new(uri.request_uri)

no gzip, no error

request[“Accept-Encoding”] = “deflate, gzip”

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true if uri.scheme == ‘https’

request.oauth!(http, consumer, access_token)

http.start.request(request){|response|
case response
when Net::HTTPSuccess
buffer = ""
response.read_body{|chunk|
next if chunk == “”

  print "S: chunk tail: #{chunk[-6 .. -1].inspect}\n"
  print "S: chunk size: #{chunk.size}\n"
  if response["content-encoding"] == "gzip"
    zstream << chunk
    buffer += zstream.flush_next_out
  else
    buffer += chunk
  end
  while json = buffer.slice!(Json_regexp)
    #print "S: JSON: #{json.inspect}\n"
    json.sub!(/\r\n\r\n$/, "")
    if json == ""
      print "S: alive\n"
    else
      message = JSON.parse(json)
      if message.has_key?("for_user")
        user_id = message["for_user"]
        print "S: for #{user_id}, #{message.keys.inspect}\n"
      else
        print "S: message: #{message.inspect}\n"
        if message.has_key?("control") and message["control"].has_key?("control_uri")
          control_uri = message["control"]["control_uri"]
        end
      end
    end
  end
}

end
}

Thank you.