POST to upload/media fails often with "media type unrecognized"

slinkp23 · 2015-12-01T20:50:41.942Z

I am POSTing with base64-encoded data in the media_data parameter to media/upload. This usually works fine, but there are some images that seem to sometimes succeed and sometimes fail.

Here’s some example s3 buckets pointing to JPEG images that seem to fail:
https://percolate.com/cdn/library/image/60304017
https://percolate.com/cdn/library/image/58461991

This one uploaded fine, even though I have confirmed it is byte-for-byte identical to the second image above:
https://percolate.com/cdn/ugc/image/raw/e9ed458e66c0b726c7b713adf7711f4d.jpg

Any hints what might be wrong here? The fact that it usually works - and sometimes even works with exactly the same data - makes me suspect a problem on Twitter’s side.

niall · 2015-12-01T21:23:00.642Z

Image 60304017 responds with a generic Content-Type of binary/octet-stream; image 58461991 responds with image/jpeg.

The bodies might be byte-for-byte identical but it looks like key headers vary.

slinkp23 · 2015-12-01T22:14:28.681Z

True, but that header has no bearing on what we send to twitter. We load the bytestream, base64-encode it, and POST to media/upload. The content-type returned by those S3 objects doesn’t get used anywhere in our upload process.

rchoi · 2015-12-02T07:10:58.010Z

I’ve been playing around with this Python code sample:

GitHub

twitterdev/django-rest-apis

django-rest-apis - Sample Django App using Twitter Sign in (OAuth) and REST APIs.

In using python-twitter and the “media” parameter for the below sample code, it seems to upload properly.

Can you try either:

1.) using the media parameter and seeing if that works,
2.) ensure you’re doing chunking for video (it’s required)
3.) Trying the below code to see if it works

For chunking the code is available here in python:

github.com

twitterdev/django-rest-apis/blob/master/home/views.py

import base64
import json

from django import forms
from django.shortcuts import *
from django.contrib.auth.decorators import login_required, user_passes_test
from django.conf import settings

from social_django.models import UserSocialAuth

import twitter
from twitter import TwitterError

from home.models import Image 

# QUERY_MAX_STATUSES = 3200
QUERY_MAX_STATUSES = 200

SIZE_5MB = 5 * 1024 * 1024

This file has been truncated. show original

rchoi · 2015-12-02T08:20:08.989Z

FWIW: I have the code to test your own file types (and see any errors) on Heroku:

http://django-rest-apis.herokuapp.com/media/video

I attempted all your files – in both “media” and “media_data” modes, and they uploaded and attached to a tweet fine.

I’m sure there’s something I’m missing, but give your files an upload yourself, and feel free to download the code and test it out to see what I’m missing:

GitHub

twitterdev/django-rest-apis

django-rest-apis - Sample Django App using Twitter Sign in (OAuth) and REST APIs.

slinkp23 · 2015-12-05T07:24:59.781Z

@rchol thanks very much for the replies. I’ve been continuing to try to reproduce this - no luck so far. It’s odd, because we’re still seeing the symptoms in production logs with customer data. I’ll keep at it.

slinkp23 · 2015-12-08T23:17:33.727Z

@rchoi I still haven’t been able to reproduce this at will, but found a likely culprit on our side … before upload, the image data is fetched via an old library that assumed urllib2.urlopen(...).read() is guaranteed to return all the data, which is false. Anytime that assumption is violated, we’d be uploading truncated data. If I manually truncate the data to an arbitrary length, I have verified that Twitter (sometimes) gives this “media type unrecognized” response.

I’m going to push out a fix for this and watch the logs to see if this solves the problem for good. Optimistic, but we’ll see

Thanks again. I’ll report back here with my results.

jbulava · 2015-12-08T23:20:04.831Z

Glad to hear that you may have pinpointed the issue. Is the library in question internal to your team or is it public?

slinkp23 · 2015-12-08T23:21:09.383Z

Internal.

jbulava · 2015-12-09T00:55:21.030Z

Ok, thanks. Let us know if your change makes a difference when pushed.

slinkp23 · 2015-12-18T19:54:28.434Z

We are pretty sure nothing is being truncated now, but still seeing this issue occasionally. Will follow up again if we find a consistent recipe to reproduce it, eg. with twurl.

brianatperc · 2016-01-06T21:24:48.675Z

Here is an example that fails consistently:

We also tried posting the Tweet natively (not through the API), and it fails with the error:
“Uploading photo failed! Please try again later.”

The image doesn’t seem to be corrupt, but it wouldn’t surprise me. Any ideas?

jbulava · 2016-01-07T06:42:05.802Z

Going back and forth with the Media Platform team. The one odd thing about this image to me is that it doesn’t have a color space or embedded color profile. I’m working to confirm if this would through an “unsupported” error.

jbulava · 2016-01-12T05:27:08.720Z

Upon further inspection, the team is getting the following message when parsing headers:
invalid TIFF header in EXIF data

This is causing the entire header parsing to fail. The rest of the image is fine though, so you may just be able to correct the EXIF data and try again. The team will investigate a way to get this messaging back to the user.

slinkp23 · 2016-01-12T18:55:06.286Z

@jbulava Thanks. Was that for the images I originally linked to in my original report, or the one Brian Chance commented about on jan 6th? Do they all fail the same way?

slinkp23 · 2016-01-19T21:05:20.496Z

@jbulava did you guys make relevant fixes recently? I just tried my bad PNG and Brian’s bad JPEG from this thread, and both worked fine - both natively and via API!

jbulava · 2016-01-26T15:37:04.289Z

@slinkp23, my response was indeed referring to Brian’s image. The media platform team did mention that they may be able to clean an image’s exif data rather than forcing the upload to fail, so this might have been shipped last week.

bcen_ · 2016-02-09T21:01:02.975Z

Sorry to hi-jack this, I am having a similar issue here: 131 "Internal Error" when creating a PNG image tweet

Wonder if this is related or not, and can the media platform team take a look on those images to see what’s causing the issue?

Thanks.

Grant_Hyatt · 2016-06-15T13:45:08.049Z

Hi Guys.
For over a year I’ve not been able to add more than 3 photographs to a tweet. I just get “Uploading Photo failed, please try again later” But I try and try and try and try… and never get anywhere. Have tweeted support numerous times but never get a response. Is there anything anyone can do here?