Hello!
I developed a twitter collector some time ago using TwitterAPI 2v, and i’ve noticed that the tool isn’t returning information about all the media_keys.
For instance:

{
    "id": "1418216298203340805",
    "attachments": {
        "media_keys": [
            "3_1418212271789010949",
            "3_1418213471561289729",
            "3_1418214734940102660",
            "3_1418215386072313860"
        ],
        "media_keys_hydrate": [
            {
                "media_key": "3_1418215386072313860",
                "type": "photo",
                "url": "https://pbs.twimg.com/media/E66DYS3VcAQ_nOo.png"
            }
        ]
    },
    "created_at": "2021-07-22T14:28:09.000Z",
    "text": "ROCK N ROLL, TAN VIEJO COMO EL TIEMPO.\n\nBill Haley & His Comets- Rock Around The Clock\nStage Dolls- Love cries\nThe Bangles- Eternal Flame\nTNT- 10,000 Lovers\nDereck /Dominos- Layla\nBoston- More Than a Feeling\nThe Allman Brothers Band- Whipping Post\nJethro Tull- Thick as a Brick https://t.co/o2whuMhxF3",
    "author_id": "4123358326",
    "author_id_hydrate": {
        "id": "4123358326",
        "name": "mundi",
        "username": "kranvitter"
    }
}

I have also tried to make the requests using another code(provided by the autor of the library ‘TwitterAPI’ in the GitHub page of the library), but it has the same behavior.

The media_keys_hydrate field always just returns information about one media key. Is it a limitation of the library ‘TwitterAPI’ or is just that Twitter’s API doesn’t send the information of all the media keys?
Do the same thing happen using Twarc?
Is there a way to retrieve the link of the media (to download it) through the media key?
Thanks in advance!!

twarc will give you the original API response:

echo "1418216298203340805" | twarc2 hydrate
{"data": [{"author_id": "4123358326", "created_at": "2021-07-22T14:28:09.000Z", "public_metrics": {"retweet_count": 0, "reply_count": 0, "like_count": 0, "quote_count": 0}, "attachments": {"media_keys": ["3_1418212271789010949", "3_1418213471561289729", "3_1418214734940102660", "3_1418215386072313860"]}, "entities": {"urls": [{"start": 282, "end": 305, "url": "https://t.co/o2whuMhxF3", "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1", "display_url": "pic.twitter.com/o2whuMhxF3"}, {"start": 282, "end": 305, "url": "https://t.co/o2whuMhxF3", "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1", "display_url": "pic.twitter.com/o2whuMhxF3"}, {"start": 282, "end": 305, "url": "https://t.co/o2whuMhxF3", "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1", "display_url": "pic.twitter.com/o2whuMhxF3"}, {"start": 282, "end": 305, "url": "https://t.co/o2whuMhxF3", "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1", "display_url": "pic.twitter.com/o2whuMhxF3"}]}, "conversation_id": "1418216298203340805", "id": "1418216298203340805", "context_annotations": [{"domain": {"id": "10", "name": "Person", "description": "Named people in the world like Nelson Mandela"}, "entity": {"id": "1241108409371332608", "name": "The Bangles"}}, {"domain": {"id": "45", "name": "Brand Vertical", "description": "Top level entities that describe a Brands industry"}, "entity": {"id": "781974596161376261", "name": "Dining"}}, {"domain": {"id": "46", "name": "Brand Category", "description": "Categories within Brand Verticals that narrow down the scope of Brands"}, "entity": {"id": "781974597017083904", "name": "QSR"}}, {"domain": {"id": "47", "name": "Brand", "description": "Brands and Companies"}, "entity": {"id": "10026427301", "name": "Domino's Pizza"}}, {"domain": {"id": "54", "name": "Musician", "description": "A musician in the world, like Adele or Bob Dylan"}, "entity": {"id": "1241108409371332608", "name": "The Bangles"}}, {"domain": {"id": "55", "name": "Music Genre", "description": "A category for a musical style, like Pop, Rock, or Rap"}, "entity": {"id": "806571747730341888", "name": "Rock", "description": "Rock"}}, {"domain": {"id": "55", "name": "Music Genre", "description": "A category for a musical style, like Pop, Rock, or Rap"}, "entity": {"id": "1108100469778345984", "name": "Classic rock", "description": "Classic Rock"}}, {"domain": {"id": "65", "name": "Interests and Hobbies Vertical", "description": "Top level interests and hobbies groupings, like Food or Travel"}, "entity": {"id": "825047692124442624", "name": "Food", "description": "Food"}}, {"domain": {"id": "66", "name": "Interests and Hobbies Category", "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"}, "entity": {"id": "860133563332648963", "name": "Cuisines", "description": "Cuisines"}}, {"domain": {"id": "30", "name": "Entities [Entity Service]", "description": "Entity Service top level domain, every item that is in Entity Service should be in this domain"}, "entity": {"id": "847865239886417922", "name": "Fast food", "description": "Fast food"}}], "lang": "en", "text": "ROCK N ROLL, TAN VIEJO COMO EL TIEMPO.\n\nBill Haley & His Comets- Rock Around The Clock\nStage Dolls- Love cries\nThe Bangles- Eternal Flame\nTNT- 10,000 Lovers\nDereck /Dominos- Layla\nBoston- More Than a Feeling\nThe Allman Brothers Band- Whipping Post\nJethro Tull- Thick as a Brick https://t.co/o2whuMhxF3", "possibly_sensitive": false, "source": "Twitter Web App", "reply_settings": "everyone"}], "includes": {"media": [{"url": "https://pbs.twimg.com/media/E66AjBQVcAUIqWf.png", "height": 350, "width": 350, "type": "photo", "media_key": "3_1418212271789010949"}, {"url": "https://pbs.twimg.com/media/E66Bo2wVkAEJZOs.png", "height": 594, "width": 599, "type": "photo", "media_key": "3_1418213471561289729"}, {"url": "https://pbs.twimg.com/media/E66CyZNUYAQI3yo.jpg", "height": 800, "width": 1200, "type": "photo", "media_key": "3_1418214734940102660"}, {"url": "https://pbs.twimg.com/media/E66DYS3VcAQ_nOo.png", "height": 718, "width": 800, "type": "photo", "media_key": "3_1418215386072313860"}], "users": [{"verified": false, "created_at": "2015-11-06T01:04:09.000Z", "pinned_tweet_id": "1414196689305542656", "description": "Canciones Favoritas= M\u00f6tley Cr\u00fce- Girls, Girls, Girls/ W.A.S.P- Forever Free.\nComida Favoritas= Sancocho+ ensalada de papa con huevo\nLibro= El arte de la guerra", "username": "kranvitter", "name": "mundi", "profile_image_url": "https://pbs.twimg.com/profile_images/1236461717477097472/xgMGgMtw_normal.jpg", "id": "4123358326", "url": "", "public_metrics": {"followers_count": 370, "following_count": 310, "tweet_count": 30951, "listed_count": 1}, "protected": false}]}, "__twarc": {"url": "https://api.twitter.com/2/tweets?expansions=author_id%2Cin_reply_to_user_id%2Creferenced_tweets.id%2Creferenced_tweets.id.author_id%2Centities.mentions.username%2Cattachments.poll_ids%2Cattachments.media_keys%2Cgeo.place_id&user.fields=created_at%2Cdescription%2Centities%2Cid%2Clocation%2Cname%2Cpinned_tweet_id%2Cprofile_image_url%2Cprotected%2Cpublic_metrics%2Curl%2Cusername%2Cverified%2Cwithheld&tweet.fields=attachments%2Cauthor_id%2Ccontext_annotations%2Cconversation_id%2Ccreated_at%2Centities%2Cgeo%2Cid%2Cin_reply_to_user_id%2Clang%2Cpublic_metrics%2Ctext%2Cpossibly_sensitive%2Creferenced_tweets%2Creply_settings%2Csource%2Cwithheld&media.fields=duration_ms%2Cheight%2Cmedia_key%2Cpreview_image_url%2Ctype%2Curl%2Cwidth%2Cpublic_metrics&poll.fields=duration_minutes%2Cend_datetime%2Cid%2Coptions%2Cvoting_status&place.fields=contained_within%2Ccountry%2Ccountry_code%2Cfull_name%2Cgeo%2Cid%2Cname%2Cplace_type&ids=1418216298203340805", "version": "2.3.10", "retrieved_at": "2021-07-26T21:01:05+00:00"}}

Or the “flattened” version with includes, this is equivalent to TwitterAPI _hydrate but twarc does it slightly differently:

echo "1418216298203340805" | twarc2 hydrate | twarc2 flatten
{
  "__twarc": {
    "url": "https://api.twitter.com/2/tweets?expansions=author_id%2Cin_reply_to_user_id%2Creferenced_tweets.id%2Creferenced_tweets.id.author_id%2Centities.mentions.username%2Cattachments.poll_ids%2Cattachments.media_keys%2Cgeo.place_id&user.fields=created_at%2Cdescription%2Centities%2Cid%2Clocation%2Cname%2Cpinned_tweet_id%2Cprofile_image_url%2Cprotected%2Cpublic_metrics%2Curl%2Cusername%2Cverified%2Cwithheld&tweet.fields=attachments%2Cauthor_id%2Ccontext_annotations%2Cconversation_id%2Ccreated_at%2Centities%2Cgeo%2Cid%2Cin_reply_to_user_id%2Clang%2Cpublic_metrics%2Ctext%2Cpossibly_sensitive%2Creferenced_tweets%2Creply_settings%2Csource%2Cwithheld&media.fields=duration_ms%2Cheight%2Cmedia_key%2Cpreview_image_url%2Ctype%2Curl%2Cwidth%2Cpublic_metrics&poll.fields=duration_minutes%2Cend_datetime%2Cid%2Coptions%2Cvoting_status&place.fields=contained_within%2Ccountry%2Ccountry_code%2Cfull_name%2Cgeo%2Cid%2Cname%2Cplace_type&ids=1418216298203340805",
    "version": "2.3.10",
    "retrieved_at": "2021-07-26T21:02:45+00:00"
  },
  "id": "1418216298203340805",
  "public_metrics": {
    "retweet_count": 0,
    "reply_count": 0,
    "like_count": 0,
    "quote_count": 0
  },
  "text": "ROCK N ROLL, TAN VIEJO COMO EL TIEMPO.\n\nBill Haley & His Comets- Rock Around The Clock\nStage Dolls- Love cries\nThe Bangles- Eternal Flame\nTNT- 10,000 Lovers\nDereck /Dominos- Layla\nBoston- More Than a Feeling\nThe Allman Brothers Band- Whipping Post\nJethro Tull- Thick as a Brick https://t.co/o2whuMhxF3",
  "attachments": {
    "media_keys": [
      "3_1418212271789010949",
      "3_1418213471561289729",
      "3_1418214734940102660",
      "3_1418215386072313860"
    ],
    "media": [
      {
        "url": "https://pbs.twimg.com/media/E66AjBQVcAUIqWf.png",
        "height": 350,
        "media_key": "3_1418212271789010949",
        "width": 350,
        "type": "photo"
      },
      {
        "url": "https://pbs.twimg.com/media/E66Bo2wVkAEJZOs.png",
        "height": 594,
        "media_key": "3_1418213471561289729",
        "width": 599,
        "type": "photo"
      },
      {
        "url": "https://pbs.twimg.com/media/E66CyZNUYAQI3yo.jpg",
        "height": 800,
        "media_key": "3_1418214734940102660",
        "width": 1200,
        "type": "photo"
      },
      {
        "url": "https://pbs.twimg.com/media/E66DYS3VcAQ_nOo.png",
        "height": 718,
        "media_key": "3_1418215386072313860",
        "width": 800,
        "type": "photo"
      }
    ]
  },
  "possibly_sensitive": false,
  "reply_settings": "everyone",
  "created_at": "2021-07-22T14:28:09.000Z",
  "lang": "en",
  "context_annotations": [
    {
      "domain": {
        "id": "10",
        "name": "Person",
        "description": "Named people in the world like Nelson Mandela"
      },
      "entity": {
        "id": "1241108409371332608",
        "name": "The Bangles"
      }
    },
    {
      "domain": {
        "id": "45",
        "name": "Brand Vertical",
        "description": "Top level entities that describe a Brands industry"
      },
      "entity": {
        "id": "781974596161376261",
        "name": "Dining"
      }
    },
    {
      "domain": {
        "id": "46",
        "name": "Brand Category",
        "description": "Categories within Brand Verticals that narrow down the scope of Brands"
      },
      "entity": {
        "id": "781974597017083904",
        "name": "QSR"
      }
    },
    {
      "domain": {
        "id": "47",
        "name": "Brand",
        "description": "Brands and Companies"
      },
      "entity": {
        "id": "10026427301",
        "name": "Domino's Pizza"
      }
    },
    {
      "domain": {
        "id": "54",
        "name": "Musician",
        "description": "A musician in the world, like Adele or Bob Dylan"
      },
      "entity": {
        "id": "1241108409371332608",
        "name": "The Bangles"
      }
    },
    {
      "domain": {
        "id": "55",
        "name": "Music Genre",
        "description": "A category for a musical style, like Pop, Rock, or Rap"
      },
      "entity": {
        "id": "806571747730341888",
        "name": "Rock",
        "description": "Rock"
      }
    },
    {
      "domain": {
        "id": "55",
        "name": "Music Genre",
        "description": "A category for a musical style, like Pop, Rock, or Rap"
      },
      "entity": {
        "id": "1108100469778345984",
        "name": "Classic rock",
        "description": "Classic Rock"
      }
    },
    {
      "domain": {
        "id": "65",
        "name": "Interests and Hobbies Vertical",
        "description": "Top level interests and hobbies groupings, like Food or Travel"
      },
      "entity": {
        "id": "825047692124442624",
        "name": "Food",
        "description": "Food"
      }
    },
    {
      "domain": {
        "id": "66",
        "name": "Interests and Hobbies Category",
        "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
      },
      "entity": {
        "id": "860133563332648963",
        "name": "Cuisines",
        "description": "Cuisines"
      }
    },
    {
      "domain": {
        "id": "30",
        "name": "Entities [Entity Service]",
        "description": "Entity Service top level domain, every item that is in Entity Service should be in this domain"
      },
      "entity": {
        "id": "847865239886417922",
        "name": "Fast food",
        "description": "Fast food"
      }
    }
  ],
  "conversation_id": "1418216298203340805",
  "entities": {
    "urls": [
      {
        "start": 282,
        "end": 305,
        "url": "https://t.co/o2whuMhxF3",
        "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1",
        "display_url": "pic.twitter.com/o2whuMhxF3"
      },
      {
        "start": 282,
        "end": 305,
        "url": "https://t.co/o2whuMhxF3",
        "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1",
        "display_url": "pic.twitter.com/o2whuMhxF3"
      },
      {
        "start": 282,
        "end": 305,
        "url": "https://t.co/o2whuMhxF3",
        "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1",
        "display_url": "pic.twitter.com/o2whuMhxF3"
      },
      {
        "start": 282,
        "end": 305,
        "url": "https://t.co/o2whuMhxF3",
        "expanded_url": "https://twitter.com/kranvitter/status/1418216298203340805/photo/1",
        "display_url": "pic.twitter.com/o2whuMhxF3"
      }
    ]
  },
  "source": "Twitter Web App",
  "author_id": "4123358326",
  "author": {
    "username": "kranvitter",
    "description": "Canciones Favoritas= Mötley Crüe- Girls, Girls, Girls/ W.A.S.P- Forever Free.\nComida Favoritas= Sancocho+ ensalada de papa con huevo\nLibro= El arte de la guerra",
    "created_at": "2015-11-06T01:04:09.000Z",
    "profile_image_url": "https://pbs.twimg.com/profile_images/1236461717477097472/xgMGgMtw_normal.jpg",
    "url": "",
    "public_metrics": {
      "followers_count": 370,
      "following_count": 310,
      "tweet_count": 30951,
      "listed_count": 1
    },
    "id": "4123358326",
    "name": "mundi",
    "protected": false,
    "pinned_tweet_id": "1414196689305542656",
    "verified": false
  }
}

It’s worth reporting in issue GitHub - geduldig/TwitterAPI: Minimal python wrapper for Twitter's REST and Streaming APIs in TwitterAPI if it’s not picking up all the media items like that

Thank you very much, Igor!
Already made the post some time ago, but nobody has answered me yet.
Will try to take a look in the code later, to see if i can fix it.
Cheers! :slight_smile:

1 Like