No easy way to deal with this unless you have a plugin for your CMS that can make intelligent choices about what kind of card you need. You can only have one card type per URL / page.
For a Player card, you will need twitter:card content="player" and not a Vimeo URL, though.