https://dev.twitter.com/docs/cards/getting-started
Twitter uses the User-Agent of Twitterbot (with version, such as *Twitterbot/1.0), which can be used to create an exception in your robots.txt file. (Is *Twitterbot/1.0 most up to date version? Or is it(Twitterbot) switches as API1.0 moved up to version API1.1?)
Robot.txt
For example, here is a robots.txt which disallows crawling for all robots except Twitter’s fetcher:
User-agent: Twitterbot
Disallow:
User-agent: *
Disallow: /
another example, which specifies which directories are allowed to be crawled by Twitterbot (in this case, disallowing all except the images and archives directories)
User-agent: Twitterbot
Disallow: *
Allow: /images
Allow: /archives
In both above examples, it seems Asterisks(*) mark is used as wild card but not in set as .
Your server’s robots.txt file must be saved as plain text with ASCII character encoding. To verify this, you can run the following command:
$ file -I robots.txt
robots.txt: text/plain; charset=us-ascii
Despite the example given I’m not sure how to do this from WordPress. Do I have to get in to PHP?
What is in my Head.php right now
<?php
/*
* Print the tag based on what is being viewed.
*/
global $page, $paged;
wp_title( '|', true, 'right' );
// Add the blog name.
bloginfo( 'name' );
// Add the blog description for the home/front page.
$site_description = get_bloginfo( 'description', 'display' );
if ( $site_description && ( is_home() || is_front_page() ) )
echo " | $site_description";
// Add a page number if necessary:
if ( $paged >= 2 || $page >= 2 )
echo ' | ' . sprintf( __( 'Page %s', 'twentyeleven' ), max( $paged, $page ) );
?></title>
<?php
/* We add some JavaScript to pages with the comment form
* to support sites with threaded comments (when in use).
*/
if ( is_singular() && get_option( 'thread_comments' ) )
wp_enqueue_script( 'comment-reply' );
/* Always have wp_head() just before the closing </head>
* tag of your theme, or you will break many plugins, which
* generally use this hook to add elements to <head> such
* as styles, scripts, and meta tags.
*/
wp_head();
?>
from WordPress Head.php
URL http://jrestaurant.resellrightsnetwork.com/w/