Twitter uses the User-Agent of Twitterbot (with version, such as *Twitterbot/1.0), which can be used to create an exception in your robots.txt file. (Is *Twitterbot/1.0 most up to date version? Or is it(Twitterbot) switches as API1.0 moved up to version API1.1?)
For example, here is a robots.txt which disallows crawling for all robots except Twitter’s fetcher:
another example, which specifies which directories are allowed to be crawled by Twitterbot (in this case, disallowing all except the images and archives directories)
In both above examples, it seems Asterisks(*) mark is used as wild card but not in set as .
Your server’s robots.txt file must be saved as plain text with ASCII character encoding. To verify this, you can run the following command:
$ file -I robots.txt
robots.txt: text/plain; charset=us-ascii
Despite the example given I’m not sure how to do this from WordPress. Do I have to get in to PHP?
What is in my Head.php right now
* Print the tag based on what is being viewed.
global $page, $paged;
wp_title( '|', true, 'right' );
// Add the blog name.
bloginfo( 'name' );
// Add the blog description for the home/front page.
$site_description = get_bloginfo( 'description', 'display' );
if ( $site_description && ( is_home() || is_front_page() ) )
echo " | $site_description";
// Add a page number if necessary:
if ( $paged >= 2 || $page >= 2 )
echo ' | ' . sprintf( __( 'Page %s', 'twentyeleven' ), max( $paged, $page ) );
* to support sites with threaded comments (when in use).
if ( is_singular() && get_option( 'thread_comments' ) )
wp_enqueue_script( 'comment-reply' );
/* Always have wp_head() just before the closing </head>
* tag of your theme, or you will break many plugins, which
* generally use this hook to add elements to <head> such
* as styles, scripts, and meta tags.
from WordPress Head.php