Problem with Robots.txt

• 

  stuspence · Member · Jun 3, 2015 at 9:14 am

  • Copy link
  • Add topic to favorites

  Hi Guys,

  I am running a wordpress.com site that received the following message from Google:

  WHEN a MaN SNaPS
  whenamansnaps.com/
  A description for this result is not available because of this site’s robots.txt – learn more.

  The txt file itself reads:

  # This file was generated on Mon, 18 May 2015 08:23:45 +0000
  # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
  # Please see http://en.wordpress.com/firehose/ for more details.

  Sitemap: http://whenamansnaps.com/sitemap.xml
  Sitemap: http://whenamansnaps.com/news-sitemap.xml

  User-agent: IRLbot
  Crawl-delay: 3600

  User-agent: *
  Disallow: /next/

  User-agent: *
  Disallow: /mshots/v1/

  # har har
  User-agent: *
  Disallow: /activate/

  User-agent: *
  Disallow: /public.api/

  # MT refugees
  User-agent: *
  Disallow: /cgi-bin/

  User-agent: *
  Disallow: /wp-login.php

  User-agent: *
  Disallow: /wp-admin/

  Please let me know how to fix this.

  The site address is:
  http://whenamansnaps.com

  The blog I need help with is: (visible only to logged in users)

• 

  simplysaru · Staff · Jun 3, 2015 at 12:42 pm

  • Copy link

  Hi,
  Could please check your indexing settings?

  At Settings -> Reading in your blog’s Dashboard, you might have selected “Discourage search engines from indexing this site.”
  Select “Allow search engines to index this site” instead.

  If that doesn’t fix it, remove your site from Google Webmaster tools and add it back again. Hopefully that should fixed it.

• 

  stuspence · Member · Jun 3, 2015 at 11:14 pm

  • Copy link

  Thanks for your advice. “Allow search engines…” was already selected, so I’ll try your second suggestion.

  Cheers