google won't crawl

• 

  acupunctuurpraktijk · Member · May 27, 2013 at 8:54 am

  • Copy link
  • Add topic to favorites

  Hi guys,

  In google webmaster I get the message that google can’t crawl my website due to the robot.txt.

  In my settings in wordpress I changed the setting to allow crawling last week but google still seems to think someting is wrong. Here is the mesage I get in webmaster (translated from Dutch):

  The url http://www.acupunctuurbaidu.nl
  Blocked by line three: Disallow: /
  Recognised as a directory; specific files might have different restrictions

  The robot.txt is:

  # This file was generated on Wed, 22 May 2013 21:55:52 +0000
  # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
  # Please see http://en.wordpress.com/firehose/ for more details.

  Sitemap: http://acupunctuurbaidu.nl/sitemap.xml
  Sitemap: http://acupunctuurbaidu.nl/news-sitemap.xml

  User-agent: IRLbot
  Crawl-delay: 3600

  User-agent: *
  Disallow: /next/

  User-agent: *
  Disallow: /mshots/v1/

  # har har
  User-agent: *
  Disallow: /activate/

  User-agent: *
  Disallow: /wp-login.php

  User-agent: *
  Disallow: /signup/

  User-agent: *
  Disallow: /related-tags.php

  User-agent: *
  Disallow: /public-api/

  # MT refugees
  User-agent: *
  Disallow: /cgi-bin/

  User-agent: *
  Disallow: /wp-admin/
  Disallow: /wp-includes/
  ——-

  Any ideas at why google can’t crawl my site?

  Daniel

  The blog I need help with is: (visible only to logged in users)

• 

  timethief · Member · May 27, 2013 at 8:38 pm

  • Copy link

  See here please > https://en.forums.wordpress.com/topic/blocked-urls-robotstxt-google-wembaster-reports?replies=6#post-1059738