Google crawler access

• 

  citymove · Member · Jan 2, 2012 at 2:47 am

  • Copy link
  • Add topic to favorites

  I’ve inadvertently created a robots.txt file for my blog (on Google webmaster tools). I didn’t realize until now that this file is restricting access to my site and the majority of my posts show up as 404 (not found). Suggestions? Thanks.

  The blog I need help with is: (visible only to logged in users)

• 

  thesacredpath · Member · Jan 2, 2012 at 2:53 am

  • Copy link

  You need to deactivate or delete that robots.txt file at google webmaster tools. Your site here already has one created automatically that gives Google and the other search engines access to all the stuff they need access to.

• 

  citymove · Member · Jan 2, 2012 at 3:24 am

  • Copy link

  Yes, I was trying to to just that – deactivate or delete that file – but every time I try, the original one pops back up. Do you know of a simple way to remove an existing file and replace it with:

  User-agent: *
  Allow: /

  or leave it blank, I’d be happy with either one.

• 

  thesacredpath · Member · Jan 2, 2012 at 3:35 am

  • Copy link

  Your actual robots.txt file here is located at http://citymove.wordpress.com/robots.txt .

  Try adding that as your robots.txt. If you view it, you will see a lot more stuff in it and that is to keep the search engines honest and to keep them from tying up the wordpress servers by camping out for long periods.

• 

  citymove · Member · Jan 2, 2012 at 3:46 am

  • Copy link

  The robots.txt file you suggest is the one already in place. Here is the problem, when I view the crawl errors on WMT I see this (one example);
  http://citymove.wordpress.com/2010/12/22/eco-friendly-plastic-moving-boxes-are-b-s/
  404 (Not found)
  unavailable
  Dec 24, 2011

  * there are 21 more.

  If you click on this link it takes you to my site and indicates “We can’t find what you are looking for”.
  It seems as though the robots.txt file is restricting access to my site hence my desire to remove it or modify it.

• 

  thesacredpath · Member · Jan 2, 2012 at 4:13 am

  • Copy link

  You have only one post in December of 2010: http://citymove.wordpress.com/2010/12/ .

  Did you delete some posts, or possibly change the date or title one some of them?

  You need to go through that list at google one by one and compare it to your posts. I think you will find that either those posts are not there or that the date or title/URL was changed.

  I looked a couple months both directions from Dec. 2010 and did not see that post on your site.

• 

  citymove · Member · Jan 2, 2012 at 4:42 am

  • Copy link

  Thanks for having a look, I’ll go check.

• 

  thesacredpath · Member · Jan 2, 2012 at 4:45 am

  • Copy link

  You are welcome.