Robots.txt not working correctly
-
Hello,
I have had an issue with my blog. I recently opened a Google Webmaster account. Everything seemed to be going smoothly. On my wordpresss.com dashboard I went to Tools>Available Tools and verified my site. I then went to Settings>Privacy and clicked ‘allow search engines to index this site’.
However, my google webmaster account keeps saying “URL restricted by robots.txt” I have searched dozens of sites and have not found a solution. I have also given it a couple of days to maybe ‘work itself out’.
It still says ‘Disallow’ at http://foleysfolly.com/robots.txt
Anyone have a solution?
The blog I need help with is: (visible only to logged in users)
-
http://ismyblogworking.com/foleysfolly.com
Shows your robots.txt as allowing search engines in – the syntax as I remember is a bit odd – if Google was locked out of an area there would be a directory showing
-
Thank you for your reply.
I did check ismyblogworking and did see my sites was working properly, However, I was unsure as to why It keeps showing up in my webmaster account as disallowed.
If I’m in my webmaster account and I go to Site Configuration>Crawler Access and run a test, it gives the following error:
Disallowed:Allowed by line 2: Disallow:
Detected as a directory; specific files may have different restrictionsShould I trust ismyblogworking over my webmaster account?
-
Your robots.txt file is now updated and looks way different than it did a while ago – forgot the time lag factor – it can take a while for the file to be updated.
-
Wow, thats pretty amazing the robots.txt file changed within the 30 minutes I posted this lol. Thank you for the help, although I ran another test on webmaster and it still says its disallowed. As always, these things seem to work themselves out.
Thanks again.
-
- The topic ‘Robots.txt not working correctly’ is closed to new replies.
WordPress.com Forums 