how does a correct robots.txt have to look?
-
i wonder if the robots.txt for my wordpress.com blog is ok and doesnt block any searches?
thats what it looks like:———8<——-
User-agent: IRLbot
Crawl-delay: 3600User-agent: *
Disallow: /next/User-agent: *
Disallow:
———–>8———–perhaps there should be a faq entry on how wordpress.com generates the robots.txt files.
-
The target audience for wordpress.com isn’t supposed to know what robots.txt is, let alone what it looks like ;) Mine is identical, if that helps reassure you. I would imagine it’s the same for all free blogs.
If you’ve selected the option to hide your blog from search engines, WP sticks a ‘noindex, nofollow’ metatag in the header of your theme, rather than amending robots.txt. So if you’re still worried, view source on your blog.
-
Thanks for the answer!
And about target audience: Isn’t it a wonderful thing when the target audience is seeking to aquire some knowledge :)
-
Absolutely. In fact, I think the wordpress.com userbase is a lot smarter than Automattic would like it to be.
-
Searching the forums also helps.
https://en.forums.wordpress.com/topic.php?id=13125&page&replies=17:)
- The topic ‘how does a correct robots.txt have to look?’ is closed to new replies.