how does a correct robots.txt have to look?

• 

  dontstop · Member · Nov 3, 2007 at 7:50 pm

  • Copy link
  • Add topic to favorites

  i wonder if the robots.txt for my wordpress.com blog is ok and doesnt block any searches?
  thats what it looks like:

  ———8<——-
  User-agent: IRLbot
  Crawl-delay: 3600

  User-agent: *
  Disallow: /next/

  User-agent: *
  Disallow:
  ———–>8———–

  perhaps there should be a faq entry on how wordpress.com generates the robots.txt files.

• 

  wank · Member · Nov 4, 2007 at 10:01 pm

  • Copy link

  The target audience for wordpress.com isn’t supposed to know what robots.txt is, let alone what it looks like ;) Mine is identical, if that helps reassure you. I would imagine it’s the same for all free blogs.

  If you’ve selected the option to hide your blog from search engines, WP sticks a ‘noindex, nofollow’ metatag in the header of your theme, rather than amending robots.txt. So if you’re still worried, view source on your blog.

• 

  dontstop · Member · Nov 5, 2007 at 2:30 pm

  • Copy link

  Thanks for the answer!

  And about target audience: Isn’t it a wonderful thing when the target audience is seeking to aquire some knowledge :)

• 

  wank · Member · Nov 5, 2007 at 4:23 pm

  • Copy link

  Absolutely. In fact, I think the wordpress.com userbase is a lot smarter than Automattic would like it to be.

• 

  justjennifer · Member · Nov 5, 2007 at 8:27 pm

  • Copy link

  Searching the forums also helps.
  https://en.forums.wordpress.com/topic.php?id=13125&page&replies=17

  :)