Robot.txt is preventing Google from finding my site

• 

  lisacastelly · Member · Jun 9, 2015 at 1:41 pm

  • Copy link
  • Add topic to favorites

  My site has been public for more than 2 months now, but I discovered yesterday that Google won’t find it, unless I type the entire URL and with a description saying it’s the fault of robot.txt.

  I did allow the search engines to find the site, and it’s not set as private.
  I registered on google master tools, but it doesn’t seem to make any difference …

  Here is what appears when checking on the bot :

  User-agent: IRLbot
  Crawl-delay: 3600

  User-agent: *
  Disallow: /next/

  User-agent: *
  Disallow: /mshots/v1/

  # har har
  User-agent: *
  Disallow: /activate/

  User-agent: *
  Disallow: /public.api/

  # MT refugees
  User-agent: *
  Disallow: /cgi-bin/

  User-agent: *
  Disallow: /wp-login.php

  User-agent: *
  Disallow: /wp-admin/

  Could the staff do something ?

  Thank you !

  The blog I need help with is: (visible only to logged in users)

• 

  galois · Member · Jun 9, 2015 at 1:56 pm

  • Copy link

  Maybe this thread can help you:
  https://en.forums.wordpress.com/topic/a-description-for-this-result-is-not-available-because-of-this-sites-robotstxt-4?replies=16#post-2369893