Blog blocked by robots.txt Need Help

• 

  interiordesignsanddecor · Member · May 9, 2012 at 6:32 pm

  • Copy link
  • Add topic to favorites

  I updated my blog to domain .net and blog is being blocked by robots.txt , why?

  The blog I need help with is: (visible only to logged in users)

• 

  bryanvillarin · Staff · May 9, 2012 at 11:52 pm

  • Copy link

  Hi,

  What’s your blog URL?

  Where are you seeing that robots.txt is being blocked? Can you reply with the error message that you’re seeing?

  You can configure your blog privacy settings in Settings -> Privacy from your blog dashboard, and you’ll want it to be set to “Allow search engines to index this site”.

  WordPress.com automatically sends notifications to Google with every post and page update. The search engines can take 4-6 weeks or more to add any new sites. Here’s more information about WordPress.com and Search Engines.

• 

  interiordesignsanddecor · Member · May 10, 2012 at 2:12 am

  • Copy link

  Hi Bryan
  Here’s my URL
  http://whitelineninteriors.net

  Google analytics – (Crawls error) and Blocked URL says that it hasn’t been able to crawls because of the robots.txt

  Also the server connectivity blocked and robots.txt fetch inaccesable
  I have the privacy settings to allow search engines to index site.

  thanks for your help.

• 

  motre · Member · May 10, 2012 at 2:32 am

  • Copy link

  When I fetch http://whitelineninteriors.net/robots.txt myself, I see a normal looking robots.txt file that is only blocking things that should be blocked. I also accessed http://whitelineninteriors.wordpress.com/robots.txt and it is the same except that it also blocks the /admin/ URLs (no reason for Google to crawl any page like that which it has no access to anyway).

  What do you mean by “the server connectivity blocked”? If you are blocked from the server, then you would not be able to access anything in your page. Can you explain what you see happening with each specific URL you try?

• 

  2050ruedessources · Member · May 24, 2012 at 2:15 pm

  • Copy link

  Would’nt be the following instructions blocking the crawl on keywords ?
  “User-agent: *
  Disallow: /related-tags.php”

  I have the same problem here on http://2050ruedessources.net/robots.txt
  No trafic on keywords :(
  Can’t manage to fix it !

• 

  cadolphin · Member · May 27, 2012 at 3:51 am

  • Copy link

  I’m also interested in how to fix this issue. I have the same problem: http://www.earth4energysolarpowerreview.com/robots.txt

  I’ve double checked all the privacy settings to make sure that “Allow search engines to index this site” has been checked. I’ve also checked all the plug-ins to see if any are blocking access.

  I can’t find a robots.txt file in the root directory to make a change.

  Help would be most appreciated.

• 

  auxclass · Member · May 27, 2012 at 4:03 am

  • Copy link

  @cadolphin

  The site you are asking about does not seem to be hosted on WordPress.COM so you need to make friends over at WordPress.ORG the keepers of the software you are using.

  https://en.forums.wordpress.com/topic/7-things-to-know-before-posting-in-wordpresscom-forums?replies=1

  This site is for support of sites hosted on WordPress.COM. You should address your questions to WordPress.ORG the keepers of the software you are using: http://wordpress.org/support/

  For more on the difference: http://support.wordpress.com/com-vs-org/

• 

  cadolphin · Member · May 27, 2012 at 4:12 am

  • Copy link

  Sorry. I didn’t realize that they were different. My apologies.

• 

  auxclass · Member · May 27, 2012 at 5:32 am

  • Copy link

  It’s OK – good luck – some days 30% of the traffic here is WordPress.ORG installs.