robot.txt disallows search engines

• 

  vane7m · Member · Oct 11, 2015 at 1:31 am

  • Copy link
  • Add topic to favorites

  Hi

  I’m a newbie and this world and I had to face the robot.txt vs Google search engine.

  I tried going to dashboard – settings – reading – “Allow search engines to index this site”, but the same problem persists. Please help?

  # This file was generated on Sun, 11 Oct 2015 01:31:23 +0000
  # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
  # Please see http://en.wordpress.com/firehose/ for more details.

  Sitemap: https://travellingclassynista.wordpress.com/sitemap.xml
  Sitemap: https://travellingclassynista.wordpress.com/news-sitemap.xml

  User-agent: IRLbot
  Crawl-delay: 3600

  User-agent: *
  Disallow: /next/

  User-agent: *
  Disallow: /mshots/v1/

  # har har
  User-agent: *
  Disallow: /activate/

  User-agent: *
  Disallow: /public.api/

  # MT refugees
  User-agent: *
  Disallow: /cgi-bin/

  User-agent: *
  Disallow: /wp-login.php

  User-agent: *
  Disallow: /wp-admin/

  The blog I need help with is: (visible only to logged in users)

• 

  auxclass · Member · Oct 11, 2015 at 2:54 am

  • Copy link

  Search engines can index your site

  http://ismyblogworking.com/travellingclassynista.wordpress.com

  What you see is normal – it is good practice to keep search engines out of areas of your site that have no content worth reading (wp-admin for instance)

• 

  vane7m · Member · Oct 11, 2015 at 9:58 am

  • Copy link

  Thank you :)

• 

  auxclass · Member · Oct 11, 2015 at 3:10 pm

  • Copy link

  YOu be welcome & good luck