robot.txt

• 

  graceelizabethbiz · Member · Mar 22, 2024 at 3:08 am

  • Copy link
  • Add topic to favorites

  Hi, my free plan website is not showing up on Google, because of the robot.txt automatically built onto my site by WordPress host. If I upgrade my plan to the personal plan, will WordPress change the robot.txt and allow Google search engine to crawl my site?

  Thank you for your time.

  WP.com: Yes
  Jetpack: No
  Correct account: Yes

  The blog I need help with is: (visible only to logged in users)

• 

  justjennifer · Member · Mar 22, 2024 at 7:40 am

  • Copy link

  Hi there, upgrading does not affect your robots.txt. As long as your site is set to Public in your General Settings dashboard>Privacy then search engines can crawl your site content. I don’t see anything in your current robots.txt that would block Google or any other search engine.

  Can you tell us why you think Google is being blocked?

• 

  myytabcchannel · Member · Mar 22, 2024 at 11:06 am

  • Copy link

  I have same issue, my website is also not showing up on Google. I think have same problem.

  Thank you!!!

• 

  graceelizabethbiz · Member · Mar 22, 2024 at 2:46 pm

  • Copy link

  Also, my site settings are already set to public, and I do not have the field checked that says “Discourage search engines from indexing this site”.
  
  Another free site that I have been working on for a client is having the same issue, their website is not showing up in Google search results either, due to the same reason. Her on-page SEO is good, but two of her sites are blocked from appearing on Google search due to robots.txt.

• 

  graceelizabethbiz · Member · Mar 22, 2024 at 2:52 pm

  • Copy link

  Niel Patel’s SEO Analyzer Ubersuggest https://app.neilpatel.com/en/seo_analyzer/site_audit?domain=kitiarascrochets.wordpress.com&mode=domain&view=details&issue=page_allowed tells me that it’s my robots.txt that is blocking my sites from appearing on Google search.

• 

  justjennifer · Member · Mar 22, 2024 at 7:21 pm

  • Copy link

  Thanks for that link. So what you see as “blocked” in that report are internal dashboard pages for the Site Editor. There is no reason that those internal dashboard pages should appear in search results.

  Again, looking at the robots.txt file for kitiarascrochets.wordpress.com I do not see anything there that would block search engines from crawling your site content.

  One thing I did notice is that kitiarascrochets.wordpress.com (and it always helps to mention up front which WordPress.com site you’re referring to when you post for help) is a static site and currently there is no “News” “Blog” “Events” or other page that generates a feed and pings new content to search engines.

  Is your site very new? Did you submit the XML sitemap to search engines or verify your site with search engines? Here is a guide for WordPress.com sites to getting found in search engines and a number of suggestions that might help.

• 

  graceelizabethbiz · Member · Mar 22, 2024 at 7:36 pm

  • Copy link

  I’ll check those links out. Thank you very much!