edit robot.txt
-
I am trying to let the following robot crawl my website (http://crystalmixing.com)
(bot IP: 46.229.173.67)I was told that I need to make sure that the bot is whitelisted and to add the following 2 lines to my robot.txt file (I was told by WordPress.com that I cannot edit the robot.txt though since I am only using wordpress.com)
I have now spoke to all parties involved with my hosting and domain and they are all saying that they cannot do anything about it. So I am pretty confused as to what to do now as I am sure that there must be an answer to my problem.
I am pointing my website towards CloudFlare in which I have ensured to allow the robots IP through.
Please can you help me with my problem as I am sure there must be a fix to this small problem.
Below is the last email that I sent and received from WildWestDomains who my domain is registered to.Support Staff Response
Dear Graeme,Thank you for contacting our Support team.
Please be aware that Wild West Domains is a parent Registrar for multiple companies, such as WordPress.com, who offer Domain Registration Services. To clarify, Wild West Domains would not have access to any accounts set up through WordPress.com. You will need to contact the correct company WordPress at http://en.support.wordpress.com/contact/ for assistance with any domains registered through their service.
Please contact us if you need further assistance.
Regards,
John G
Support
Wild West Domains
Customer Inquiry
Hi there. I am trying to let SemRush robot (bot IP: 46.229.173.67) crawl my domain but I am getting a 502 error. I have been told that I need to add the following 2 lines to my robot.txt file. User-agent: SemrushBot-SADisallow: I have attached a screen shot which I hope will help explain the issue. Because I am using WordPress.com then I cannot change the robot.txt file. WordPress told me to speak with my domain host, so here I am. :) Please can you allow the robot from SemRush to crawl my website? If you could, can you please give me detailed instructions on how to allow certain robots to crawl my domain? Kindest Regards, Graeme – Head Audio Engineer at Crystal Mixing and Mastering — For Professional yet affordable Mixing and Mastering go to CrystalMixing.comThank you for your time.
GraemeThe blog I need help with is: (visible only to logged in users)
-
Hi there,
The problem is your site is still hosted on WordPress.com, which means its still not possible to add the lines to the robot.txt file.
You did point your domain to the CloudFare name servers, but then you used DNS records to point your domain back to your WordPress.com site. That means your site is still hosted here, and the restrictions on editing the Robot.txt file are still in play.
In order to modify the file, you’d need to create a new WordPress site that is hosted somewhere else, move your content to the new site and then point your domain to the new name servers. Here’s more info on moving a WordPress.com site to a self hosted WordPress install.
https://en.support.wordpress.com/moving-a-blog/#moving-to-wordpress-org
Let me know if you have any more questions about that.
-
Much appreciated for the reply and the great info lizthefair.
Unfortunately I do not have the funds to get hosting at the moment.I don’t understand why WordPress.com allows google bots to crawl my webpage but not this certain bot.
There must be a way round this without having to get hosting of my own.
I thought it was impossible to get statistics from Google Analytics while us WordPress.com but I managed to find an easy work around by pointing my domain to CloudFlare.I bet there is an easy way of getting round this small problem without getting my own hosting.
A Chocolate Easter Egg is up for grabs for the person who cracks this little puzzle!! -
I’m afraid I don’t have any ideas for you. You might try talking to the support team at CloudFlare or even posting to our public forums to see if anyone has found an unofficial workaround that works.
-
- The topic ‘edit robot.txt’ is closed to new replies.
WordPress.com Forums 