Bad robots.text when using public-api.wordpress.com
-
Hey Guys,
Thanks for creating an REST api to allow access of wordpress.com data.https://www.sweetlightstudios.com is a React Redux app that uses the WordPress API to access sweetlightstudiosen.wordpress.com.
This works AOK within the APP, but when google tries to crawl, it gets a robots.txt that tells them not to.
For example:
If I tell the Google Search Console to “Fetch as Google”, https://www.sweetlightstudios.com/blog/ it sees the header and footer but no blog information.
One of the AJAX calls is:
https://public-api.wordpress.com/rest/v1.1/sites/sweetlightstudiosen.wordpress.com/posts/?page=1
The page and the call both work AOK in a browser, but when I tell the google console to “Fetch as Google”, the AJAX is blocked by a robots.txt at: https://public-api.wordpress.com/robots.txt that says:User-agent: *
Disallow: /So, the blog part of the site isn’t getting crawled and I would like it to be crawled along with the rest of the site.
I checked the Reading Settings and they are set to allow search engines to index the site.
Any ideas, work arounds or fix for the robots.txt Issue???TY my Friends
The blog I need help with is: (visible only to logged in users)
-
Let’s ask Staff to assist you with sorting this out. I will type modlook into the sidebar tags on this thread for Staff help. How do I get a Moderator/Staff reply for my question? https://en.support.wordpress.com/getting-help-in-the-forums/#how-do-i-get-a-moderatorstaff-reply-for-my-question Please subscribe to this thread so you are notified when they respond and be patient while waiting. To subscribe look in the sidebar of this thread, find the subscribe to topics link and click it.
-
Hello there, For help with API issues please head over to the Developer’s site at https://developer.wordpress.com/contact/
- The topic ‘Bad robots.text when using public-api.wordpress.com’ is closed to new replies.
WordPress.com Forums 