‘blocked by robots.txt’ on bing webmaster tools

  • Unknown's avatar
    ekornutiak · Member ·

    Google can index/crawl my page just fine.

    Bing Webmaster Tools gives me this:
    ‘Crawl
    Last crawl attempted
    26 Nov 2023 at 13:51
    Crawl allowed?
    No
    Page Fetch
    Failed
    Indexing allowed?
    Yes
    The inspected URL has been indexed, despite being blocked from crawling by robots.txt. Please make sure that the block is valid as this might prevent us from showing description about the page in the search results.’

    HOW ON EARTH do I fix this without paying $40/month for WordPress and another some odd dollars/month for the plugins??

    WP.com: Yes
    Jetpack: No
    Correct account: Yes

    The blog I need help with is: (visible only to moderators and staff)

  • Unknown's avatar
    faisalahammad · Member ·

    Hi there,

    Good morning. I hope you’re doing great.

    I have tagged your post so the staff can see it. Please wait patiently, someone will get back to you shortly. 😃

    Have a good day!

    Thanks
    Faisal

  • Unknown's avatar
    2020testjj · Member ·

    Hello there, At one point Bing’s robots were actively crawling and scraping content for their LLM. Here is a previous Staff reply on that https://wordpress.com/forums/topic/blocked-by-robots-txt-on-bing-webmaster-tools/

    However, since then, WordPress.com introduced a control in our site’s Privacy settings. You can learn more about that here: https://wordpress.com/blog/2024/02/27/more-control-over-the-content-you-share/

    —

    Aside/For anyone who might be adding unrelated tags in an effort to follow a topic, there are tools built in to each forum topic to allow you to subscribe or favorite a topic.

    After doing so, you can also find these topics in your forum profile by clicking on your username at the top right of the forum pages, Welcome, yourusername. In any case, all topics a user has participated in is shown in their forum profile. Cheers!

  • staff-sjclaen · Staff ·

    To clarify, it sounds like you want the Bing crawlers to go through crawling the site again, is that right? Have you tried to initiate to try to crawl the site again? From your provided information, it looks like the last time was 26 Nov 2023. I would suggest reviewing their help resources to see if they can crawl the site again.

  • The topic ‘‘blocked by robots.txt’ on bing webmaster tools’ is closed to new replies.