How can I make robots.txt allow Bingbot? (WordPress.com Free)

  • Unknown's avatar
    27gc · Member ·

    My robots.txt file has been blocking Bing Webmasters from indexing my site because it blocks Bingbot.

    I researched online and found that the robots.txt file is created by WordPress automatically and can only be changed on the free plan through Privacy settings.

    However, my Privacy settings are currently set to “Public,” which means that it should be allowing user agents.

    I have already verified my website through Bing Webmasters (as shown in this article) and it should not be blocking Bingbot. Is this WordPress article outdated? Are there any new methods for allowing Bing Webmasters to index a Free WordPress.com site?

    The blog I need help with is: (visible only to logged in users)

  • Unknown's avatar
    dragstor · Member ·

    Hi there! 👋🏼

    Recently, we discovered that Microsoft is using its generic Bingbot crawler to scrape sites, which is currently having detrimental effects. We can’t go into detail about the outcome of this scraping for security and privacy purposes (for now).

    They have not yet documented a way to block the scraping behavior, so for the moment, we have blocked Bingbot from indexing Simple sites via robots.txt directives, and this is something we’re monitoring closely.

    I hope this helps to provide some information!

  • Unknown's avatar
    27gc · Member ·

    Thank you, this provides me with a lot of new information.

    Will Bingbot be allowed by the default WordPress.com robots.txt in the near future? If it is, will my robots.txt automatically update?

  • jmills39 · Staff ·

    Hello there,

    We’re unable to provide an update on if and when that might happen, but if it did, then you wouldn’t need to make any changes on your end.

    Please feel free to reply to this message if you have any other questions and have a great day!

  • Unknown's avatar
    27gc · Member ·

    Thank you, you too.

  • Unknown's avatar
    pyodizer · Member ·

    I’m not at all concerned about the scraping of my website. I am aware that it can possibly affect my site’s bandwidth. But BingBot itself never caused my site any headaches while it was allowed.

    I petition WordPress.com staff to please allow BingBot to crawl our sites again. Disallowing BingBot not only affects personal websites, but also those of us who have paid content when we can’t be indexed or have our indexing updated…

  • Unknown's avatar
    staartmees · Member ·

    @revjosephdsmithdd do read the answer dragstor gave – https://wordpress.com/forums/topic/how-can-i-make-robots-txt-allow-bingbot-wordpress-com-free/#post-4006983.”Microsoft is using its generic Bingbot crawler to scrape sites, which is currently having detrimental effects. We can’t go into detail about the outcome of this scraping for security and privacy purposes (for now).”

  • The topic ‘How can I make robots.txt allow Bingbot? (WordPress.com Free)’ is closed to new replies.