WordPress Crawler BOT is killing my website

  • Unknown's avatar
    doyleri · Member ·

    Why is wordpress.com crawling my site so much? It is causing my site to fail or make miserably slow. I get hundreds and hundreds of crawls per day. What is going on? Datadome, I just installed and I already of intense scraping of 44 requests in the last 20 minutes. Before I block this crap, I need answers why is WordPress killing their own platform?

  • kokkieh · Staff ·

    Hi there,

    WordPress.com doesn’t crawl sites, period. We’re not a search engine, so there’s no reason for us to do that.

    Can you tell us the URL of the site in question, and provide more details on exactly what you’re seeing, and where?

  • Unknown's avatar
    doyleri · Member ·

    After reading more about the cron jobs it sounds like WP will run system cron jobs on every page load someone lands on. Is this true?

    https://www.xyz.com/wp-cron.php?doing_wp_cron=1622757873.2993180751800537109375

    Datadome, a bot analyzer, shows WordPress.com as a bad bot. It is my highest server drain. I have had over 279 hits in the last 12 hours for this URL. It shows: IP 35.208.49.171 and AS = Google. Which I do not understand what Google has to do with it.

    I need a better understanding on this drain for the CRON system and Alt ways to run a true scheduled CRON instead of running any page load.

  • Unknown's avatar
    helper-heroponriki · Member ·

    Hi there. That doesn’t seem to be a valid URL as it redirects to https://gen.xyz/ which is a domain registrar. If that is not your site, please provide your actual site URL so we can take a closer look.

    Additionally, the IP address 35.208.49.171 is assigned to Google Cloud as noted here:

    https://www.ip-adress.com/ip-address/ipv4/35.208.49.171

    The WordPress software doesn’t have an IP address because it’s not a hosting provider. WordPress.com does, but for sites hosted here at WordPress.com and would reflect IP addresses owned by Automattic (our parent company).

    Since that IP is not one of ours, you should reach out to your host for help with this issue. Since they’re hosting your site’s files, they also have access to those files and are the best ones to help you with this issue.

  • The topic ‘WordPress Crawler BOT is killing my website’ is closed to new replies.