Crawl Delay

• 

  bigbear007 · Member · Sep 18, 2016 at 8:38 am

  • Copy link
  • Add topic to favorites

  Hello, how do I change the crawl delay for my website?

  my sitemap is:

  # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
  # Please see https://developer.wordpress.com/docs/firehose/ for more details.

  Sitemap: https://mysayar.org/sitemap.xml
  Sitemap: https://mysayar.org/news-sitemap.xml

  User-agent: *
  Disallow: /wp-admin/
  Allow: /wp-admin/admin-ajax.php
  Disallow: /wp-login.php
  Disallow: /activate/ # har har
  Disallow: /cgi-bin/ # MT refugees
  Disallow: /mshots/v1/
  Disallow: /next/
  Disallow: /public.api/

  User-agent: IRLbot
  Crawl-delay: 3600

  Thank you!!

  The blog I need help with is: (visible only to logged in users)