AI Crawlers Visibility, Analytics & Control

AI Crawlers Visibility, Analytics & Control is a WordPress plugin that helps site owners see how AI systems crawl their content, control access with robots.txt and PHP blocking, and keep a clear audit trail of bot traffic.

As AI search engines, chatbots, and training crawlers visit your site more often, you need visibility into who is accessing your pages and whether that access aligns with your content strategy. This plugin gives you a dedicated AI crawler analytics dashboard and practical bot control tools without replacing your existing SEO or security plugins.

Why monitor AI crawlers on WordPress?

AI companies use automated crawlers to index, train on, or retrieve website content. Common examples include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Bytespider (ByteDance), CCBot (Common Crawl), and many others.

With this plugin you can:

• See which AI bots visit your site and which pages they request.
• Compare traffic across date ranges and spot trends over time.
• Decide which crawlers to allow, discourage, or block.
• Keep logs for compliance, debugging, and content licensing decisions.

Key features

AI crawler analytics

• Live dashboard with total requests, unique crawlers, pages crawled, and blocked attempts.
• Charts for requests over time and traffic by crawler category.
• Top crawlers and most-crawled pages tables with period comparison.
• Recent activity log with filters for crawler, HTTP status, and content type.
• Blocked-attempt logging when PHP blocking is enabled.

68+ built-in AI crawlers

• Pre-configured detection for major AI, search, and scraper bots.
• Category labels (training, search, assistant, scraper, and more).
• Per-crawler info with user-agent pattern details.
• Enable or disable monitoring for individual crawlers.

Custom crawler detection

• Add your own user-agent patterns for bots not in the built-in list.
• Assign company name and category for consistent reporting.

robots.txt control for AI bots

• Add Disallow rules for selected AI crawlers.
• Rules are appended to your WordPress robots.txt output.
• Live preview before you publish changes.

PHP blocking (403)

• Hard-block selected AI crawlers before WordPress serves a response.
• Blocked requests are logged in the analytics dashboard.
• Works alongside robots.txt for stronger enforcement when needed.

Data management

• Configurable log retention (7 days to 1 year, or keep all data).
• Optional delete-all-data action from the admin.
• Delete plugin data on uninstall (optional setting).

Clean admin experience

• WordPress admin menu: AI Crawlers
• Analytics and AI Crawlers tabs with grouped side navigation.
• Built for site administrators — no code required.

Who is this plugin for?

• Bloggers and publishers who want to know when AI crawlers access their articles.
• Business and marketing sites tracking AI bot traffic to key landing pages.
• Developers and agencies managing crawler policy across client WordPress sites.
• Site owners exploring robots.txt and PHP options before broader AI visibility strategy.

Free vs premium

The free version includes crawler analytics, monitoring, robots.txt rules, PHP blocking, custom crawlers, and data retention controls. Premium features (AI Visibility Score, llms.txt, JSON-LD, email alerts, exports, and MCP integrations) are planned for a future release.