FocusWeb AI Crawler Monitor

FocusWeb AI Crawler Monitor is an observation-only plugin that helps you track and understand how AI crawlers interact with your WordPress site, and how much AI referral traffic it sends you. It never blocks or alters requests — it simply logs matching visits and gives you a dashboard to review them. Think of it as AI SEO / answer engine optimization (AEO) visibility for site owners: see which AI crawlers are reading your content, and which AI answer engines (ChatGPT, Perplexity, and more) are actually sending you visitors.

Quick start:

1. Install and activate the plugin — no configuration required, logging starts immediately.
2. Wait for real traffic (or test with a curl request spoofing a bot User-Agent) — GPTBot, ClaudeBot, and others are detected automatically.
3. Check AI Crawlers > Dashboard for crawler visits, or AI Crawlers > AI Referrals for human click-throughs from AI answer engines like ChatGPT and Perplexity.
4. Generate a suggested llms.txt file from AI Crawlers > llms.txt Generator and place it at your site root.

How to use this data to optimize for AI:

• A high number of crawler visits with few AI referrals is normal and expected today — most AI crawling feeds training or indexing, not real-time answers. The Dashboard’s most-crawled-URL table shows you which pages get the most attention; make sure those pages are accurate, current, and well-structured.
• The AI Referrals page’s Crawl-to-Referral Gap panel puts a number on that relationship, so you can track whether it improves over time.
• Check the Captured Queries table for the (best-effort) questions that led people to your site — these are real signals of what your audience is asking AI assistants, and can guide what to write about next.
• Favor clear headings, concise direct answers, and FAQ-style content — AI answer engines tend to extract and cite well-structured, unambiguous text more readily than dense marketing copy.
• Use the llms.txt Generator to publish a curated, machine-readable summary of your key pages — see the FAQ below on llms.txt’s actual adoption status before treating it as a guaranteed lever.

Detected bots (built in, configurable):

• GPTBot (OpenAI)
• ChatGPT-User (OpenAI, on-demand)
• ClaudeBot (Anthropic)
• Claude-Web (Anthropic)
• PerplexityBot
• Google-Extended
• CCBot (Common Crawl)
• Bytespider (ByteDance)
• Amazonbot
• Applebot-Extended

Features:

• Lightweight init-hook based detection, no server log file access required.
• Custom database table with indexes for fast dashboard queries.
• Dashboard with summary cards, a Chart.js time-series chart, a most-crawled URL table, and a paginated raw log with CSV export.
• Configurable log retention with a daily cleanup cron job.
• Optional IP anonymization (masks the last octet/segment) for GDPR-friendly logging, enabled by default.
• Excludable URL path patterns (e.g. /wp-admin/*, /checkout/*).
• AI Referral Traffic tracking (AEO / answer-engine-optimization visibility): detects and tracks human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, captures the source query when the referring platform exposes one, and shows a crawl-to-referral conversion insight alongside a per-day-per-source chart.
• Built-in llms.txt generator: produces suggested content from your site title, tagline, pages, and posts for manual placement at your site root (see FAQ for llms.txt’s actual adoption status).
• Opt-in auto-update toggle right on the Settings page — stays in sync with WordPress core’s native ”Enable auto-updates” link on the Plugins screen.
• Fully functional on shared/managed hosting — no filesystem writes outside the database, no special server permissions needed.

Roadmap (planned direction, not a commitment to specific versions or dates):

• Correlating individual crawler visits with later referral visits to the same page, to see which crawled content actually converts into AI-driven traffic.
• Per-post/per-page llms.txt summaries, instead of one sitewide file.
• Additional AI referral sources as new answer engines emerge.
• A combined ”AI visibility” trend view comparing crawl volume and referral traffic side by side over longer time ranges.
• Revisiting llms.txt’s role in this plugin if/when major AI platforms confirm they actually consume it.

Suggestions are welcome via the support forum.