FocusWeb AI Crawler Monitor

FocusWeb AI Crawler Monitor is an observation-only plugin that helps you track and understand how AI crawlers interact with your WordPress site, and how much AI referral traffic it sends you. It never blocks or alters requests — it simply logs matching visits and gives you a dashboard to review them. Think of it as AI SEO / answer engine optimization (AEO) visibility for site owners: see which AI crawlers are reading your content, and which AI answer engines (ChatGPT, Perplexity, and more) are actually sending you visitors.

Quick start:

1. Install and activate the plugin — no configuration required, logging starts immediately.
2. Wait for real traffic (or test with a curl request spoofing a bot User-Agent) — GPTBot, ClaudeBot, and others are detected automatically.
3. Check AI Crawlers > Dashboard for crawler visits, or AI Crawlers > AI Referrals for human click-throughs from AI answer engines like ChatGPT and Perplexity.
4. Generate a suggested llms.txt file from AI Crawlers > llms.txt Generator and place it at your site root.

Detected bots (built in, configurable):

• GPTBot (OpenAI)
• ChatGPT-User (OpenAI, on-demand)
• ClaudeBot (Anthropic)
• Claude-Web (Anthropic)
• PerplexityBot
• Google-Extended
• CCBot (Common Crawl)
• Bytespider (ByteDance)
• Amazonbot
• Applebot-Extended

Features:

• Lightweight init-hook based detection, no server log file access required.
• Custom database table with indexes for fast dashboard queries.
• Dashboard with summary cards, a Chart.js time-series chart, a most-crawled URL table, and a paginated raw log with CSV export.
• Configurable log retention with a daily cleanup cron job.
• Optional IP anonymization (masks the last octet/segment) for GDPR-friendly logging, enabled by default.
• Excludable URL path patterns (e.g. /wp-admin/*, /checkout/*).
• AI Referral Traffic tracking (AEO / answer-engine-optimization visibility): detects and tracks human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, and captures the source query when the referring platform exposes one.
• Built-in llms.txt generator: produces suggested content from your site title, tagline, pages, and posts for manual placement at your site root.
• Fully functional on shared/managed hosting — no filesystem writes outside the database, no special server permissions needed.