AJ Agent Crawl Optimizer – AI Readiness Scanner & llms.txt for WordPress

AJ Agent Crawl Optimizer is a free WordPress plugin that scans your site against 21 AI-agent-readiness checks — the same set used by Cloudflare’s isitagentready.com — grades it on a Level 0–5 readiness ladder, fixes failing checks in one click, and lets you hand-curate the llms.txt that AI agents like ChatGPT, Claude, and Perplexity read.

It makes your site legible to AI agents — and, since 2.0, proves it. The built-in readiness scanner runs 21 checks against your live site, grades you on the Level 0–5 agent-readiness ladder (Not Ready → Basic Web Presence → Bot-Aware → Agent-Readable → Agent-Integrated → Agent-Native), and shows the full evidence trail — every request and response — behind every verdict. Failing checks get a Fix now button that enables the right feature and re-scans that single check to prove it went green; anything the plugin can’t fix in WordPress (DNS records, server config) gets a copy-paste prompt for your coding agent.

This is the technical, agent-readiness layer of answer engine optimization (AEO) and generative engine optimization (GEO): it improves your AI visibility by making your content legible to AI agents through llms.txt, JSON-LD, Markdown negotiation, and AI crawler rules. It is not a content SEO plugin — it doesn’t write titles or meta descriptions, and it doesn’t track brand mentions or AI citations. It prepares your WordPress site so AI search engines and agents can read, trust, and route on it, then verifies that over real HTTP.

The scan → fix → verify loop (new in 2.0)

• Dashboard (Agent Ready → Dashboard) — segmented category gauge, Level badge, and a «Next level» panel listing exactly which checks unlock the next level.
• 21 checks across 5 categories — Discoverability (robots.txt, sitemap, Link headers, DNS-AID), Content Accessibility (Markdown negotiation), Bot Access Control (AI bot rules, Content Signals, Web Bot Auth), API/Auth/MCP Discovery (API catalog, OAuth discovery, OAuth Protected Resource, auth.md, MCP server card, A2A agent card, Agent Skills, WebMCP), and Commerce (x402, MPP, UCP, ACP, AP2 — informational, never scored).
• Evidence timelines — fetch/parse/conclude steps with request/response snapshots, so you can audit why a check passed or failed (and catch page caches or server rules silently breaking your endpoints).
• One-click fixes with verification — 9 checks are fixable in one click; the plugin re-scans just that check afterwards. A bulk «Fix all safe items» sheet handles the rest.
• Hosting diagnosis — the plugin writes no files (every endpoint is served by WordPress), but some hosts block the requests before WordPress runs: nginx dot-path rules 403 /.well-known/*, and static-file rules 404 /llms.txt. When a feature is enabled and the server blocks it, the Dashboard says so and hands you copy-paste nginx and Apache fixes.
• REST API — POST /wp-json/ajaco/v1/scan, POST /wp-json/ajaco/v1/scan/check, POST /wp-json/ajaco/v1/fix, POST /wp-json/ajaco/v1/llms/preview, and a public GET /wp-json/ajaco/v1/health.
• WP-CLI — wp agent-ready scan --format=summary|json|agent, wp agent-ready fix <check>|--all-safe, wp agent-ready status for agency fleets and AI agents operating over SSH.

Each publishing capability remains a separate toggle under Agent Ready → Settings and ships opt-in (everything starts off). On first activation, a Quick Setup wizard suggests sensible defaults based on your environment (for example, it skips JSON-LD when an SEO plugin is detected so you don’t get duplicate structured data) — re-runnable any time.

Curated llms.txt — you decide what agents read

Most plugins generate llms.txt from fixed rules and hand you the result. This one hands you the controls.

• A dedicated editor (Agent Ready → llms.txt) — write the intro that tells an agent what your site is for, choose which content types appear (custom post types and WooCommerce products are detected automatically), and set each section’s heading, item count and order. Add your own Markdown block above or below.
• Live preview — see exactly what /llms.txt will serve, with entry, byte and token counts, before you save.
• Per-post control — every post and page gets an «Agent Ready (llms.txt)» panel in the editor: keep an entry out of the agent indexes entirely, or write a Summary for AI agents that overrides the excerpt. An excerpt is written for a human skimming; that line is written for a model deciding whether to fetch the page — and it is the single highest-leverage field in the whole plugin.
• /llms-full.txt serves the full content of the same curated entries. Password-protected content is always excluded from both.

The defaults keep the same sections and ordering as the previous automatic output (2.0 adds a one-line pointer to /llms-full.txt), so upgrading needs no attention until you decide to curate.

Discovery — help agents find what your site offers

• API Catalog (RFC 9727) — /.well-known/api-catalog linkset advertising your REST API, plus a Link: rel="api-catalog" header on every response so agents discover it from any URL.
• MCP Server Card (SEP-1649 draft) — /.well-known/mcp/server-card.json describing the site to MCP-aware agents.
• Agent Skills Index (RFC v0.2.0) — /.well-known/agent-skills/index.json listing six skills (content-query, posts-read, pages-read, media-library, categories, tags) plus per-skill SKILL.md artifacts with verifiable sha256 digests.
• llms.txt + llms-full.txt (per llmstxt.org) — /llms.txt is a curated LLM-readable index: you write the intro, choose which post types appear (custom post types and WooCommerce products included), set each section’s heading, count and order, and add your own Markdown block. /llms-full.txt serves the full content of those same entries as Markdown. Password-protected content is always excluded.
• Per-post curation — an «Agent Ready (llms.txt)» panel in the editor (block editor and classic) lets an author keep a page out of the agent indexes entirely, or write a Summary for AI agents that overrides the excerpt — a line aimed at a model deciding whether to fetch the page, not a human skimming.
• IndexNow — non-blocking ping to Bing and Yandex on every post publish so search engines re-crawl within minutes.

Presentation — format content for agents

• Markdown Negotiation — when a request includes Accept: text/markdown, the page is served as clean Markdown with X-Markdown-Tokens for context budgeting. Browsers (which send text/html) are completely unaffected.
• JSON-LD Schema — Schema.org structured data: WebSite, Organization, Article, BreadcrumbList, and auto-detected FAQPage. Logo resolved from your theme’s custom logo or site icon.
• OpenAPI 3.0.3 — /openapi.json (and legacy /?format=openapi) returns a complete spec generated dynamically from rest_get_server(), including plugin-registered REST routes.
• WebMCP Tools — registers four tools (search, posts, pages, site info) via navigator.modelContext.provideContext() for browsers that support the W3C WebMCP draft.

Declarations

• AI Bot Rules (new in 2.0) — to be cited by an AI engine, you must first allow its crawler. The plugin writes explicit robots.txt User-agent groups for the 15 AI crawlers readiness scanners check for (GPTBot and ChatGPT-User from OpenAI, Claude-Web, PerplexityBot, CCBot, Google-Extended, Bytespider, …), each allowed or blocked per a filterable per-bot policy. Groups are self-contained (they replicate core’s wp-admin protections) and honor «Discourage search engines».
• Content-Signals — appends a Content-Signal: ai-train=no, search=yes, ai-input=no directive to robots.txt (inside an explicit User-agent: * group) declaring your AI-usage preferences per contentsignals.org. Composes with Yoast/Rank Math/AIOSEO — their additions are preserved, our directive lands last.
• auth.md (new in 2.0) — publishes /auth.md honestly documenting how agents authenticate to your REST API via Application Passwords: how a human creates one, Basic-auth usage, scope, and revocation. No fictional OAuth endpoints.

How it compares to other AI / llms.txt plugins

Most AI and llms.txt plugins only generate a file from fixed rules. AJ Agent Crawl Optimizer also scans your live site over real HTTP, scores it on the Level 0–5 ladder, shows the request/response evidence behind every verdict, and fixes failures in one click — then re-scans to prove the check turned green. It writes per-bot robots.txt rules for 15 AI crawlers and lets you hand-curate what AI agents read instead of auto-dumping every post. Scan → score → fix → verify, not generate-and-hope.

Why use it

• No conflicts with your SEO plugin. JSON-LD auto-suppresses when Yoast SEO, Rank Math, All in One SEO, SEOPress, The SEO Framework, Slim SEO, Squirrly SEO, Schema Pro, or SASWP is active.
• Multisite-aware. Every endpoint also resolves at /{subsite}/... paths automatically.
• Cached. OpenAPI is cached for a day, llms.txt for an hour, with proper invalidation on plugin activation, theme switch, post changes, and setting toggles.
• Performance-conscious. Markdown handler runs at PHP_INT_MAX priority so it doesn’t break object-cache flushes or Query Monitor. IndexNow pings are non-blocking.
• Extensible. Seventeen filter hooks let you customize skills, schemas, bot policies, llms.txt curation, scan behavior, and content. See «For Developers» below.
• Accessible. The Dashboard gauge carries a dynamic aria-label, copy buttons announce success via aria-live, and decorative icons are hidden from screen readers.

For developers

The plugin exposes seventeen filter hooks for extension. Examples:

Delegate plugin access to a non-admin role.

Register custom skills that ship in the Agent Skills Index and are served as SKILL.md artifacts with verifiable sha256 digests.

Customize the Content-Signal directive (e.g. permit AI training).

Other hooks: ajaco_api_catalog_linkset, ajaco_mcp_server_card, ajaco_json_ld_graph, ajaco_openapi_spec, ajaco_llms_txt_content, ajaco_llms_full_txt_content, ajaco_llms_post_types, ajaco_llms_exclude_post, ajaco_auth_md_content, ajaco_ai_bot_list, ajaco_ai_bot_policy, ajaco_commerce_signals, ajaco_scan_sslverify, ajaco_active_seo_plugin. The Help tab (top right of the settings screen) lists all of them with descriptions.

Per-post curation is stored in two post meta fields you can read or set programmatically: _ajaco_llms_exclude (boolean) and _ajaco_llms_summary (string, 300 chars). Both are registered with show_in_rest, so they round-trip through the REST API and WP-CLI (wp post meta update 42 _ajaco_llms_summary "...").

External services

This plugin connects to external services only in two explicitly user-triggered situations: the opt-in IndexNow feature, and DNS lookups made while an administrator runs a readiness scan.

DNS-over-HTTPS resolvers (cloudflare-dns.com, dns.google) — during scans only

What they are and what they’re used for: the readiness scanner’s DNS-AID check queries public DNS-over-HTTPS resolvers (Cloudflare’s https://cloudflare-dns.com/dns-query, with Google’s https://dns.google/resolve as fallback) to look up _agents SVCB/HTTPS/TXT records for your own domain.

When data is sent: only while a scan that an administrator started (Dashboard, REST, or WP-CLI) is running. The scanner also sends ordinary HTTP requests to your own site’s URLs to verify your endpoints.

What data is sent: only DNS query names derived from your own domain (e.g. _index._agents.example.com). No visitor information, IP addresses of visitors, or content is sent. Cloudflare’s and Google’s resolver privacy policies apply:

• Cloudflare 1.1.1.1 privacy: https://developers.cloudflare.com/1.1.1.1/privacy/public-dns-resolver/
• Google Public DNS privacy: https://developers.google.com/speed/public-dns/privacy

IndexNow (api.indexnow.org)

What it is and what it’s used for: IndexNow is an open protocol (originally from Microsoft Bing and Yandex) that lets sites notify search engines the moment a URL is published or updated, so search engines can re-crawl within minutes instead of days.

When data is sent: only when the IndexNow feature toggle is turned on AND an IndexNow API Key is configured on the settings page. In that case, every time a post of a public post type transitions to the publish status, the plugin fires a single non-blocking HTTPS POST to https://api.indexnow.org/indexnow.

What data is sent: the request body is a JSON document containing exactly three fields — your site’s host (e.g. example.com), your IndexNow API key (which you generated yourself at Bing Webmaster Tools), and the permalink URL of the post that was just published. No visitor information, IP addresses, user-agents, or post content is sent.

If the IndexNow feature toggle is off (the default) and no scan is running, the plugin makes no outbound network requests of any kind.

This service is provided by Microsoft (Bing) and the IndexNow project. Their terms and privacy policies apply:

• IndexNow protocol documentation: https://www.indexnow.org/documentation
• IndexNow FAQ and terms: https://www.indexnow.org/faq
• Microsoft Bing Webmaster Guidelines : https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
• Microsoft Privacy Statement: https://www.microsoft.com/en-us/privacy/privacystatement

Privacy

AJ Agent Crawl Optimizer stores data only on your own server — there is no telemetry, no analytics, and no third-party logging. Specifically:

Local data

• Plugin option rows in the WordPress options table store toggle states, the per-bot AI crawler policy, the most recent scan result (including its evidence snapshots of your own site’s responses), and the IndexNow API Key (stored as plain text — keep your database secure).
• Transients cache the OpenAPI document and the llms.txt / llms-full.txt bodies. These are deleted on plugin uninstall.
• No request data, IP addresses, User-Agents, or visitor information is recorded by the plugin.

Outbound network calls

The plugin makes outbound HTTP requests only in these cases:

• IndexNow — when the IndexNow toggle is on and a key is configured, the plugin sends a non-blocking POST to https://api.indexnow.org/indexnow on every post publish. The payload contains the site host, the IndexNow key, and the URL of the published post (no visitor data).
• Readiness scans — while an administrator-initiated scan is running, the scanner sends HTTP requests to your own site’s URLs (to verify endpoints exactly as an agent would see them) and DNS-over-HTTPS queries for your own domain to Cloudflare/Google resolvers (DNS-AID check). Scans never run automatically.

Outside those two cases the plugin makes zero outbound network requests. All publishing features (manifests, JSON-LD, robots.txt) only respond to incoming HTTP requests; they never call out.

Cookies

The plugin does not set any cookies.

Uninstall

When the plugin is deleted (not just deactivated), all plugin options, the IndexNow key, and all transients are removed from the database. Multisite networks have every site cleaned in turn.