Agent Ready Audit

The first file any crawler reads. Must return HTTP 200 with text/plain and contain at least one valid User-agent: directive.

Either a Sitemap: line in robots.txt OR a reachable /sitemap.xml at the apex. Sitemaps power agent crawl planning.

HTTP Link: response headers point agents to discovery resources without parsing HTML. Recommended rel values: api-catalog, mcp-server-card, agent-skills, llms-txt, sitemap.

When an agent sends Accept: text/markdown, the server returns a Markdown variant of the page. Cloudflare's Markdown for Agents and equivalent origin negotiation both qualify.

Whether robots.txt leaves known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, etc.) able to read the site. Blocking training-only crawlers (GPTBot, ClaudeBot, CCBot, Bytespider) is reported as an AI-training opt-out and does not fail the probe. Per-bot groups pass, and so does an open policy: a crawler with no group of its own follows User-agent: *, so a wildcard group that allows the site covers every AI crawler. The probe fails when robots.txt cannot be read, or when a known AI crawler is blocked from the whole site, by its own group or by a blanket Disallow: / it inherits. Google-Extended and Applebot-Extended are usage-control tokens that never fetch pages, so a Disallow for them is reported as an AI-training preference, not a block.

A structured directive in robots.txt declaring AI usage preferences: Content-Signal: search=yes, ai-input=yes, ai-train=yes. Adopted by Cloudflare AI Crawl Control as the default response.

An IETF draft for cryptographically authenticating bots via HTTP Message Signatures. Public keys live at /.well-known/http-message-signatures-directory. Mainly relevant if you operate a friendly bot; absence on a content site is fine.

/.well-known/api-catalog serves application/linkset+json. Lets agents discover OpenAPI specs, service docs, status endpoints without parsing markup.

/.well-known/oauth-authorization-server or /.well-known/openid-configuration. Required only if your APIs are protected; intentionally absent on read-only public sites. Scored only when the site shows a protected API (/api or /api/v1 answering 401 or sending a Bearer challenge, or an OAuth or bearer scheme in /openapi.json); otherwise reported as not applicable.

/.well-known/oauth-protected-resource. Tells agents which authorization server issues tokens for this resource. Static read-only sites can skip this honestly, so like OAuth discovery it is scored only when the site shows a protected API.

/.well-known/mcp/server-card.json. Declares serverInfo (name, version), one or more transports (http-stream, websocket, webmcp), and capabilities (tools, resources, prompts).

/.well-known/agent-skills/index.json. Lists SKILL.md files describing what an agent can do with your site. Each entry has name, type, description, url, and optionally a SHA-256 digest.

JS code that calls document.modelContext.registerTool() once per tool on page load to register tools with browser-side AI agents (navigator.modelContext is accepted as an alias). The draft removed the older provideContext() call on 2026-03-05, so a page that only calls provideContext() gets a warning. Detected by reading the homepage HTML and up to three WebMCP scripts it loads.

Coinbase-led protocol for agent-native HTTP payments. Protected routes return 402 with payment requirements that compatible agents fulfill automatically.

/openapi.json with x-payment-info extensions on payable operations.

/.well-known/ucp. Co-developed by Google, Shopify, Etsy.

/.well-known/acp.json. Stripe + OpenAI specification for agent-driven checkout.