Agent readiness past llms.txt: eleven files, and the fi...

Most of what has been written about making a site legible to AI agents stops at llms.txt. That file has been the whole conversation for about two years now, and it is the least interesting part of the surface. It is a plain text summary. Writing one takes an afternoon and nothing about it can really go wrong.

The rest of the surface is a different problem. There are now roughly a dozen discovery documents an agent might look for, spread across /.well-known/, the site root, HTTP response headers, robots.txt, the HTML head, and DNS. Several of them are drafts. Several of them disagree with each other about where to live. And five of the ones I published this week parsed as perfectly valid JSON, read correctly to a human, and were rejected outright by a validator.

That last part is the reason for this post. A missing file is easy: something tells you it is missing and you write it. A file that is present, well formed, and wrong is the expensive kind, because every tool in your own pipeline says it is fine.

I put a small client site through this. It is a service business with 51 pages, a site I build and maintain, so I could break things and measure them rather than theorize. Before I started it scored Level 1 of 4, "Basic Web Presence", with 3 checks passing and 13 failing. After a day it was Level 4, "Agent-Integrated", with 11 passing. The 5 that still fail, I am not going to fix, and the last section explains why that is the correct answer.

The eleven things, and why each one exists

Link response headers (RFC 8288)

A Link header on the homepage response, pointing at your other discovery documents.

Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json",
      </agents.md>; rel="service-doc"; type="text/markdown",
      </llms.txt>; rel="describedby"; type="text/plain"

Why it matters: a large share of automated clients issue a HEAD or a GET, read the headers, and never parse the body. For those clients the head of your HTML does not exist. This is the cheapest item on the list, it is one line of config, and it is the one most sites are missing.

One rule: only use relations that are actually registered with IANA. An invented rel value is indistinguishable from a typo to anything doing strict matching, and you get no error telling you so.

Content Signals in robots.txt

Allow and Disallow have never been able to say what a crawler may do with what it takes. Content Signals fill that gap with three declarations: search, ai-input, and ai-train.

Content-Signal: search=yes, ai-input=yes, ai-train=yes

Why it matters: it separates "you may fetch this" from "you may train on this", which robots.txt has conflated since 1994. It is a statement of preference and not an access control, and it is worth being clear-eyed about that. Nobody is obligated to honor it.

This is the trap that cost me the most, and I have not seen it written down anywhere. Under RFC 9309, a crawler obeys exactly one group: the most specific one matching its token. A named group replaces the wildcard group rather than inheriting from it. So if your robots.txt names GPTBot, ClaudeBot, PerplexityBot and Google-Extended in their own groups, which every well-maintained robots.txt now does, and you put one Content-Signal line under User-agent: *, you have published a preference that is invisible to every single AI crawler it was written for.

It also passes every presence-based scanner, including the one I was testing against, because they grep the file for the directive rather than parsing it into groups. The site I was working on has 21 user-agent groups. It now has 21 Content-Signal lines. If you take one thing from this post, take this one.

The API catalog (RFC 9727)

/.well-known/api-catalog, and it is not a list of endpoints. It is a linkset, the format from RFC 9264:

{ "linkset": [ { "anchor": "https://example.com/mcp",
    "service-desc": [ { "href": "…/server-card.json", "type": "application/json" } ] } ] }

Why it matters: it is the registered, standards-track answer to "what APIs does this origin have", as opposed to the half dozen vendor-specific manifests competing to answer the same question.

Two things bite. The file has no extension, so nothing can infer its type and you have to set application/linkset+json on the server yourself. And the whole value is in the href targets resolving, which is exactly the thing nobody checks after a rename.

Markdown for Agents

Respond to Accept: text/markdown with markdown instead of HTML.

Why it matters: it is the only item on this list with a directly measurable payoff. The pricing page on the site I tested is 71,005 bytes of HTML and 5,743 bytes of markdown. That is a 92 percent reduction in what a model has to read to get the same facts, and the markdown has no nav, no inline CSS, and no JSON-LD block competing for attention.

The obvious implementation converts HTML in an edge function on every request. I would push back on that. It pays a parse cost per request, it cannot be tested in CI, and when it breaks it breaks silently for the clients least able to report it. Pre-generate a markdown twin per page at build time instead. It becomes a build artifact you can diff and gate, and the edge function shrinks to picking a file.

The trap: Vary: Accept has to go on both branches, the markdown one and the HTML one. Two different bodies now live at one URL, separated only by a request header. A cache that has not been told this will hand a browser the markdown, or hand an agent the HTML, depending on who asked first. Also make sure */*, which is what curl sends, still gets HTML. It is not a request for markdown.

ARD, the capability manifest

/.well-known/ai-catalog.json. Lists your MCP servers, agents, schemas and skills in one place, each with 2 to 5 representativeQueries so a registry can build embeddings and route a question to you.

Why it matters: it is the only document here designed for discovery by a third party rather than by a crawler that already has your URL. The representative queries are the interesting part. You are describing the questions you want to be the answer to.

The trap, and it is a good one: the entry key is identifier. Not id. I wrote id, because id is what every other JSON schema on earth calls that field. The result was a file that parsed, validated as JSON, read correctly to me, and came back from the scanner as "ARD capability manifest has no valid entries (entry 0 is missing identifier)". There is no linter that catches this. The host object needs its own identifier too, conventionally did:web:yourdomain.com.

Agent Skills

/.well-known/agent-skills/index.json, listing SKILL.md files that describe tasks an agent can carry out with your site.

Why it matters: this is the one I was most skeptical of and ended up liking most. A skill is not documentation, it is an instruction set with an opinion. The three I wrote for the client site are "request a quote", "check coverage" and "estimate cost", and each one contains a section on what the agent must not do. The estimate skill says outright: do not present an estimate as a quote, and never state a final price. That is a constraint I want travelling with the data, and there was previously nowhere to put it.

Each entry carries a sha256 of the file it names. Compute it at build time. A digest maintained by hand is a digest that is wrong within a week.

The MCP server card

/.well-known/mcp/server-card.json, declaring a server's name, transport, endpoint and capabilities.

Why it matters: it turns "read my pages and infer" into "call this and know". For the client site I wrote five read-only tools: services, pricing, service area coverage, contact, and lead times. The coverage one is the clearest illustration of why this beats scraping. Asked about a city that has a page, it returns the city, the distance and the drive time. Asked about a city that does not, it returns served: null and says so, with the phone number. A model reading the pages would have guessed, and the guess would have been reasonable and possibly wrong.

The rule that matters more than the format: the card and a working server ship together or neither ships. A card naming an endpoint that 404s is worse than no card at all. It costs an agent a round trip and teaches it that your domain misdescribes itself.

Deliberately, none of those five tools writes anything. No create_lead, no book_appointment. A tool that writes into a one-person business's inbox is a tool that fills a one-person business's inbox, and the person sorting the real enquiries out of the synthetic ones is the owner.

WebMCP

The same idea inside the browser: the page registers tools on navigator.modelContext so an agent operating in the tab gets named functions with schemas instead of a DOM to scrape.

Why it matters: the in-page agent and the server-side agent are different clients with different constraints, and until now only one of them had an interface. Feature-detect it, keep the file inert when the API is absent, and have the tools delegate to your MCP endpoint rather than embedding a second copy of your business data.

auth.md

A root-level markdown file saying how an agent authenticates.

Why it matters: because "there is no authentication" is a real answer that currently has nowhere to live, and an agent otherwise has to discover it by probing.

The trap: the H1 must contain the literal string auth.md. I wrote # Authentication, which is the obvious heading, and it was rejected. The heading is how a scanner distinguishes a real auth.md from any other markdown that happens to sit at that path. # auth.md for yourdomain.com works.

If you have no OAuth server, the spec's own guidance is to keep the file self-contained: name the agent audience, state that there is no registration endpoint, list the supported method, and say what credentials to send. Mine says: send none, an Authorization header will be ignored rather than rejected.

DNS based discovery

Two separate things that get discussed as one, and they have very different costs.

The cheap one is a TXT record at _catalog._agents.yourdomain.com containing url=https://yourdomain.com/.well-known/ai-catalog.json. Almost any DNS host can do this. I published one in about thirty seconds.

The expensive one is DNS-AID proper, which wants a ServiceMode SVCB record at _index._agents.yourdomain.com and the zone signed with DNSSEC. Check what your DNS provider actually supports before you plan around this. The site I was working on is on Netlify DNS, whose API reports its supported record types explicitly: A, AAAA, MX, CNAME, TXT, NS, SPF, ALIAS, SRV, NETLIFY, NETLIFYv6, CAA and HTTPS. SVCB is not on that list, and Netlify DNS offers no DNSSEC signing at all. So DNS-AID on that domain is not a configuration change, it is a zone migration, with the email and web resolution risk that implies.

That is a legitimate reason to leave a check failing. Which brings me to the last section.

The RSS feed you probably already have

Worth a line, because it keeps getting left out of these lists. A feed is the oldest machine-readable surface on the web and it still works. If you publish anything dated, publish a feed, and declare it in the head. It costs nothing and it is the one format every aggregator already understands.

What I did not publish, and why that is the right call

Five checks still fail on that site. Four of them are deliberate.

OAuth discovery metadata and Protected Resource Metadata. There is no authorization server and there are no protected resources. Publishing the metadata would advertise a registration endpoint that does not exist. There is also a specific reason not to: MCP clients probe /.well-known/oauth-protected-resource to decide whether a server needs auth, and a 404 is the specified way to say "it does not". Returning a document there would be actively worse than returning nothing.

The A2A agent card. A2A describes an agent that other agents delegate tasks to. This domain publishes information and a phone number. There is no autonomous worker behind it, and a card would claim a capability that is not there.

The commerce specs, x402, ACP, UCP, MPP and AP2. Nothing is sold on the site. The scanner correctly marks these neutral rather than failed, which is a nice piece of design on its part.

There is a real temptation, once a number is attached to something, to publish files until the number goes up. Resist it in this particular case, because every one of these documents is a promise made to a client that cannot see your face. A wrong Content-Signal is a preference nobody reads. A wrong MCP server card is an agent that tried to talk to you and failed. The scoring is a means, and the moment you publish a manifest for a thing that does not exist, you have made your site worse and your score better.

Gate it, because none of this is visible

Every one of these files is invisible in a browser. You will never notice a broken one by using your own site. Everything I published is generated from the same data file the pages read, and a build gate resolves every URL any manifest names, recomputes every sha256 against the file it points at, checks that every user-agent group carries its Content-Signal, and confirms the H1 of auth.md says what it has to say. It also refuses to let any internal field name from the source data escape into a published document, because the source is full of editorial notes sitting right next to the real values.

That gate took about an hour and it has already caught two of my own mistakes: it failed the build when I renamed id to identifier in one place and not the other, and it failed again when the server card advertised a tool the server did not implement.

Which is the whole point. These are not files you write once. They are an interface, and an interface without tests is a rumor.

If you are working out where a small site's effort actually pays back, The $20 Dollar Agency covers the wider version of this argument, which is that the cheap technical work compounds and the expensive marketing work usually does not.

Fact-check notes and sources

RFC 8288, Web Linking: the Link header format and the requirement that relation types be registered. rfc-editor.org/rfc/rfc8288
RFC 9727, api-catalog: the well-known URI and its use of the linkset format. rfc-editor.org/rfc/rfc9727
RFC 9264, Linksets: the linkset JSON structure used by the API catalog. rfc-editor.org/rfc/rfc9264
RFC 9309, Robots Exclusion Protocol: group matching, and the rule that a crawler obeys exactly one group. This is the normative basis for the Content-Signal placement trap. rfc-editor.org/rfc/rfc9309
RFC 9728, OAuth Protected Resource Metadata: the document deliberately not published here. rfc-editor.org/rfc/rfc9728
RFC 9460, SVCB and HTTPS resource records: the record type DNS-AID builds on. rfc-editor.org/rfc/rfc9460
Content Signals: contentsignals.org and the IETF draft, draft-romm-aipref-contentsignals.
DNS-AID: draft-mozleywilliams-dnsop-dnsaid, at datatracker.ietf.org. Still a draft; treat accordingly.
Agentic Resource Discovery: agenticresourcediscovery.org and the ai-catalog data model at github.com/Agent-Card/ai-catalog. The identifier field name is specified there. ARD is a v0.9 draft; specVersion refers to the ai-catalog data model, not to ARD.
MCP server card: SEP-1649, in flight at modelcontextprotocol PR #2127. The path is still settling: some scanners look at /.well-known/mcp.json, the SEP uses /.well-known/mcp/server-card.json. Publishing both is defensible today.
Agent Skills Discovery: the v0.2.0 RFC at github.com/cloudflare/agent-skills-discovery-rfc.
Markdown for Agents: Cloudflare's documentation.
Netlify DNS supported record types: read from the Netlify API's own supported_record_types field on the zone object on 2026-08-31. The absence of SVCB and of DNSSEC is that response, not an inference.
The byte counts (71,005 HTML against 5,743 markdown) are the live responses for one page of the site described, measured on 2026-08-31. The 3-pass-to-11-pass and Level 1 to Level 4 figures are before and after scans of that same site on the same day.

Agent readiness past llms.txt: eleven files, and the five that silently fail validation