Agent readiness past llms.txt: eleven files, and the fi...

Most of what has been written about making a site legible to AI agents stops at llms.txt. That file has been the whole conversation for about two years now, and it is the least interesting part of the surface. It is a plain text summary. Writing one takes an afternoon and nothing about it can really go wrong.

The rest of the surface is a different problem. There are now roughly a dozen discovery documents an agent might look for, spread across /.well-known/, the site root, HTTP response headers, robots.txt, the HTML head, and DNS. Several of them are drafts. Several of them disagree with each other about where to live. And five of the ones I published this week parsed as perfectly valid JSON, read correctly to a human, and were rejected outright by a validator.

That last part is the reason for this post. A missing file is easy: something tells you it is missing and you write it. A file that is present, well formed, and wrong is the expensive kind, because every tool in your own pipeline says it is fine.

I put a small client site through this. It is a service business with 51 pages, a site I build and maintain, so I could break things and measure them rather than theorize. Before I started it scored Level 1 of 4, "Basic Web Presence", with 3 checks passing and 13 failing. After a day it was Level 4, "Agent-Integrated", with 11 passing. The 5 that still fail, I am not going to fix, and the last section explains why that is the correct answer.

The eleven things, and why each one exists

Link response headers (RFC 8288)

A Link header on the homepage response, pointing at your other discovery documents.

Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json",
      </agents.md>; rel="service-doc"; type="text/markdown",
      </llms.txt>; rel="describedby"; type="text/plain"

Why it matters: a large share of automated clients issue a HEAD or a GET, read the headers, and never parse the body. For those clients the head of your HTML does not exist. This is the cheapest item on the list, it is one line of config, and it is the one most sites are missing.

One rule: only use relations that are actually registered with IANA. An invented rel value is indistinguishable from a typo to anything doing strict matching, and you get no error telling you so.

Content Signals in robots.txt

Allow and Disallow have never been able to say what a crawler may do with what it takes. Content Signals fill that gap with three declarations: search, ai-input, and ai-train.

Content-Signal: search=yes, ai-input=yes, ai-train=yes

Why it matters: it separates "you may fetch this" from "you may train on this", which robots.txt has conflated since 1994. It is a statement of preference and not an access control, and it is worth being clear-eyed about that. Nobody is obligated to honor it.

This is the trap that cost me the most, and I have not seen it written down anywhere. Under RFC 9309, a crawler obeys exactly one group: the most specific one matching its token. A named group replaces the wildcard group rather than inheriting from it. So if your robots.txt names GPTBot, ClaudeBot, PerplexityBot and Google-Extended in their own groups, which every well-maintained robots.txt now does, and you put one Content-Signal line under User-agent: *, you have published a preference that is invisible to every single AI crawler it was written for.

It also passes every presence-based scanner, including the one I was testing against, because they grep the file for the directive rather than parsing it into groups. The site I was working on has 21 user-agent groups. It now has 21 Content-Signal lines. If you take one thing from this post, take this one.

The API catalog (RFC 9727)

/.well-known/api-catalog, and it is not a list of endpoints. It is a linkset, the format from RFC 9264:

{ "linkset": [ { "anchor": "https://example.com/mcp",
    "service-desc": [ { "href": "…/server-card.json", "type": "application/json" } ] } ] }

Why it matters: it is the registered, standards-track answer to "what APIs does this origin have", as opposed to the half dozen vendor-specific manifests competing to answer the same question.

Two things bite. The file has no extension, so nothing can infer its type and you have to set application/linkset+json on the server yourself. And the whole value is in the href targets resolving, which is exactly the thing nobody checks after a rename.

Markdown for Agents

Respond to Accept: text/markdown with markdown instead of HTML.

Why it matters: it is the only item on this list with a directly measurable payoff. The pricing page on the site I tested is 71,005 bytes of HTML and 5,743 bytes of markdown. That is a 92 percent reduction in what a model has to read to get the same facts, and the markdown has no nav, no inline CSS, and no JSON-LD block competing for attention.

The obvious implementation converts HTML in an edge function on every request. I would push back on that. It pays a parse cost per request, it cannot be tested in CI, and when it breaks it breaks silently for the clients least able to report it. Pre-generate a markdown twin per page at build time instead. It becomes a build artifact you can diff and gate, and the edge function shrinks to picking a file.

The trap: Vary: Accept has to go on both branches, the markdown one and the HTML one. Two different bodies now live at one URL, separated only by a request header. A cache that has not been told this will hand a browser the markdown, or hand an agent the HTML, depending on who asked first. Also make sure */*, which is what curl sends, still gets HTML. It is not a request for markdown.

ARD, the capability manifest

Updated October 13, 2026: this section now follows ARD v0.91 and the way Lighthouse grades the file. The first version told you to put a url in host, which Lighthouse 13.5 now scores 0. Later the same day I added the right entry types for MCP and A2A, how the catalog and the two cards fit together, how to check the file, and what fixing it across 31 sites taught me.

/.well-known/ard.json. Lists your MCP servers, agents and skills in one place, each with 2 to 5 representativeQueries so a registry can build embeddings and route a question to you. Until August 2026 the address was /.well-known/ai-catalog.json. ARD v0.91 moved it to ard.json and says a reader may still check the old name, but does not have to. Lighthouse 13.5.0, the current release, still reads only the old name. So publish one file at both addresses.

Why it matters: it is the only document here designed for discovery by a third party rather than by a crawler that already has your URL. The representative queries are the interesting part. You are describing the questions you want to be the answer to. It also now has a grader: Lighthouse runs an ard-schema audit in its Agentic Browsing category.

The trap that zeroes the score: host is closed. In Lighthouse 13.5.0 it may hold only displayName, identifier, documentationUrl, logoUrl and trustManifest, and the top level only specVersion, host and entries. I had put the site's url and a description in host, because that is where a publisher's address and blurb obviously go. One extra field and the whole audit scores 0, with a single line of explanation: "must NOT have additional properties at path 'host'". Every catalog I had published had that field.

The second trap: only agent resources are entries, and each one needs its real media type. An MCP server card is application/mcp-server-card+json, an A2A agent card is application/a2a-agent-card+json, and each SKILL.md is its own entry with text/markdown; profile="urn:air:agent-skills". Typing an MCP or A2A entry as application/json is a mistake, even though the card is JSON: Lighthouse 13.5.0 gives that entry a Low warning, which caps the score at 0.9, and it tells a reader nothing about what kind of card sits at the address. The same goes for text/plain, text/html and plain text/markdown. And the MCP entry's url is the card, not the /mcp endpoint itself. Listing llms.txt as text/plain, a sitemap, an HTML page or a JSON data file as entries earns a warning per entry. Keep those addresses as flat values in an entry's metadata and in the API catalog, where they belong.

The original trap still applies: the entry key is identifier. Not id. I wrote id, because id is what every other JSON schema on earth calls that field. The result was a file that parsed, validated as JSON, read correctly to me, and came back from the scanner as "ARD capability manifest has no valid entries (entry 0 is missing identifier)". The format is urn:air:yourdomain.com:name. The host.identifier (did:web:yourdomain.com) is a good idea but optional.

A catalog that passes both Lighthouse 13.5.0 and the next Lighthouse version with no issues looks like this:

{
  "specVersion": "1.0",
  "host": {
    "displayName": "Your Business",
    "identifier": "did:web:yourdomain.com",
    "documentationUrl": "https://yourdomain.com/AGENTS.md"
  },
  "entries": [
    {
      "identifier": "urn:air:yourdomain.com:mcp:site",
      "displayName": "Your Business MCP server",
      "type": "application/mcp-server-card+json",
      "url": "https://yourdomain.com/mcp/server-card",
      "representativeQueries": ["Are you open on Sunday?", "How much is a quote for a small job?"]
    },
    {
      "identifier": "urn:air:yourdomain.com:skills:request-a-quote",
      "displayName": "Request a quote (agent skill)",
      "type": "text/markdown; profile=\"urn:air:agent-skills\"",
      "url": "https://yourdomain.com/.well-known/agent-skills/request-a-quote/SKILL.md",
      "representativeQueries": ["How do I get a quote?", "What do you need from me for an estimate?"],
      "metadata": { "llms-txt": "https://yourdomain.com/llms.txt" }
    }
  ]
}

Then point at it from robots.txt (Agentmap: https://yourdomain.com/.well-known/ard.json) and from your Link header (</.well-known/ard.json>; rel="ard"). One more thing to check: if your site is a single-page app that answers every address with its home page, Lighthouse reads that page at /.well-known/ai-catalog.json as a broken catalog and scores it 0 even though you never published one. Make unknown /.well-known/ addresses answer 404.

The MCP entry points at https://yourdomain.com/mcp/server-card, the card address the MCP server card extension recommends. If you also serve the same card at /.well-known/mcp/server-card.json for older scanners, that is fine, but list the recommended address in the catalog.

How the catalog, the A2A card and the MCP card fit together

Three files, three jobs. The chain is catalog, then card, then endpoint.

The catalog is the site-wide index. One file at one well-known address lists every agent resource the site has. Each entry is short: an identifier, a displayName, a media type, the url of a card, and 2 to 5 representativeQueries. It does not need to repeat what the card says.
The MCP server card describes one MCP server. It holds $schema, a name like com.example/site, a version, a short description, and a remotes list that names the endpoint and its transport. It does not list tools, resources or prompts. A client gets those from tools/list when it connects. Its catalog entry: identifier like urn:air:yourdomain.com:mcp:site, type application/mcp-server-card+json, url the card.
The A2A agent card lives at /.well-known/agent-card.json and describes one A2A agent: something that accepts A2A messages at an endpoint. It holds the name, description, version, the interfaces (each with a url, a protocolBinding and a protocolVersion), capabilities, input and output modes, and the agent's skills. It is not a general "about this site" card. Do not publish one unless an A2A endpoint answers it. Its catalog entry: type application/a2a-agent-card+json, url the card.
The endpoints (/mcp, /a2a) do the work. Nothing in the catalog points at them directly.

Neither card has a field for "here is my catalog". So on jwatte.com the card responses carry a header that points back: Link: </.well-known/ard.json>; rel="ard". The catalog is also listed in llms.txt, AGENTS.md and the API catalog, so a reader who starts anywhere can find the index.

How to check it

Run Lighthouse's Agentic Browsing category:

npx lighthouse https://yourdomain.com/ --only-categories=agentic-browsing

Or open Chrome DevTools, the Lighthouse panel, and pick Agentic Browsing. Look at the ard-schema audit, "ARD manifest is valid", in the Agent Discoverability group next to the llms-txt audit. The score works like this:

0: any Error. A field host does not allow, a missing required field, a body that is not JSON, or an HTML page served at the catalog address.
0.9: no errors, but at least one "Low" warning. The usual causes are an entry type outside the standard list (such as application/json or text/plain) or representativeQueries missing or outside 2 to 5.
1: a clean file.

Lighthouse 13.5.0 looks for the file in this order and fetches only the first one that exists: a robots.txt Agentmap: line, a <link rel="ai-catalog"> tag, a Link header with rel="ai-catalog", then /.well-known/ai-catalog.json. It checks the body, not the Content-Type, and never looks at DNS. The next version, already merged into Lighthouse's main branch on October 9, 2026, reads rel="ard" and /.well-known/ard.json first, and warns when it finds the file only at the old address.

If you would rather not install anything, the Agent Endpoint Inspector and the Mega Analyzer both run Lighthouse's own validator code on your catalog and give two verdicts, Lighthouse 13.5 and ARD v0.91 (Lighthouse next), with every message explained in plain words. For the cards: check an MCP card against schema.json in the ext-server-card repository, and an A2A card against the required fields in section 4.4.1 of the A2A specification.

Fixing it across many sites

On October 13, 2026 I checked a fleet of 31 small sites I run. All 31 catalogs scored 0 in Lighthouse 13.5.0, for the same reason: url and description inside host. Most also had application/json, text/plain or text/html entries. Every one of them parsed as valid JSON and passed every older check I had. The 31 files came from nine different generator families, all built from one shared template design, which is why one mistake showed up everywhere. The fix that lasts is in the thing that writes the file and the gate that checks it, not in the file. By 18:40 UTC that day, 29 of the 31 scored 1 under both Lighthouse 13.5.0 and the ARD v0.91 rules, and 2 still served the old shape. The full lesson is in One CORS header, 28 sites.

Agent Skills

/.well-known/agent-skills/index.json, listing SKILL.md files that describe tasks an agent can carry out with your site.

Why it matters: this is the one I was most skeptical of and ended up liking most. A skill is not documentation, it is an instruction set with an opinion. The three I wrote for the client site are "request a quote", "check coverage" and "estimate cost", and each one contains a section on what the agent must not do. The estimate skill says outright: do not present an estimate as a quote, and never state a final price. That is a constraint I want travelling with the data, and there was previously nowhere to put it.

Each entry carries a sha256 of the file it names. Compute it at build time. A digest maintained by hand is a digest that is wrong within a week.

The MCP server card

Updated October 13, 2026: the card format is now final (SEP-2127), and a current card names the server and its endpoint but does not list tools or capabilities. The recommended address moved too.

https://yourdomain.com/mcp/server-card, the MCP endpoint plus /server-card, served as application/mcp-server-card+json. It declares a server's name, version, a short description and the endpoint with its transport. It does not list tools or advertise capabilities: a client asks the server with tools/list. A .well-known card path is allowed but not recommended. Some scanners still read /.well-known/mcp/server-card.json and /.well-known/mcp.json, so serving the same bytes there too does no harm.

Why it matters: it turns "read my pages and infer" into "call this and know". For the client site I wrote five read-only tools: services, pricing, service area coverage, contact, and lead times. The coverage one is the clearest illustration of why this beats scraping. Asked about a city that has a page, it returns the city, the distance and the drive time. Asked about a city that does not, it returns served: null and says so, with the phone number. A model reading the pages would have guessed, and the guess would have been reasonable and possibly wrong.

The rule that matters more than the format: the card and a working server ship together or neither ships. A card naming an endpoint that 404s is worse than no card at all. It costs an agent a round trip and teaches it that your domain misdescribes itself.

Deliberately, none of those five tools writes anything. No create_lead, no book_appointment. A tool that writes into a one-person business's inbox is a tool that fills a one-person business's inbox, and the person sorting the real enquiries out of the synthetic ones is the owner.

WebMCP

The same idea inside the browser: the page registers tools on navigator.modelContext so an agent operating in the tab gets named functions with schemas instead of a DOM to scrape.

Why it matters: the in-page agent and the server-side agent are different clients with different constraints, and until now only one of them had an interface. Feature-detect it, keep the file inert when the API is absent, and have the tools delegate to your MCP endpoint rather than embedding a second copy of your business data.

auth.md

A root-level markdown file saying how an agent authenticates.

Why it matters: because "there is no authentication" is a real answer that currently has nowhere to live, and an agent otherwise has to discover it by probing.

The trap: the H1 must contain the literal string auth.md. I wrote # Authentication, which is the obvious heading, and it was rejected. The heading is how a scanner distinguishes a real auth.md from any other markdown that happens to sit at that path. # auth.md for yourdomain.com works.

If you have no OAuth server, the spec's own guidance is to keep the file self-contained: name the agent audience, state that there is no registration endpoint, list the supported method, and say what credentials to send. Mine says: send none, an Authorization header will be ignored rather than rejected.

DNS based discovery

Two separate things that get discussed as one, and they have very different costs.

The cheap one is a TXT record at _catalog._agents.yourdomain.com containing url=https://yourdomain.com/.well-known/ard.json. Almost any DNS host can do this. I published one in about thirty seconds. (October 13, 2026: that record came from an early ARD draft. ARD v0.91 names an SVCB record at _entries._agents.yourdomain.com instead, no ARD version defines a TXT format, and Lighthouse does not look at DNS at all. The TXT record does no harm, but treat DNS as optional.)

The expensive one is DNS-AID proper, which wants a ServiceMode SVCB record at _index._agents.yourdomain.com and the zone signed with DNSSEC. Check what your DNS provider actually supports before you plan around this. The site I was working on is on Netlify DNS, whose API reports its supported record types explicitly: A, AAAA, MX, CNAME, TXT, NS, SPF, ALIAS, SRV, NETLIFY, NETLIFYv6, CAA and HTTPS. SVCB is not on that list, and Netlify DNS offers no DNSSEC signing at all. So DNS-AID on that domain is not a configuration change, it is a zone migration, with the email and web resolution risk that implies.

That is a legitimate reason to leave a check failing. Which brings me to the last section.

The RSS feed you probably already have

Worth a line, because it keeps getting left out of these lists. A feed is the oldest machine-readable surface on the web and it still works. If you publish anything dated, publish a feed, and declare it in the head. It costs nothing and it is the one format every aggregator already understands.

What I did not publish, and why that is the right call

Five checks still fail on that site. Four of them are deliberate.

OAuth discovery metadata and Protected Resource Metadata. There is no authorization server and there are no protected resources. Publishing the metadata would advertise a registration endpoint that does not exist. There is also a specific reason not to: MCP clients probe /.well-known/oauth-protected-resource to decide whether a server needs auth, and a 404 is the specified way to say "it does not". Returning a document there would be actively worse than returning nothing.

The A2A agent card. A2A describes an agent that other agents delegate tasks to. This domain publishes information and a phone number. There is no autonomous worker behind it, and a card would claim a capability that is not there.

The commerce specs, x402, ACP, UCP, MPP and AP2. Nothing is sold on the site. The scanner correctly marks these neutral rather than failed, which is a nice piece of design on its part.

There is a real temptation, once a number is attached to something, to publish files until the number goes up. Resist it in this particular case, because every one of these documents is a promise made to a client that cannot see your face. A wrong Content-Signal is a preference nobody reads. A wrong MCP server card is an agent that tried to talk to you and failed. The scoring is a means, and the moment you publish a manifest for a thing that does not exist, you have made your site worse and your score better.

Gate it, because none of this is visible

Every one of these files is invisible in a browser. You will never notice a broken one by using your own site. Everything I published is generated from the same data file the pages read, and a build gate resolves every URL any manifest names, recomputes every sha256 against the file it points at, checks that every user-agent group carries its Content-Signal, and confirms the H1 of auth.md says what it has to say. It also refuses to let any internal field name from the source data escape into a published document, because the source is full of editorial notes sitting right next to the real values.

That gate took about an hour and it has already caught two of my own mistakes: it failed the build when I renamed id to identifier in one place and not the other, and it failed again when the server card advertised a tool the server did not implement.

Which is the whole point. These are not files you write once. They are an interface, and an interface without tests is a rumor.

If you are working out where a small site's effort actually pays back, The $20 Dollar Agency covers the wider version of this argument, which is that the cheap technical work compounds and the expensive marketing work usually does not.

Fact-check notes and sources

RFC 8288, Web Linking: the Link header format and the requirement that relation types be registered. rfc-editor.org/rfc/rfc8288
RFC 9727, api-catalog: the well-known URI and its use of the linkset format. rfc-editor.org/rfc/rfc9727
RFC 9264, Linksets: the linkset JSON structure used by the API catalog. rfc-editor.org/rfc/rfc9264
RFC 9309, Robots Exclusion Protocol: group matching, and the rule that a crawler obeys exactly one group. This is the normative basis for the Content-Signal placement trap. rfc-editor.org/rfc/rfc9309
RFC 9728, OAuth Protected Resource Metadata: the document deliberately not published here. rfc-editor.org/rfc/rfc9728
RFC 9460, SVCB and HTTPS resource records: the record type DNS-AID builds on. rfc-editor.org/rfc/rfc9460
Content Signals: contentsignals.org and the IETF draft, draft-romm-aipref-contentsignals.
DNS-AID: draft-mozleywilliams-dnsop-dnsaid, at datatracker.ietf.org. Still a draft; treat accordingly.
Agentic Resource Discovery (ARD) v0.91: a proposal dated August 26, 2026, at agenticresourcediscovery.org/spec, source in github.com/ards-project/ard-spec (spec/ard.md and spec/schemas/). Read October 13, 2026. Section 5.1 gives /.well-known/ard.json, rel="ard", Agentmap and the _entries._agents SVCB record; section 3.3 lists the standard media types.
Lighthouse ard-schema audit: the npm package lighthouse 13.5.0 (published September 18, 2026), files core/audits/agentic/ard-schema.js and third-party/ard/, which validate against ai-catalog.schema.json from ard-spec commit b76f235 (closed root and host). The ARD v0.91 port merged into Lighthouse's main branch on October 9, 2026 and ships in the 13.5.0-dev nightly. Read October 13, 2026.
MCP server card: SEP-2127, "MCP Server Cards", status Final, Extensions Track, extension id io.modelcontextprotocol/server-card. The wire format is in github.com/modelcontextprotocol/ext-server-card (schema.json and docs/discovery.md, snapshot 526201bb of August 12, 2026). The recommended card address is <mcp-url>/server-card; a .well-known path is listed as not recommended. Read October 13, 2026. (This bullet used to call the card SEP-1649, still in flight.)
October 13, 2026 update: the ARD section now says application/json is the wrong type for MCP and A2A entries, explains how the catalog and the cards fit together, and shows how to check the file. Sources, read October 13, 2026: ARD v0.91 and ard-spec; Lighthouse 13.5.0 on npm (the ard-schema audit, its discovery order and scoring); SEP-2127 and ext-server-card (card fields, no tools list, the recommended address); the A2A specification (sections 4.4.1, 5.7 and 8.6). The fleet numbers (31 sites, nine generator families, 29 of 31 scoring 1 at 18:40 UTC) were measured October 13, 2026 with a validator that matches Lighthouse row for row.
Agent Skills Discovery: the v0.2.0 RFC at github.com/cloudflare/agent-skills-discovery-rfc.
Markdown for Agents: Cloudflare's documentation.
Netlify DNS supported record types: read from the Netlify API's own supported_record_types field on the zone object on 2026-08-31. The absence of SVCB and of DNSSEC is that response, not an inference.
The byte counts (71,005 HTML against 5,743 markdown) are the live responses for one page of the site described, measured on 2026-08-31. The 3-pass-to-11-pass and Level 1 to Level 4 figures are before and after scans of that same site on the same day.

Agent readiness past llms.txt: eleven files, and the five that silently fail validation