A second check on the corrected text found more errors ...

About thirty long-form pages came out of one generation wave. Every single one failed its automated fact check against the source pack, with five to twenty-nine findings each.

That was expected. What happened next was not. The findings went back for correction, the corrected text was re-checked with the identical checker, and it found five to seventeen more findings per page. Not leftovers. New ones.

The correction pass is a generator. So is the normaliser, so is the reformatter, so is the pass you added last month to fix a specific recurring problem. Every stage that touches generated text is a writer, including the ones you added to check, fix and clean up, and no property you established survives unless you assert it on the artifact that actually ships.

1. The correction pass is a generator too

The mistake is architectural, and it is easy to make because the stage is named after what it removes rather than what it produces. You wrote a "fixer", so you think of it as subtractive. It is a model call. It emits new text. New text has new errors in it.

One pass is never enough, and a fixed number of passes is a guess. Loop to a fixed point:

let round = 0, findings;
do {
  findings = await check(artifact);
  log(`round ${++round}: ${findings.length} findings`);
  if (findings.length) artifact = await correct(artifact, findings);
} while (findings.length && round < MAX_ROUNDS);
if (findings.length) throw new Error(`did not converge after ${round} rounds`);

Log findings per round, because the shape of the sequence is the diagnosis. A run going 22, 9, 3, 0 is converging. A run going 22, 14, 17, 12 is oscillating, which means your corrector and your checker disagree about the rule, and no number of rounds will fix that. Throwing on non-convergence is the point; a pipeline that silently gives up after three passes ships the third pass.

2. What the checker actually catches

The findings were not typos. They had a shape, and it is worth knowing because it tells you what to gate on.

Figures the source pack never contained. Concrete numbers that read as researched and came from nowhere.

Claims about who the customers are. Demographic and behavioural assertions with no basis in the inputs.

Comparisons over time when the pack held no time series. "Up from last year" where there is no last year in the data.

An attribute invented on top of a bare category. The pack said what something was; the text said what it was like.

The counted claims were consistently sound, which is the useful asymmetry: a count is a measurement over data the model can see, so it tends to be right. An adjective is not.

And one behaviour worth planning for specifically. Told not to name a particular entity, a model may satisfy the instruction and defeat its purpose by identifying the same entity another way, for instance by its street address. The rule was followed. The intent was not. Write explicit negative patterns for the routing-around case, not just the literal prohibition.

Two gates cover most of this:

# Every number in the output must be derivable from the pack
# (extract numerals, require an exact or computed match)

# No URL may originate from the model. It returns identifiers; the renderer builds links.
jq -r '..|strings' model_out.json | grep -Eo 'https?://[^"]+' | wc -l   # must be 0

That second one deserves emphasis. Let the model emit identifiers only and have the renderer construct every URL deterministically. A model-authored URL is a hallucination with a clickable surface, and it will be plausible.

3. The prompt is an artifact you can lint

Run your output linter over your prompt file. This sounds like a joke and it caught hundreds of defects.

One batch of writing prompts was itself drafted in British English. The models matched the dialect of the instructions rather than the dialect of the requested audience, and produced hundreds of off-dialect spellings and date formats. A later batch, whose prompt named the target dialect explicitly, produced zero.

# The same regex you run over dist/, pointed at the instructions
grep -RInE '\b(colour|behaviour|whilst|organis|realis)' prompts/

A hit in the prompt predicts hits in every artifact it generates.

The same method catches inert instructions. Prompts told a model to watch for specific named sources using their friendly names. The records stored those sources as domain strings. The model had no way to connect the instruction to the data, so an instruction that read as precise did nothing at all.

// Every literal entity token in the prompt must appear in the payload you actually send
for (const token of extractEntityTokens(promptText)) {
  const n = (JSON.stringify(payload).match(new RegExp(escapeRe(token), 'g')) || []).length;
  if (n === 0) throw new Error(`inert instruction: "${token}" never appears in the data`);
}

4. Assert on the shipped file, not on the log

Generated text was normalised partway through the pipeline for house style. A later re-check pass then replaced whole paragraphs with corrected versions from a model that had never seen the normalisation rules. The normalised spellings and dates came back.

The normaliser ran. Its log said done. Both facts were true and the shipped file was wrong.

Make the normaliser an assertion over the output directory, not a step in the middle, and make it exit non-zero:

grep -RInE '\b(colour|behaviour|whilst|organis|realis)' dist/ && exit 1
grep -RInE '\b[0-9]{1,2} (January|February|March|April|May|June|July|August|September|October|November|December) [0-9]{4}\b' dist/ && exit 1

A step in the middle establishes a property. Only an assertion at the end proves it survived.

Two adjacent traps in the same family, both about batch patches over generated pages.

A String.replace that matches nothing returns its input and reports nothing. Pages shipped missing entire blocks: one set with no social-card tags and titles ending in a bare pipe character. Assert the match count, and assert it exactly, because two matches where you expected one is a different bug and usually a worse one:

const n = (src.match(re) || []).length;
if (n !== expected) throw new Error(`${file}: expected ${expected} matches, found ${n}`);

A patch that assigns a literal version number rolls advanced targets backwards. Normalising a cache version by writing a literal downgraded every target that had already moved past it, which for a cache version means clients keep serving the old bundle. Parse and compare rather than assign:

if (!(next > current)) throw new Error(`${file}: refusing to move ${current} -> ${next}`);

5. Order, identity, and the two runs nobody does

A pipeline whose stages consume each other's output needs two runs to converge. One stage wrote pages that conditionally linked to counterparts in a second-language edition, gated on an existence check. The stage that wrote those counterparts ran later in the same build. On a clean tree the check was false for every page, so the links were omitted, and they appeared only on a second build against a warm tree. Which is what everyone does locally, and never what CI does.

rm -rf dist && build && cp -r dist dist.a && rm -rf dist && build && diff -r dist.a dist

Any difference proves the pipeline is order-dependent and has not converged. Wire it as a gate.

Key parallel results by the id you dispatched, never by a field the worker reports. Verification work was fanned out to workers that each returned a result object naming the job it had checked. The collector keyed a dictionary on that self-reported field. One worker reported a sibling's name, so its result overwrote the sibling's and one job's verification silently vanished.

// Zip by index or by the dispatched id. Never read the key out of the payload.
const byId = jobs.map((job, i) => [job.id, results[i]]);
if (results.length !== jobs.length) throw new Error('worker count mismatch');

An output keyed by date, written by a job that fires more often than the date changes, destroys itself. A generation job ran four times a day and wrote to a blob keyed only by date. Runs two through four paid full inference cost to overwrite equally good output. The test is mechanical: fires_per_period > 1 && key_granularity === period is a destructive overwrite. Grep your writers for date-string keys and cross-reference each one's schedule.

Two deploy paths that disagree about what exists. The same site was deployed sometimes by CLI, which uploads the working directory including untracked files, and sometimes by pushing a branch, which deploys only committed state. Over time real dependencies and dozens of generated files lived on the server and on disk and were never committed. The routine action then becomes the destructive one.

Before switching mechanisms, diff repo against live on content, not on bytes. Normalise line endings and trailing newlines first. A raw checksum comparison flagged 231 changed files when 27 differed in substance, and that noise is exactly why people skip the check that would have saved them.

6. Instruments that measure themselves

Three failures where the tooling, not the pipeline, was the thing that was wrong. They are worth a section because each one produces confident output.

A harness that returns the same value for every input is measuring itself. An automated check reported an identical failing ratio, to two decimals, across four genuinely different variants. The harness was reading a property that is empty in one common case and comparing against a default. Assert that at least one variant differs, and feed the harness a known-good and known-bad fixture on every run.

An upstream that ignores your parameter still returns 200. An API accepted a mode parameter and returned confident, well-formed, identical results for five different spellings of it. It was serving one mode for all of them, and the derived figures went into generated prose. Two probes fix it: a differential assertion in the integration test, requesting the same job under two values that must differ and failing if the responses are equal; and a plausibility band on every derived quantity before it reaches a prompt.

A wrapper that returns null on every failure collapses five causes into one. Timeout, content filter, auth failure, parse failure and empty completion all came back as null, and none called the failure recorder. Diagnosing a multi-day outage then means doing arithmetic on counters. Count distinct recorded failure reasons over the last thirty days: if the answer is one, or the field does not exist, your wrapper is blind.

And the diagnostic I would most like people to steal:

A failure that repeats within a one-second spread on different days is a transport wall, not the model. A scheduled job failed every run for days, and attention went to the model, the content filter, and the app's own timeout setting. Three failures at 302,506 and 302,695 and 303,279 milliseconds across three different days is a 773 millisecond spread. A slow model varies by tens of seconds. A fixed limit does not vary. It was the default header timeout in the runtime's HTTP client, and the app's own abort was set above it, so that knob was dead code.

jq -r '.runs[]|"\(.date) \(.status) \(.ms)"' health.json | sort -k3 -n

Record elapsed milliseconds for every model call. Sort the failures. If they cluster inside a second, stop reading model documentation and go read your client's defaults.

Define what a healthy run looks like

Health was originally judged by whether the endpoints responded. They always did, because they served the last successful artifact. A pipeline that stopped producing anything looked identical to one that was working.

The real signal was a shape: every surface carrying one generation timestamp within about a second of the others, today's date, and exactly the expected number of entries.

assert(max(generated_at) - min(generated_at) < 5000);
assert(dateOf(generated_at) === today);
assert(distinct(batch_id).length === EXPECTED_SURFACES);

Run it on a schedule offset from the generation cycle, so it is asking a question the pipeline has already had time to answer. And alert on the count of items collected today rather than on batch status, because a batch that finished is not content that shipped. One overnight run completed with every request succeeding and nothing reached the site for fifteen hours: the submit side was healthy and the drain side never fired.

The three rules

Every stage is a writer. Including the checker, the corrector and the normaliser. Re-run your check on the corrected artifact, and treat non-zero as proof.

Assert on the file that ships, never on the log line. A middle stage establishes a property. Only an end-of-pipeline assertion proves it survived the stages after it.

Make every derived signal prove its own independence. If a confidence score, a dissent, or a cross-check can be produced by one call, it is decoration. Write provenance onto the artifact and assert it is non-null across consecutive runs.

If you want the wider map of running this kind of work without a team, that is what I wrote The $20 Dollar Agency for.

Fact-check notes and sources

A batch response reports why it stopped: stop_reason distinguishes a completed answer from one truncated at the token ceiling, which is why HTTP status alone cannot tell you an item was lost. Anthropic API, handling stop reasons

Streaming responses omit usage unless you ask for it: token accounting arrives in the final chunk only when the request opts in, so converting a call to streaming can silently end cost attribution. OpenAI API reference, stream_options

Node's HTTP client has its own header timeout independent of your abort controller: undici defaults headersTimeout, so an application-level timeout set above it never fires. undici Dispatcher options

String.prototype.replace returns the original string when the pattern does not match: there is no error and no signal, which is what makes an unmatched batch patch a silent no-op. MDN, String.prototype.replace

This post is informational, not legal or engineering-consulting advice. The pipelines described are anonymised and no client, brand, vertical or dataset is identified. Mentions of third parties are nominative fair use and no affiliation is implied.

A second check on the corrected text found more errors than the first