A scheduled generation job failed every run for days. Attention went where attention goes: the model was struggling, the content filter was rejecting something, the timeout needed raising.
Then I sorted the failures by elapsed milliseconds.
302,506 ms
302,695 ms
303,279 ms
Three different days. A spread of 773 milliseconds.
A model that is genuinely slow varies by tens of seconds. It varies with payload size, with load, with what it decides to say. It does not land within a second of the same number three days running. That shape is a fixed limit, and a fixed limit belongs to a layer you configured once and forgot.
It was the default header timeout in the runtime's built-in HTTP client. The application had its own timeout set well above it, which meant that knob was dead code and had been the whole time. Raising it, which was the obvious remedy and the one being discussed, would have changed nothing.
The lesson generalises past that one library. When a system fails, the shape of the failure tells you which layer to look at, and most instrumentation destroys the shape before you ever see it.
Record elapsed time on every call, then sort the failures
This is the whole technique and it costs one field.
jq -r '.runs[] | "\(.date) \(.status) \(.ms)"' health.json | sort -k3 -n
Read the spread among the failures:
Clustered inside a second or two. A fixed limit. Look at timeouts in your HTTP client, your proxy, your gateway, your platform's function limit. Something is cutting the call, and it is not the thing on the other end.
Clustered around a round number. Same conclusion, and the round number usually names the layer. 30,000. 60,000. 300,000.
Spread over tens of seconds. Now it may genuinely be the far end.
Bimodal. Two different failure paths wearing one error message, which is the next section.
The reason nobody does this is that elapsed time feels like a performance metric, so it lands in a dashboard nobody reads during an outage. It is a diagnostic. Put it in the failure record itself.
Your wrapper collapsed five causes into one null
The reason the duration signature was the only clue available is worth its own section, because it is extremely common.
The model wrapper returned null on failure. It returned null for a timeout, for a content-filter rejection, for an auth failure, for a parse failure, and for an empty completion. It never called the failure recorder that existed a few hundred lines away in the same codebase. Downstream, that produced one hardcoded sentence in the health blob.
Five distinguishable causes, one indistinguishable symptom. Diagnosing a multi-day outage then means doing arithmetic on counters, which is what pushed me to elapsed times in the first place.
The check is one query:
SELECT reason, count(*) FROM failures WHERE day > now() - interval '30 days' GROUP BY reason;
If the answer has one row, or the column does not exist, your wrapper is blind. Then:
grep -n 'return null' model-wrapper.mjs
Every early return should carry a distinct code, and every one should reach the recorder. A catch that returns a falsy value is not error handling. It is error deletion.
The fallback tier that is wired in code and dead in configuration
A fetch layer implemented three tiers: direct, then two proxies. It had been inert in one consuming path for months.
The proxy function returns null the moment its credential variable is empty, and one workflow never passed the credentials into its job step. Success-rate monitoring could not see this. The direct tier succeeded most of the time, so the rate looked healthy, and when it failed the request just failed, exactly as it would if the fallback had run and also failed.
Count tier usage, not just outcomes, and assert the combination that cannot happen:
assert(!(failCount > 0 && tier2Count === 0 && tier3Count === 0),
'failures occurred but no fallback tier was ever used: the chain is unwired');
A fallback tier with zero usage across a full run that also produced failures is not unneeded. It is disconnected. Add a startup assertion that every credential the chain reads is non-empty, so the process refuses to start rather than quietly degrading to one tier.
An upstream that ignores your parameter and returns 200
A public API accepted a mode parameter in the path. It returned confident, well-formed results for five different spellings of it, and the results were identical. Same distance, same duration, every time. It was serving one mode for all of them, and the derived figures were going into generated prose that implied something specific and wrong.
Nothing errored. Nothing warned. A wrong parameter and a right one are indistinguishable when the response is well formed either way.
Two probes:
// Differential assertion: two values that MUST differ
const a = await api({ mode: 'walking' });
const b = await api({ mode: 'driving' });
assert(a.duration !== b.duration, 'upstream is ignoring the mode parameter');
And a plausibility band on every derived quantity before it reaches anything that publishes:
assert(walkMinutes / km > 8 && walkMinutes / km < 20, 'implausible walking pace');
The general rule: if a parameter changes nothing observable, you have not proved it works, you have proved nothing. Vary it deliberately and assert the answer moves.
The fix that silently killed the billing
The remedy for the timeout was to convert the call to streaming, which keeps the connection producing bytes and never trips a header timeout. That worked. It also had two traps.
The usage block only arrives if you ask for it. In a streaming response, token counts come in the final chunk, and only when the request opts in. Without that, the cost-attribution recorder receives nothing, and the billing rows that the entire spend-attribution system depends on stop being written. Nothing errors, because nothing is missing from the response you asked for.
-- After the switch: accounting rows must still equal successful runs
SELECT count(*) FROM usage WHERE day = CURRENT_DATE; -- not zero
The helper had other callers. Before changing shared plumbing, enumerate the blast radius:
grep -rn 'chatJsonFallback(' --include=*.mjs .
A behaviour change in a shared helper is a change to every call site, and the ones that break are the ones you were not thinking about.
Price per token is not price per run
The other proposed remedy was to repoint the job at a cheaper, faster model.
Measured on real production-sized payloads, the fast tier cost more per run than the batch price it was meant to undercut. It forced reasoning on and locked temperature, which lengthened exactly the decode phase that was the problem.
const costPerRun = (inTok * inPrice + outTok * outPrice) / 1e6;
Compute that from measured token counts on a full-size payload, per candidate, and compare against what you are actually paying now, including any batch discount. Assert the candidate is cheaper per run and that its latency profile fits the limit you are up against. A per-token price is a unit price for a quantity you have not measured.
One related trap while wiring an alternative: reasoning-style endpoints put the answer in the standard content field while a separate field carries the chain, so reading the wrong one gets you the thinking instead of the answer. They also reject a temperature other than the default, and they truncate mid-answer under a token ceiling. All three read as poor model quality if you are not asserting on the envelope:
assert(res.choices[0].finish_reason !== 'length', 'truncated at the token ceiling');
The habit
Put elapsed milliseconds and a distinct failure code on every external call you make. That is two fields.
With them, an outage is a sorted list and a group-by. Without them, it is a story you tell yourself about the model, and the story is usually wrong because the model is the most interesting component and therefore the first suspect.
The interesting component is rarely the broken one. The broken one is usually a default you never set, in a layer you did not write, doing exactly what it documents.
If you want the wider map of running this kind of infrastructure solo, that is what I wrote The $20 Dollar Agency for.
Related reading
A second check on the corrected text found more errors than the first: the pipeline this job lived in, and the stages that undo each other.
The check was green because it measured a state no visitor is ever in: the same instinct applied to build and deploy checks.
Your confidence score is one model rating its own homework: what the null-returning wrapper does to a signal that depends on a second model answering.
Every request succeeded and nothing reached the site for fifteen hours: the scheduling half of the same system.
Fact-check notes and sources
Node's HTTP client carries its own header timeout: undici, which backs fetch in modern Node, sets headersTimeout independently of any AbortController your application configures, so an application timeout set above it never fires. undici Dispatcher options
Streaming responses omit usage unless requested: token accounting arrives in the final chunk only when the request opts in via stream_options. OpenAI API reference
finish_reason distinguishes a complete answer from a truncated one: a response cut at the token ceiling is still a successful HTTP call. OpenAI API reference, the chat completion object
Stop reasons are the honest signal on the Anthropic API too: stop_reason reports why generation ended, including max_tokens. Anthropic API, handling stop reasons
This post is informational, not engineering-consulting advice. The systems described are anonymised. Mentions of third parties are nominative fair use and no affiliation is implied.
