An overnight batch completed. Every request in it succeeded. Nothing reached the site for fifteen hours.
The submit side was healthy. The collect side never fired. And because the endpoints kept serving the last successful artifact, everything downstream looked exactly as it looks on a good day. The pages were up. The health check was green. The content was yesterday's.
"It responded" is not "it produced", and a health check that asks the first question will pass forever.
The batch finished, and the drainer is a separate failure domain
The first thing to internalise is that submission and collection are two systems that fail independently, and only one of them is the one you instinctively monitor.
Batch status is a property of the provider. Content on your site is a property of your drain path. A provider dashboard showing every request succeeded is telling the truth and answering the wrong question.
Two independent causes landed that same night, which is the sort of thing that only happens when a class of failure has been latent for a while.
Alert on the count of items collected today, not on batch status:
// A per-day counter, checked after the expected drain window
if (entriesToday === 0 && now > expectedDrainBy) page('nothing collected today');
And cross-check against the provider's own list endpoint for authoritative processing status, because your local pending marker can lag or be stale. Your marker records what you believe. Their endpoint records what happened.
A scheduler config that is silently ignored rather than rejected
Several scheduled workers combined a configuration option with a filename or type convention that the platform does not support together.
The platform did not error. It silently dropped the schedule.
The function existed. It deployed cleanly. It was invocable by hand, and every manual test passed. It simply never ran on its own, and nothing anywhere said so.
This is the single most important habit in this post:
Never verify a schedule by reading its configuration. Verify it by counting invocations at the expected fire times.
# Did it actually run in the last N expected slots?
# One invocation per slot, or the schedule is not what you think it is.
Reading the config confirms your intent. Querying the run history confirms reality, and those two disagree more often than anyone expects, because a scheduler that silently declines a config is not an unusual platform behaviour.
Pair it with a freshness watchdog that probes the output surfaces on its own independent cadence. If a schedule dies, the watchdog notices through the artifact rather than through the scheduler, which means it catches causes you have not thought of.
No retry around the submit call turns a 5xx window into an all-day outage
The provider's batch-submission endpoint returned intermittent HTTP 500 with an empty body over roughly a five-hour window.
The submit function had no retry. So the scheduled fire failed, every keepalive that landed inside that window failed, the pending marker stayed null, and the drain had nothing to look for. One transient upstream condition became a full day with no content.
Adding a retry is obvious and is not the lesson. The lesson is how to tell an upstream problem from your own payload while it is happening, because that determines whether you retry or fix:
Alternate a one-request probe batch against your real N-request batch. If the small one also flips between 200 and 500, it is the server. If the small one is consistently fine and the large one is not, it is your payload or your size.
And persist the error list, not just a count:
errors.push({ at: Date.now(), status: res.status, body: text.slice(0, 200) });
Counts undercount. A retry loop that eventually succeeds contributes zero to a failure counter while telling you something important about the window you are in.
The item that vanished from a batch reporting 100 percent success
Two generated surfaces were configured with a max-output-tokens ceiling half that of their siblings. Their responses hit the ceiling and truncated mid-string, dropping a comma. The JSON was unparseable, a truncation-repair helper could not salvage it, and those items were dropped.
The batch reported complete success, because from the provider's side every request did succeed. The failure was in parsing a valid response that had been cut short.
Two assertions at collect time, and they catch different things:
// On the response: the honest signal is the stop reason, not the HTTP status
const truncated = results.filter((r) => r.result?.message?.stop_reason === 'max_tokens');
assert(truncated.length === 0, `${truncated.length} response(s) hit the token ceiling`);
// On the pipeline: everything submitted must be accounted for
assert(writtenCount === submittedCount,
`submitted ${submittedCount}, wrote ${writtenCount}`);
The second is the one to add first. A count that goes in and a count that comes out, compared, catches every silent drop regardless of cause, including the ones you have not seen yet.
Log which identifiers went missing, not just how many.
A date-keyed write plus a more frequent schedule destroys itself
One generation job ran four times a day and wrote its result to a blob keyed only by date.
Runs two, three and four did not accumulate anything and did not version anything. Each overwrote the previous, paying full inference cost to replace equally good output. If a later run happened to be worse, it destroyed a better artifact.
The test is mechanical and you can run it across a whole codebase:
// A destructive-overwrite bug, stated as a rule
if (firesPerPeriod > 1 && keyGranularity === period) throw new Error('overwrite');
Grep your writers for keys built from a date string, and cross-reference each one against its schedule. Either the key needs finer granularity, or the job needs to run less often, and deciding which is a product question rather than a bug fix.
Define the fingerprint of a healthy run
All of the above is really one problem: health was judged by whether endpoints responded, and they always did, because they served the last good artifact.
The real signal turned out to be a shape. Every surface carrying one generation timestamp within about a second of the others, today's date, and exactly the expected number of entries.
Write the fingerprint as assertions over the served output:
assert(max(generatedAt) - min(generatedAt) < 5000, 'surfaces are from different runs');
assert(dateOf(generatedAt) === today, 'serving a stale artifact');
assert(distinct(batchId).length === EXPECTED_SURFACES, 'a surface is missing');
assert(entriesPerSurface.every((n) => n > 0), 'a surface is empty');
Three properties of that check are what make it work:
It runs on a schedule offset from the generation cycle, so it asks its question at a time when the answer should already exist.
It reads what is served, not what was written. Those differ whenever caching, a CDN, or a deploy path is involved, and every one of those is in play.
It opens or closes an issue by itself. A check that needs a human to read its output is a log line, and log lines are read after somebody notices, which is the thing that failed here.
The pattern
Every failure in this post produced a green run. Submitted successfully, deployed cleanly, responded with 200, reported complete.
The unifying question to ask of any pipeline you own is not "did it error". It is:
If this stopped producing output right now, how long would it be before anything told me?
If the honest answer is "when somebody looks at the site", you have the same setup that ran fifteen hours on yesterday's content while every indicator stayed green.
Three things, in order of value: count what came out and compare it to what went in; verify schedules by counting invocations rather than by reading config; and assert freshness on the served artifact from a separate cadence.
If you want the wider map of running scheduled, generated systems without a team, that is what I wrote The $20 Dollar Agency for.
Related reading
A 200 with an empty body is the most expensive response you can get: the same class one layer up, on the way in rather than the way out.
Three failures 773 milliseconds apart across three days is not the model: what to do once you know a scheduled job is failing and want to know which layer.
A second check on the corrected text found more errors than the first: the stages between collection and publication.
The check was green because it measured a state no visitor is ever in: the general form of a check that cannot fail.
Fact-check notes and sources
Stop reasons distinguish a complete response from a truncated one: a response cut at the token ceiling is a successful request, which is why batch success rates cannot see it. Anthropic API, handling stop reasons
Batch APIs report per-request status separately from the batch's own state: a batch can end with requests in differing states, so the batch object alone is not a completeness signal. Anthropic Message Batches API
5xx responses are defined as server-side conditions and are the retryable class: which is why a probe distinguishing your payload from the server's condition is the right diagnostic before adding retries. RFC 9110, Server Error 5xx
Scheduled functions are configured declaratively and can fail to register: platform scheduling depends on the config being both valid and supported in combination, and support varies by function type. Netlify scheduled functions
This post is informational, not engineering-consulting advice. The systems described are anonymised and no client or vendor is identified. Mentions of third parties are nominative fair use and no affiliation is implied.
