Surveying a large family of official feeds, a meaningful minority returned HTTP 200 with an empty body. Not intermittently. Permanently. And several of the dark ones were among the highest-value endpoints in the whole set.
Every one of them passes a health check that asks whether the fetch succeeded, because the fetch did succeed. The endpoint is up. It is serving you nothing, forever, with a green status code.
An ingestion pipeline's health is not a property of its requests. It is a property of its inputs' coverage, and almost every default in such a pipeline silently decides something you will never see.
Assert on payload, never on status
The fix is small and the framing is the point. Two separate assertions, because they answer different questions.
// Per source: did this one give me anything?
assert(items.length > 0, `${source} returned an empty payload`);
// Across sources: is my coverage intact?
const ratio = sourcesWithItems / sourcesTotal;
assert(ratio >= 0.85, `coverage dropped to ${(ratio * 100).toFixed(0)}%`);
The second one matters more than it looks. A pipeline pulling forty sources where six have gone dark still produces output, still fills a page, and still looks busy. Nothing errors, because nothing errored. The only symptom is that some part of the world stopped appearing, and nobody notices an absence.
Store the per-source count on every run. The count over time is the diagnostic. A source that used to return 40 and now returns 0 is a story; a source that returns 0 today tells you nothing on its own, because it may never have returned anything.
Rolling window or archive, and the difference is unrecoverable
A high-volume feed that looked like an ordinary endpoint held only about a day of items at observed volume, with older entries dropping off permanently.
Nothing in the response says so. There is no field for it. And the consequence is severe in one direction: on a rolling window, any outage on your side is permanent data loss. A token expiry, a failed deploy, a rate-limit day, and that window is simply gone. On an archive, the same outage is a backfill.
You can measure it in one afternoon:
# Fetch twice, several hours apart, and compare the OLDEST item in each
curl -s "$FEED" | jq -r '[.items[].id] | min' # then again later
If the oldest identifier or timestamp advanced, it is a rolling window. The delta divided by your interval gives you the retention budget in hours, which tells you how long an outage can last before it costs you data you cannot recover.
Measure publication lag in the same pass by diffing an item's stated publication time against when it first appeared to you. That number decides how often polling is worth anything.
A robots gate quietly decides which markets you cover, and it takes the small ones first
An ingestion pipeline that honours robots directives, which it should, is also making an editorial decision it never announces.
Publishers who disallow crawling are not evenly distributed. In one survey the sources that declined were concentrated among smaller, more local outlets, because those are more likely to sit behind a platform that disallows by default. The pipeline was not biased by design. It inherited the bias of its sources' hosting choices.
The result is that coverage of smaller markets degrades first and silently, and a summary that says "we ingest 340 sources" is true while telling you nothing about which places have disappeared from your corpus.
Account for the disposition, not just the successes. Every source gets a recorded status: ingested, disallowed, unreachable, empty. Then report coverage grouped by whatever dimension your product cares about, rather than as one total. A single percentage across a whole corpus hides exactly the structure that matters, which is that one whole class of source is at zero.
Clustered input plus a naive slice excludes whole sources
The corpus feeding a set of generators was accumulated per feed, so items arrived grouped by source rather than interleaved, and the exporter preserved that order.
The generators took corpus.slice(0, N).
Sources sitting late in that order fell entirely outside the window. Not underrepresented. Absent. One had twenty items in the corpus and zero in every slice.
This is a two-line check and it is almost never present:
const inCorpus = new Set(corpus.map((x) => x.source)).size;
const inSlice = new Set(slice.map((x) => x.source)).size;
assert(inSlice === inCorpus, `slice dropped ${inCorpus - inSlice} source(s) entirely`);
Print the per-source count table alongside it. Any source with items in the corpus and zero in the slice is a silent exclusion, and the fix is either to interleave before slicing or to slice per source.
The general form: a truncation applied to sorted data is a filter on the sort key. If your data arrives grouped by anything, slice is not sampling. It is selection.
Independently capped layers, and a facet list built from a truncated slice
A request travelled through three layers, each of which independently clamped a limit parameter and sliced the result. Raising the cap at the outer layer alone was dead weight, because an inner layer had already sliced before the outer one saw anything.
Worse, a filter dropdown in the interface was populated by scanning the returned items for distinct categories. Built from a truncated slice, it lost whole categories, and always the smallest ones. A user could not filter to a category because the category was not in the list, because no item from it survived the slice.
# Find the binding cap: ask for far more than you need and count what returns
curl -s "$API?limit=5000&_cb=$(date +%s)" | jq '.items | length'
A count that plateaus below your ask means an inner layer is clamping, and you have to walk the chain. Bust caches at every hop while testing, or you will measure a cached copy of the old cap and conclude the change worked.
And a rule for interfaces: facet lists come from a distinct-values query over the total, never from the page of results you happen to be showing.
Record every dead source with its failure mode
A survey of free data sources produced as much value from its negatives as its positives, but only because each negative recorded the exact failure. The modes need genuinely opposite responses:
| Mode | What it means | Response |
|---|---|---|
BLOCKED-ENVIRONMENTAL |
refused or 403 from your network specifically | re-request from a second egress before declaring it dead |
LICENCE |
the free tier excludes your use | permanently unusable, stop re-evaluating it |
AUTH-GATED-ALTERNATIVE-EXISTS |
gated here, open elsewhere | go find the other door |
TOO-STALE |
real data, old data | usable only with a date attached to every claim |
One row per rejected source with the date and the verbatim failure. Without the mode, a future you re-tests all of them from scratch every six months, including the ones that can never work, and re-reaches the same conclusions at the same cost.
The environmental case deserves the extra care because it is the one that is wrong most often. A source that refuses one network and answers another is not dead, and declaring it dead removes a working source permanently on the strength of a local condition.
A default in an ingest path can disable a whole stage
One default value in an ingestion path silently disabled a translation stage for a dozen sources. The pipeline ran, reported success, and produced output for those sources in their original language, which is a plausible-looking result if you do not read that language.
This is the same shape as everything above. A default is a decision that nobody made and nobody reviews. Enumerate them: for each stage, what does it do when its input is missing, empty or unset, and is that behaviour the one you would choose if asked?
The specific check is to assert the stage's output differs from its input where it should:
assert(translated !== original || sourceLang === targetLang,
`translation stage produced identical output for ${source}`);
One caution about correcting public datasets
A short coda, because it is the sort of thing that is obvious once said and expensive if not.
Corrections you contribute to an open dataset are usually permanently and publicly attributed to your account. That is by design and it is a good property of those datasets. It also means that a body of corrections made on behalf of a client turns into a published, searchable list of that client's assets, tied to your identity, with dates.
Decide that is acceptable before you start, not after. For some work it is completely fine. For some clients it is not, and the alternative is to have the owner make the edits from their own account.
The through line
Every failure here produced a green run.
A 200 with no body. A slice that dropped a source. A cap that clamped upstream. A gate that excluded a class of publisher. A default that turned off a stage. In each case the job succeeded, the dashboard was fine, and the only evidence was an absence, which is the one thing no monitor alerts on.
Count what you got, per source, every run. Compare it to last run. Almost everything in this post is visible in that one table and invisible everywhere else.
If you want the wider map of running data-backed sites without a team, that is what I wrote The $20 Dollar Agency for.
Related reading
Every request succeeded and nothing reached the site for fifteen hours: the same class one layer down, in scheduling and batch collection.
A second check on the corrected text found more errors than the first: what happens to this corpus once it reaches a generator.
Your listing links to a working website that is not yours any more: third-party datasets from the other side, when the record is about you.
Three failures 773 milliseconds apart across three days is not the model: diagnosing the fetch layer when the failures are not silent.
Fact-check notes and sources
A 200 response may legitimately carry a zero-length body: status codes describe the request's outcome, not the presence of content, which is why payload assertions and status assertions answer different questions. RFC 9110, HTTP Semantics, section 15.3.1
A crawler must obey the most specific matching group in robots.txt: which is why disposition accounting has to be per source rather than inferred from a site-wide rule. RFC 9309, Robots Exclusion Protocol
204 exists precisely to signal a successful request with no content: a feed returning 200 with zero bytes is not using the status code that would have made this visible. RFC 9110, 204 No Content
Contributions to open collaborative datasets carry public attribution: edit history and contributor identity are part of the published record by design. OpenStreetMap contributor terms
This post is informational, not legal advice. The pipelines and sources described are anonymised and no client or dataset record is identified. Mentions of third parties are nominative fair use and no affiliation is implied.
