When a Data Source Refuses You, Look for the Bulk File

I spent a while last year adding egress paths to a data pipeline. A second server in a different cloud. A third in a different country. Better browser automation. Each one recovered a handful of sources and cost real time to build.

Then I looked properly at the feed that was failing worst, and found that the publisher gives the entire dataset away as plain files on a different host, with no key, no quota, and no gatekeeping of any kind. The query API I had been fighting was the rationed front door. The bulk files were the loading dock, and the loading dock had been standing open the whole time.

Switching took an afternoon. Topic coverage went from 2 of 7 to 7 of 7. Runtime went from six and a half minutes of mostly failure to about four seconds. The pipeline now reads roughly 15,000 documents per run instead of at most 50 per topic. I added no servers, no proxies, and no credentials.

That is the most useful thing I know about collecting public data, so it goes first: before you build anything to get past a refusal, find out whether the refusal is even about you.

The five reasons a request fails, and why they need different fixes

Almost all the wasted effort in this area comes from misdiagnosing which of these you have. They look similar from the outside and they respond to completely different treatment.

1. Your address has a bad reputation. Whole ranges of server addresses are scored as commercial hosting and treated with suspicion. This is not about how your request looks. It is about where it comes from. The tell is a hard, instant, identical refusal from every machine you own.

2. Your location is wrong. The content is fenced to a country or region. Someone else sees the page fine and you get a redirect or a polite nothing.

3. Your client looks automated. Headers, connection characteristics, or the browser environment give you away. A plain script gets refused and a real browser works.

4. There is a challenge in the way. An interactive puzzle or verification step. Either you satisfy it or you stop.

5. Nothing is blocking you at all. You get a completely successful response and the information simply is not in it, because the page assembles itself after loading. This one is dangerous precisely because it looks like success.

The expensive mistake, and I have made it, is treating the first cause as the third. There is a large and enthusiastic body of writing about making automated clients look more human. None of it addresses reputation. A flawless client running from a scored address still loses, and no amount of tuning changes that. If you get the same refusal from every machine you control, stop adjusting the client. You need a different kind of address, or you need to stop knocking on that door.

Try the front door first, seriously

I want to belabour this because it is the step people skip, including me.

When a source rate limits you aggressively, that is often not a defence against you personally. It is a publisher protecting a query service that was never designed to serve the volume people want from it. Publishers who do that very frequently also publish the underlying data in bulk, because it is cheaper for them than serving a million small queries. The bulk route usually has no rate limit, no key, and no gatekeeping, because it costs them a static file server and nothing else.

The signal I missed for months was sitting in the error responses themselves. The refusals were not generic. They contained a note pointing heavy users toward the bulk datasets. I had been retrying the message instead of reading it.

So: read the body of the error, not just the status. Then go looking in these places, roughly in this order.

The publisher's own bulk or archive host. Often a plain directory of dated files, updated on a schedule, sitting on a subdomain that nothing guards.
Public dataset programmes at the large cloud providers. Several host major public corpora as directly queryable tables with a free monthly allowance. This is frequently the same dataset the API is rationing, and you can ask it questions the API cannot answer.
Open data registries and mirrors, where projects deposit copies of their own datasets.
Web archives, for material that blocks you today but was captured earlier. Archive services rate limit too, and sometimes globally rather than per visitor, so space those requests out.
Government and institutional interfaces with free keys. The coverage is enormous, the limits are generous, and there is no adversarial machinery at all. If you are pulling information out of an agency's web pages, check whether that agency publishes the same information properly. It usually does.

There is a real cost to the bulk route and I will not pretend otherwise. You download much more than you need and filter locally. In my case that meant about 50 MB per run instead of a few kilobytes. That trade was overwhelmingly worth it, because 50 MB of reliable data beats a few kilobytes of intermittent data every single time. Check the shape of your own numbers before you assume the same.

If you do need more than one way out, each addition should change exactly one thing

Sometimes the front door genuinely does not exist. Fine. Then you want more than one path to the outside world, and the useful way to think about them is that each one should change a single variable.

The ladder, roughly in order of cost:

Direct requests with good manners. One request per second per host, a couple of concurrent connections at most, and honouring the wait interval when a server asks you to slow down. This handles far more of the world than people expect, and it is free.

An edge function acting as a relay. Different address, different network. Cheap, quick to stand up, and it recovers a useful slice of sources whose objection was to your particular server.

A second relay with a different provider. Another network with a different reputation profile. Also cheap.

A machine pinned to a specific region. This is the only thing on the list that addresses geography.

A machine on a home internet connection. This is the only thing on the list that changes the kind of address you have, and it is therefore the only real answer to cause one.

A real browser. Addresses cause three and cause five at once.

A challenge service. Addresses cause four, at a small per-item price.

Here is the part worth internalising: the first three all give you different addresses of the same kind. If a source objects to commercial hosting, adding a fourth provider of commercial hosting will fail in exactly the same way. I have watched that assumption eat a week. When the refusal is identical everywhere, the ladder has nothing left to offer and you should go back to the previous section.

Two things I believed that turned out to be false when I measured them

Both of these are widely repeated. Both were wrong on my equipment, and I would not have found out without testing.

Newer automation was not stealthier

The current advice is that older browser automation patches are well known and detected, and that you should move to a newer generation of tooling built around avoiding a specific detection channel. I believed this. I installed the newer tooling and put it head to head against the older library on a list of sources I already had ground truth for.

The older library won on every single source. The newer one was refused, without a visible window, on sources where the older one sailed through. I tried five different configurations of the newer tooling, including two different browser builds. All five were refused. Only running it with a visible window worked, and running with a visible window was the exact thing I was trying to get away from, because it cannot run unattended on a server or a scheduler.

I checked my own control before believing it, which mattered. I confirmed the older library really was running without a visible window rather than quietly falling back, by inspecting the actual process arguments. It was.

The likely explanation makes sense in hindsight. The older library actively rewrites a long list of environment properties. The newer tooling deliberately does not, because its whole design principle is to avoid the detectable act of rewriting things. The sources I care about were checking for the environment properties, not for the rewriting. Different threat model, opposite result.

I uninstalled the newer tooling and kept what I had.

The transferable lesson is not "that tool is bad." It is that a recommendation derived from someone else's target list is not a measurement of yours. Keep a short list of sources you know the current status of, and re-run it before you swap anything. It takes twenty minutes and it is the difference between an upgrade and a regression.

Browser rendering at the edge did not inherit the edge's reputation

Several providers now offer a real browser that runs inside their own network. My reasoning was neat and, it turned out, wrong: my relay already had a well regarded address but could not run page code, and my local browser could run page code but had a poor address. A browser inside the relay's network should give me both.

It gave me one. When I checked the address each path actually presents to the outside world, the plain relay and the browser service, on the same account and in the same deployment, came out as two completely different addresses. They are separate pools.

The consequences were immediate and measurable. One government catalogue that served my plain relay a normal page served the browser service a bare not-found response. Two commercial sites put the browser service through a verification step that the plain relay never saw. Meanwhile the browser service did solve the problem it was actually suited to, turning a nearly empty shell of a page into real readable content.

So it earns its place, just not for the reason I built it. It runs page code from a machine that is not my desk, on a schedule, without occupying the computer I am working on. That is genuinely useful. It is not a reputation upgrade.

The check takes one minute and I recommend running it before you build anything on this assumption. Ask an address echo service what address it sees, once through your plain path and once through your rendering path, and compare the two answers. If they differ, you have added the ability to run page code. You have not added credibility.

The failures that report success

These cost me more time than every refusal combined, because nothing alerts on them.

A pipeline can pass with no data in it. If your scheduler is configured so one flaky source cannot poison the rest, a total failure and a clean run produce the same green result. The fix is a final step that inspects each source's real outcome and fails the job when a source produced nothing. One detail decides whether that step works: make an unresolved outcome a failure too. If the check only recognises success and failure, then a renamed or mistyped step yields neither, falls through both branches, and silently drops that source from coverage. That is the original bug wearing the costume of the thing built to catch it.

An empty error list is not health. A fetch that succeeds and then extracts nothing leaves you with no errors recorded and every value empty. Anything that checks whether errors occurred will report that all is well. Record why something is missing, and distinguish "could not retrieve" from "retrieved fine, found nothing." Those have different causes and different fixes.

A live response is not proof the origin works. Content delivery layers commonly serve a cached copy when the origin behind them is failing. Your uptime check passes for a week while the thing behind it is broken. Check freshness, meaning the age of the data against a budget you set in advance, rather than checking reachability.

A page that assembles itself after loading looks exactly like a win. This one bit me again while writing this piece. My detector for empty shells only triggered on large pages, on the reasoning that a shell is a lot of code wrapping very little content. Then I hit a source serving a shell of under three kilobytes that reduced to about 126 characters of readable text. It sailed through as a success, the cascade stopped there satisfied, and the tier that could actually have rendered the page never ran. The fix was to stop reasoning about size and look for the signature instead, meaning the apologetic notice that frameworks emit when their code never ran, with a length guard so a page that merely mentions the phrase survives.

And the one that produces confidently wrong conclusions: always run a control. Before you report that a source filters a term, lacks a record, or has removed something, request something you are certain it has. Compare the two responses. Twice in one week I nearly published a finding that evaporated under this test. In the first case a search endpoint returned an identical short stub for my term, which looked like filtering, and returned the same identical stub for the control, which meant the endpoint was simply not functional for anonymous visitors and a different path worked fine. In the second, a marketplace appeared to have removed an entire product category, and had not. It returns a byte-identical shell to every term including the control. Absence of evidence requires a working instrument, and the control is how you prove the instrument is working.

If you run a relay of your own, two things are not optional

I will keep this short because it is well covered elsewhere, but both get skipped constantly.

A relay that fetches a supplied address is, by definition, capable of fetching addresses you did not intend. On a rented server that reaches the provider's internal configuration service. On a machine at home it reaches your router's administration page, your storage device, your printer, and your cameras. The naive protection is a check on the text of the address, and it is close to worthless. The case that defeats it is a perfectly ordinary public name that resolves to your own machine. You have to resolve the name and judge the resulting addresses, follow redirects manually and re-check every hop, cover the reserved ranges including the ones people forget, and handle both address formats including the mapped forms. I keep a probe list and run all of it against any relay before it carries traffic.

Order your protections so the cheap ones run first. Put the per-visitor limit ahead of the credential check. If the credential check runs first, a flood of unauthorised requests still consumes billable capacity purely to be told no. Health checks stay free and unlimited so your monitoring never trips your own limiter. Cache successful responses hard, because that single decision reduces effective load more than the other two together. And use a separate credential for each relay, because one shared secret across all of them means a single leak burns everything and tells you nothing about where it leaked.

What I would actually do, in order

If a source is refusing you, work down this list and stop as soon as one works.

Read the error body. Not the status. The body. It may tell you exactly where to go instead.
Look for bulk files, a public dataset copy, or an official interface. Give this a solid hour before writing any code. It resolves this more often than everything below it combined.
Slow down and identify yourself. Include a way to be contacted. Anonymous heavy traffic gets discarded silently. Traffic with a contact address gets a warning and a second chance.
Check whether every machine you own gets the identical refusal. If yes, this is about the kind of address you have, and only a genuinely different kind of connection helps.
Check whether the content is even in the response. If the answer is no, you need something that runs page code, not a different address.
Only now consider more infrastructure, and add one variable at a time.

The uncomfortable summary

The most effective work I did on this was deleting things. I removed a query integration and replaced it with file downloads. I uninstalled a tool that benchmarking showed made things worse. I deleted an approach entirely once it became clear its blocker was an account I could not verify rather than any code I could write.

What survived is smaller, faster, and considerably more honest about when it has failed. The pipeline that used to report a cheerful green result while carrying almost no data now refuses to pass unless it actually collected something, and the thing it collects arrives from a public file server that has never once turned me away.

Most blocked requests are not a proxy problem. Check that before you go shopping.

Fact-check notes and sources

The opening example is a large public events dataset that publishes its entire corpus as timestamped bulk files on a plain file host, refreshed every fifteen minutes, sitting alongside the rationed query interface that most people use instead. I have deliberately left it unnamed. The publisher is doing nothing wrong here, the bulk route is the one they would prefer heavy users take, and their own error responses say so plainly. Naming them would turn a lesson about my own misreading into a complaint about their service. The coverage, document count and runtime figures are from my own before and after runs of the same pipeline on the same machine, not from published benchmarks. Check the file layout documentation of whichever dataset you are working with, since the record formats differ and a misread column is the most likely way to get a confident wrong answer.
Public dataset programmes at cloud providers: Google BigQuery public datasets and the Registry of Open Data on AWS. Both carry free query or access allowances; check current terms, which change.
Archived copies of pages come from the Internet Archive Wayback Machine, whose availability interface is documented at archive.org/help/wayback_api.php.
The robots exclusion standard is now a published specification, RFC 9309. It is worth reading rather than assuming.
Asking a server to be told when to retry is standardised as the Retry-After header in RFC 9110 section 10.2.3.
The relay safety problem is Server Side Request Forgery. The OWASP prevention cheat sheet covers the defences, and the bypasses I describe are the standard ones.
The reserved address ranges people miss most often are carrier grade translation space, RFC 6598, and the link local range used for instance configuration services, RFC 3927. Amazon documents the configuration service address and its hardened second version at docs.aws.amazon.com.
Government interfaces worth checking before scraping an agency's pages: the SEC's EDGAR interfaces and the catalogue at data.gov are two starting points among many.
The two measurements that contradicted common advice (older automation outperforming newer, and edge rendering presenting a different address than edge relaying) are my own, taken on one Windows machine on a single day against a list of about seven sources. They are a demonstration that you should measure, not a claim about how those tools behave everywhere.

The API Refused Almost Every Request. The Same Data Was Sitting in a Free Bulk File.