# The Full Pipeline: Measuring A Commercial Design System, Rebuilding It With An AI Agent, And Proving You Did Not Break Anything

A four-tier fetch cascade for sites that block you, a deterministic extraction pass, and two verification harnesses that turn 'I did not break the SEO' into a test. With the measured cross-corpus data from 51 pages.

Author: J.A. Watte
Published: August 24, 2026
Source: https://jwatte.com/blog/parse-premium-sites-advanced/

---


[The beginner version of this post](/blog/parse-premium-sites-beginner/) covers the case where you save a page, ask an AI to measure it, and apply the numbers to your own site. That works, and for most people it is enough.

This is the version for when it is not enough: when the site you want to measure blocks you, when the design system is in a stylesheet rather than the page, when you are retrofitting onto something that already ranks and cannot be allowed to break, and when you want the whole thing repeatable rather than a one-off.

Everything below was used to measure **51 pages across four corpora**: 16 high-end client sites in one vertical, 15 platform vendors selling into that vertical, 15 creator and personal-brand platforms, and 5 developer infrastructure hosts.

## 0. Read this before you fetch anything

I am putting this first because the rest of the post describes tooling that can get you into trouble if you use it carelessly, and burying the caveat at the bottom would be dishonest.

**Check the Terms of Service before you fetch a single page.** Most sites have one, usually linked in the footer. A large number of them contain a clause that prohibits automated access, crawling, scraping, or using the site to build a competing product. **If the terms prohibit automated access, do not automate access to that site.** Open it in your browser and read it like a person instead. That is always allowed, it is what the page is for, and for measuring a design system it is genuinely sufficient.

**Check `robots.txt` and honor it.** It lives at the root of the domain, and the Robots Exclusion Protocol was formalized as [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) in 2022. Its stated purpose is to let "service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers". `robots.txt` is not by itself a contract, but ignoring it is a clear signal of intent that will not read well anywhere.

**A block is an answer, not a puzzle.** This is the part I want to be blunt about, because sections 1 and 4 of this post describe proxy tiers and a stealth browser, and it would be easy to read them as a toolkit for defeating anything in your way.

They are not. Those tiers exist for a narrow and legitimate set of problems: your own infrastructure sitting in a cloud IP range that gets misclassified as hostile, geographic routing that serves you a different page, a bot filter that trips on a request that is genuinely a single human-paced page view. **If a site has actively decided it does not want automated traffic, escalating through tiers to get around that is circumventing a technical access control, and you should stop.** Not because a tool will refuse, but because that is the point where "measuring a public page" turns into something you would not want to explain later.

My own honest test: if the site owner read my access log, would they see one polite request or a campaign? If it is a campaign, I have already gone too far.

**Never touch these.** Anything behind a login, a paywall, or an account. Anything containing personal data about identifiable people, which pulls in GDPR, CCPA and their relatives, and which is a completely different legal universe to reading a marketing page. Anything you were sent privately. Any site whose terms you have not read.

**What you are actually allowed to learn.** Measuring that a heading is set at weight 400 with a 1.03 line-height is reading a published fact about a public page. Type scales, spacing rhythms, easing curves and color relationships are craft conventions, and studying them is how every designer has always learned. What is not yours: the code as written, the photography, the copy, the logo, the brand name, and a distinctive overall look that consumers associate with that company. **Take the 1.06 line-height. Never take the wordmark.**

**Two practical rules that cost nothing.** Cache every page you fetch so you take it exactly once instead of hammering a server across a session. And identify yourself honestly in your `User-Agent` where you reasonably can, rather than impersonating a browser you are not.

**And the disclaimer that matters:** I am not a lawyer, this is not legal advice, and the law here genuinely varies by country and is still moving. The safe path is narrow and clearly fine: read the terms, honor `robots.txt`, fetch public marketing pages slowly or just open them by hand, take values not identity, and stop the moment anything pushes back. **If you are doing this at scale, commercially, or against a competitor, talk to a lawyer before you start, not after.**

## 1. The fetch cascade, and why one tier is not enough

Everything in this section assumes you have cleared section 0: the terms permit it, `robots.txt` permits it, and nothing is pushing back.

A plain HTTP request gets you most pages. It does not get you all of them, and the ones it misses are often the interesting ones.

Use an escalating cascade, stopping at the first tier that returns real content:

**Tier 1: direct fetch.** A normal request with a browser-like `User-Agent` and the `Sec-Fetch-*` headers. This handles the large majority.

**Tier 2: an edge worker.** A tiny proxy deployed on an edge platform, which fetches the URL and returns the body. The point is not speed, it is that the request now originates from a different network with different reputation. A worker on a major edge network frequently gets through where a request from a cloud VM does not, because the data center IP ranges most scrapers run from are widely classified.

**Tier 3: a second, differently-hosted proxy.** Another small function on a *different* provider. This exists purely to give you a third IP fingerprint. A meaningful number of sites are blocked at tier 1 and tier 2 but open at tier 3, and which ones is not predictable.

**Tier 4: a headless browser, and then a stealth-patched one.** For pages that return a shell and build themselves with JavaScript. Two useful variants: a browser running *inside* an edge network, and a locally-driven browser with the standard automation fingerprints patched out.

Running the creator-platform corpus through that cascade, the tiers earned their place:

- **12 of 15** resolved on the direct tier.
- **1** needed the edge worker.
- **1** needed a browser (it serves a JavaScript shell with almost no text in the initial HTML).
- **1** needed the locally-driven stealth browser, having failed every tier above it.

That last one would simply have been recorded as "blocked" by a single-tier tool.

### Two traps worth knowing

**A 200 is not proof of content.** A page can return HTTP 200 with a JavaScript shell containing 126 characters of text. If your success check is the status code, that page silently enters your dataset as a success and quietly skews everything downstream. Check for content, not status: look for the noscript apologies ("You need to enable JavaScript to run this app") and for a text-to-markup ratio that makes no sense, and escalate when you see them.

**Size heuristics miss small shells.** An early version of my own check only treated a page as a shell above 30 KB of HTML, on the assumption that a shell would be large. One site served a **2.8 KB** shell that stripped to 126 characters of text, sailed through the size check, and stopped the cascade at the wrong tier. Detect by signature, not size.

### When to stop escalating

The tier table above is a description of what happened, not a recommendation to climb it.

Escalate when the evidence says the block is incidental: the page loads perfectly in your own browser, `robots.txt` and the terms permit automated access, and the failure looks like IP-range classification or a rendering problem rather than a decision about you.

**Stop escalating** the moment you see a deliberate signal. A `429` or a `Retry-After` header is a rate limit, and the correct response is to slow down, not to switch IPs. A challenge page, a CAPTCHA, or an explicit block message is the site telling you no. A terms clause prohibiting automated access means you never should have started the cascade.

In the corpus for this post, one page needed the stealth tier. The honest thing to say about that is not "and here is how to always win". It is that a single stealth-tier hit on a public pricing page, fetched once and cached, is about as far as I would go, and if it had come back with a challenge page I would have opened it in a browser and read it myself.

## 2. Measure the design system, not the page

This is the single biggest methodological point in this post, and I got it wrong first.

I grepped the rendered HTML of seventeen high-end sites for `clamp()`, found almost none, and nearly published the conclusion that premium sites do not use fluid type.

**That was completely wrong.** The pages are instantiations. The design system lives in a separate stylesheet, and in that file `clamp()` appears **120 times**, including a full fluid scale for every heading level:

```
h1   clamp(4rem,     4.453125vw, 14.25rem)   weight 400   line-height 1.03
h2   clamp(3.25rem,  3.125vw,    10rem)      weight 400   line-height 1.15
h3   clamp(2.625rem, 2.343vw,    7.5rem)     weight 400   line-height 1.14
h4   clamp(2rem,     1.093vw,    3.5rem)     weight 300   line-height 1.12
h5   clamp(1.5rem,   0.78125vw,  2.5rem)     weight 300   line-height 1.17
h6   clamp(1.25rem,  0.664vw,    2.125rem)   weight 300   line-height 1.3
body clamp(1rem,     0.625vw,    2rem)       weight 300   line-height 1.38
```

So: **find the system stylesheet and fetch it separately.** Look in the `<link rel="stylesheet">` tags for a file served from a different host to the page, usually a CDN subdomain. In this case it was a 1.49 MB file, and it contained the entire token contract: 184 unique custom properties covering color, type, spacing, containers, grid gaps, and about 70 icons inlined as data-URI SVGs, which is most of why it is 1.49 MB.

The token set that file publishes:

```
--color-primary-1:#fff   -2:#1a1a1a   -3:#161515   -4:#eeede9   -5:#000
--color-accent:#b18463   accent-dark:#916648   accent-light:#db9d72
--grid-gap-xsmall:.5rem  small:1rem  medium:1.5rem  large:3.875rem  xlarge:15rem
--spacing-large:  clamp(8.5rem,  9.4531vw, 30.25rem)
--spacing-xlarge: clamp(11.5rem, 6.8554vw, 21.9375rem)
--container-small:67.875rem  medium:82rem  large:90rem
```

That is the product. It took about four minutes to read once I looked in the right file.

## 3. Extract deterministically, then reason

Do not ask a language model to eyeball a 600 KB page. Run a deterministic pass first, then hand the *counts* to the model for interpretation. Counting is a job for code; judgment is a job for the model.

The passes worth running over each saved page: every `font-family`, every `font-weight`, every `cubic-bezier`, every transition duration, every hex color, every `border-radius`, every `@media` query, every `clamp()`, and the presence of `prefers-reduced-motion`, `prefers-color-scheme` and `(hover: hover)`.

Then compare across corpora. Here is what 51 pages produced.

### Font weight is the clearest signal of register

| Corpus | Dominant weights |
|---|---|
| High-end client sites | **400** (108), normal (62), 700 (49), 300 (41) |
| Vendors selling those sites | **700** (235), 600 (170), 500 (136), 400 (83) |
| Creator platforms | **400** (245), 500 (223), 700 (116) |
| Developer hosts | 600 (18), 500 (15), 400 (1) |

The client sites are light. **The companies that build them are bold on their own marketing sites.** Two different jobs, and worth knowing which one you are doing.

### Corner radius is the fastest read of category

| Corpus | Most common radii |
|---|---|
| Developer hosts | **3px** (83), 8px (31), 1.5px (16) |
| High-end client sites | 8px (60), 50% (30), 16px (29) |
| Vendor marketing sites | 50% (75), 4px (33), 999px (31) |
| Creator platforms | **15px** (57), 8px (50), 30px (24), 100px (20) |

Sharper reads technical and serious. Rounder reads friendly and consumer. If you are picking a radius by feel, this table is a shortcut.

### Easing curves cluster tightly, and differently per category

- High-end client sites: `cubic-bezier(0.645,0.045,0.355,1)` (24), `cubic-bezier(0.23,1,0.32,1)` (18), `cubic-bezier(0.835,0,0.19,1)` (16)
- Vendor sites: `cubic-bezier(0.42,0,0.58,1)` (39), `cubic-bezier(.2,.7,.2,1)` (11)
- Creator platforms: `cubic-bezier(0.433,0.217,0.355,1)` (28), `cubic-bezier(.87,0,.13,1)` (10)
- Developer hosts: `cubic-bezier(0.25,0.46,0.45,0.94)` (69), `cubic-bezier(0.55,0.085,0.68,0.53)` (47)

Interface durations cluster at **0.2s and 0.3s** almost everywhere. The developer hosts run faster, with **0.16s** and **0.1s** dominant, which is consistent with an audience that values responsiveness over drama. Anything in the 2 to 6 second range is an image pan or a marquee, never an interface transition.

### Accessibility discipline is inverted from price

This is the finding I did not expect.

| Corpus | `prefers-reduced-motion` | `prefers-color-scheme` | hover gated |
|---|---|---|---|
| High-end client sites | **0 of 16** | 0 of 16 | 13 of 16 |
| Vendor marketing sites | 7 of 15 | 2 of 15 | 1 of 15 |
| Creator platforms | 3 of 15 | 6 of 15 | 2 of 15 |
| Developer hosts | 1 of 5 | **5 of 5** | 0 of 5 |

**Not one of the sixteen most expensive sites in the sample ships a reduced-motion block.** They are the most animated sites in the corpus and the least willing to turn it off. Meanwhile every developer host respects the OS color preference, because their audience notices.

If you build your own, five lines puts you ahead of your reference material:

```css
@media (prefers-reduced-motion: reduce) {
  *, *::before, *::after {
    animation-duration: .01ms !important;
    animation-iteration-count: 1 !important;
    transition-duration: .01ms !important;
    scroll-behavior: auto !important;
  }
}
```

## 4. Synthesize, do not copy

Two rules keep this legitimate and keep the result good.

**Copy values, never identity.** Type scales, spacing rhythms, easing curves and color *relationships* are craft conventions, and measuring them is how every designer has always learned. Logos, brand names, photography, copy, and a distinctive overall trade dress are not yours. Take the 1.06 line-height. Do not take the wordmark.

**Adapt the register to your actual business.** The measured upper bounds suit pages carrying six words and a full-bleed photograph. On a content-dense site they are absurd. Keep the weights and line-heights exactly, because those are the part that reads as expensive, and pull the sizes down. Similarly, a 240px section gap is correct on a six-section page and ridiculous on a twenty-section one.

## 5. Verify, twice

A design change on a site that earns money needs two harnesses. Not screenshots.

### Harness one: the machine-readable diff

Walk the built site and record every page URL, every JSON-LD block and the full set of `@type` values inside it recursing through nested objects, whether each block still parses, every canonical, every hreflang alternate, the meta tags, `h1` presence, and images missing `alt`. Capture before, capture after, diff. Regressions fail; additions are reported and never fail.

On the real retrofit this held at **106 pages, 23 endpoints, 556 JSON-LD blocks, 33 schema types, 0 parse errors, 0 images missing alt** through four rounds of design work. That is what lets you say "presentation only" as a fact.

### Harness two: the contrast sweep

Walk every element that has its own text node. For each, composite the effective background by walking up the ancestor chain through translucent layers until you hit an opaque one, then compute the WCAG ratio.

**Compositing alpha is not optional.** My first sweep treated `rgba(179,146,78,0.14)` as opaque and reported seven failures that were entirely fictional. It also means a checker without compositing will miss real failures where a translucent overlay darkens text.

Thresholds: 4.5 to 1 for normal text, 3 to 1 for large text (24px, or 18.66px at weight 700+), 3 to 1 for non-text UI including **input borders**, which is the one everyone forgets.

Run it against **every theme independently**. A clean light-theme sweep tells you nothing about dark mode. On the real site, the light theme reached zero while dark mode still had 81 failures from a single cascade mistake.

The sweep found 68 failures on the first run, including **every primary call to action at 4.0 to 1**, on a site that had been live for months.

## 6. Where to run it

The whole pipeline is static output plus a couple of small functions, so the hosting question is nearly free:

- **Netlify**: Free at "$0 forever"; Pro at **$20/month** with unlimited members.
- **Vercel**: free Hobby tier; Pro at **$20/mo**, including $20 of usage credit.
- **Cloudflare Workers**: free plan; paid plan with a **$5/month account minimum**, 10 million requests included per month, **+$0.30 per additional million**, and **no data transfer or bandwidth charges** at all.

That last property is why an edge worker is the right home for the proxy tiers in section 1: a fetch-heavy job on a platform that meters bandwidth gets expensive, and on one that does not, it does not.

## 7. What this does not replace

Worth stating plainly, because the pipeline can make you overconfident.

You can reproduce the design layer completely. You cannot trivially reproduce a licensed industry data feed, a customer database with years of history, automated follow-up, or an on-call team. In several verticals there is a compliance layer as well.

The useful outcome is not "cancel the subscription". It is knowing which line of the invoice is design, because that line you can now produce yourself to a measurably higher accessibility standard than the reference material, and host for under $20 a month.

## The files

Both are free, need no signup, and are written to be pasted directly into an AI assistant. Per request, they contain **the full measured dataset**, not a summary, so you can adapt the numbers to your own vertical rather than taking my word for the conclusions:

- **[parse-a-site-advanced-pipeline.md](/downloads/parse-a-site-advanced-pipeline.md)** is this post as an executable procedure: the fetch cascade with escalation rules, the deterministic extraction spec, the full cross-corpus data tables for all four corpora, the synthesis rules, both verification harnesses, and per-vertical adaptation profiles.
- **[parse-a-site-starter-kit.md](/downloads/parse-a-site-starter-kit.md)** is the beginner path, with the exact prompt wording for each step.

The two design files from the earlier posts remain the companions: **[premium-design-system-spec.md](/downloads/premium-design-system-spec.md)** for the measured values, and **[design-retrofit-audit-kit.md](/downloads/design-retrofit-audit-kit.md)** for the eleven failure modes.

## Fact-check notes and sources

All figures were measured on 21 and 22 August 2026 from saved copies of publicly published pages.

- **The corpora are deliberately not named.** They are private companies used to illustrate measurable design and pricing patterns, not case studies, and none asked to be written about. Counts and ranges are quoted exactly as measured.
- **Corpus sizes**: 16 high-end client sites, 15 vertical platform vendors, 15 creator and personal-brand platform pages, 5 developer host pricing pages. 51 pages total.
- **Design counts** (weights, radii, easings, durations, media-query presence) count declarations found in saved markup and inline styles. They weight heavily-repeated declarations and are evidence of what is in each system, not a measure of visual area. The type scale in section 2 was read from a single vendor design-system stylesheet of 1.49 MB.
- **Cascade tier results** (12 direct, 1 edge worker, 1 browser, 1 stealth) are for the 15-page creator corpus fetched on 22 August 2026. Tier outcomes vary by day, by network, and by the target's own configuration; treat them as illustrative, not fixed.
- **The 2.8 KB shell and the 126-character text extraction** are from a real page that defeated a size-based shell heuristic in my own tooling, corrected on 19 August 2026.
- **Host pricing**, read 22 August 2026: [Netlify pricing](https://www.netlify.com/pricing/), [Vercel pricing](https://vercel.com/pricing), [Cloudflare Workers pricing](https://developers.cloudflare.com/workers/platform/pricing/), which documents the $5/month account minimum, the 10 million included requests, the "+$0.30 per additional million", and states there are "no data transfer (egress) or throughput (bandwidth) charges".
- **WCAG thresholds** from [SC 1.4.3 Contrast (Minimum)](https://www.w3.org/WAI/WCAG22/Understanding/contrast-minimum.html) and [SC 1.4.11 Non-text Contrast](https://www.w3.org/WAI/WCAG22/Understanding/non-text-contrast.html).
- **The 106-page verification figures** are the output of the diff harness described, run on an Eleventy site I build and maintain.

## Related reading

- **[The beginner version of this post](/blog/parse-premium-sites-beginner/)**: the five-step path, the pricing comparison, and what these platforms actually charge.
- **[GitHub Solved The Styling Layer](/blog/design-craft-beyond-github-style-repos/)**: the seventeen-site teardown that produced the measured design values.
- **[Retrofitting A Design System On A Live Site](/blog/retrofitting-design-system-what-broke/)**: the eleven failure modes, including the cascade mistake that produced 81 dark-mode failures.
- **[Self-hosting your fonts instead of loading them from a CDN](/blog/blog-self-host-cdn-fonts-speed/)**: relevant once you are serving the type yourself.

_This post is informational, and is not legal advice. I am not a lawyer, and the law on automated access varies by jurisdiction and is still developing. Before fetching any page automatically: read that site's Terms of Service, read its `robots.txt` ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)), and do not automate access where either prohibits it. Treat a block, a challenge page, a CAPTCHA or a `429` as an answer rather than an obstacle, and do not escalate through proxy or stealth tiers to defeat a site that has decided it does not want automated traffic. Never access anything behind a login or paywall, and never collect personal data about identifiable people. Fetch at a human pace and cache what you take so you take it once. Measure presentation values such as type scale, spacing and color relationships; never copy a logo, brand name, photography, written copy, or a distinctive overall look. Pricing is as published on the vendors' own pages on 22 August 2026 and will change. No affiliation with any company mentioned or measured is implied. If you are doing this at scale, commercially, or against a competitor, take legal advice before you start._


---

Canonical HTML: https://jwatte.com/blog/parse-premium-sites-advanced/
RSS: https://jwatte.com/feed.xml
JSON Feed: https://jwatte.com/feed.json
Hero image: https://jwatte.com/images/parse-premium-sites-advanced.webp