# Parse A Site: Advanced Pipeline (LLM-ready) **What this is**: the full, repeatable pipeline for measuring a commercial design system and rebuilding it, including the fetch cascade for sites that block you, deterministic extraction, cross-corpus reference data, and two verification harnesses that turn "I did not break anything" into a test. **How to use it**: paste this entire file into Claude, ChatGPT, Gemini, Cursor or any coding agent and name your target and your own site. It is written as an executable procedure, not an essay. **License**: use it, adapt it, ship it. Originally published at https://jwatte.com/blog/parse-premium-sites-advanced/ **Companions**: `parse-a-site-starter-kit.md` (beginner path plus the full reference dataset), `premium-design-system-spec.md` (the measured design values), `design-retrofit-audit-kit.md` (the eleven failure modes). --- ## INSTRUCTIONS FOR THE ASSISTANT Execute the phases in order. Do not skip Phase 0. Do not declare the job finished until Phase 5 returns zero on every check. **Hard rules:** 1. **Deterministic first, judgment second.** Counting is a job for code. Interpretation is a job for you. Never eyeball a 600 KB page and report impressions. 2. **Presentation only.** Never alter markup, structured data, canonicals, hreflang, meta, or any machine-readable output. 3. **Additive and reversible.** One stylesheet loaded last, redefining existing custom property names. 4. **Calculate, never estimate**, especially contrast, and always with alpha compositing. 5. **Copy values, never identity.** Type scales, spacing, easings, color relationships: yes. Logos, names, photography, copy, trade dress: never. 6. **COMPLIANCE GATE, and it is not optional.** Before fetching ANY page automatically you must, and must tell the user you did: - **Fetch and read `https:///robots.txt`** and honor it ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)). If the path you want is disallowed for your agent, do not fetch it. - **Locate and read the site's Terms of Service** (usually footer-linked). If it prohibits automated access, crawling, scraping, or building a competing product, **STOP and tell the user.** Offer the manual alternative: they open the page in a browser, save it, and hand you the file. That is always permitted and is sufficient for everything in this kit. - **Never** fetch anything behind a login, paywall or account, and never collect personal data about identifiable people. - **Treat a block as an answer, not a puzzle.** A challenge page, CAPTCHA, explicit block message, `403` after a clean `robots.txt`, or `429`/`Retry-After` means STOP. Do NOT escalate through proxy or stealth tiers to defeat it. On `429`, slow down; never switch IP to evade a rate limit. - **Cache every fetch** so a page is taken exactly once, and go at human pace. - **Volume check:** if the job needs more than a handful of pages, say so and ask the user to confirm scope before continuing. 7. **Report uncertainty.** If a measurement is ambiguous, say so rather than smoothing it. 8. **If in doubt, do not fetch.** Recommend the manual save-page route. You are measuring presentation values, which needs very few pages. --- ## PHASE 0: SCOPE, PERMISSION AND CONSENT **Run the compliance gate in the hard rules above FIRST and report the result before anything else.** For each target domain state, explicitly: what `robots.txt` says, what the Terms of Service say about automated access, and your go / no-go. If it is no-go, stop and offer the manual route. Then establish: - What is the target, and is it public? (Public marketing pages only.) - What is the user's own site, and does it earn money? (If yes, Phase 4 harness is mandatory.) - Does the user's site use CSS custom properties for theming? Grep for `--[a-z-]+:` and for `[data-theme`, `.dark`, `prefers-color-scheme`. - **Token-based**: the retrofit is one additive file. Scope in hours. - **Hardcoded per component**: the retrofit is a rewrite. Say so before starting. - Count hardcoded literals inside theme blocks now: ``` grep -oE '\[data-theme="[a-z]+"\][^{]{0,90}\{[^}]*\}' style.css | grep -E '#[0-9a-fA-F]{3,6}' ``` One real stylesheet had **64** such rules. At specificity `(0,2,x)` every one outranks a token override. Plan a single remap block, do not patch component by component. - Find inline styles now, because they beat every selector: ``` grep -oE 'style="[^"]*(color|background)[^"]*"' **/*.html | sort -u ``` --- ## PHASE 1: THE FETCH CASCADE **Only enter this phase if Phase 0 returned go.** Escalate only on failure, and stop at the first tier returning real content. **Escalation is bounded.** These tiers exist for incidental blocks: a cloud IP range misclassified as hostile, geographic routing, a filter tripping on a single human-paced request. They are NOT for defeating a site that has decided it does not want automated traffic. **Stop the cascade permanently, do not escalate, if you see:** a CAPTCHA or challenge page, an explicit block message, a `429` or `Retry-After` (slow down instead), or a terms clause prohibiting automated access. Report the stop to the user and recommend the manual save-page route. ### Tier 1: direct Normal HTTPS request with a realistic `User-Agent`, `Accept`, `Accept-Language`, and `Sec-Fetch-*` headers. Handles the large majority. ### Tier 2: edge worker proxy A small function on an edge platform that fetches the URL and returns the body. The value is a different originating network with different reputation. Cloud-VM IP ranges are widely classified; edge networks often are not. Guard it: require a shared token, block private and loopback ranges, resolve DNS over HTTPS and re-check on redirects to avoid being used as an open proxy. ### Tier 3: a second proxy on a different provider Purely to obtain a third IP fingerprint. A real subset of sites is blocked at tiers 1 and 2 but open at tier 3, and which ones is not predictable in advance. ### Tier 4a: browser inside the edge network For pages that ship a JavaScript shell. Renders and returns the DOM. ### Tier 4b: locally-driven stealth browser A real browser with automation fingerprints patched (`navigator.webdriver`, plugin and WebGL tells, UA consistency). Last resort, and the slowest. ### Measured tier outcomes Fetching a 15-page creator-platform corpus on 22 August 2026: | Tier | Pages resolved | |---|---| | Direct | 12 | | Edge worker | 1 | | Browser (JS shell) | 1 | | Stealth browser | 1 | The last one failed every tier above it and would have been recorded as "blocked" by any single-tier tool. Outcomes vary by day, network and target configuration. ### Two escalation traps **A 200 is not proof of content.** A page can return HTTP 200 carrying a JavaScript shell with 126 characters of text. If your success check is the status code, that shell enters the dataset as a success and skews everything downstream. **Size heuristics miss small shells.** A shell-detection check that only fired above 30 KB of HTML let through a **2.8 KB** page that stripped to 126 characters. Detect by **signature**, not size: - noscript apologies: "You need to enable JavaScript to run this app", "doesn't work properly without JavaScript" - a text-to-markup ratio far below normal - a `
` or `
` that is effectively empty - guard against false positives with a minimum-text threshold, so a docs page that merely *mentions* the phrase is not killed --- ## PHASE 2: FIND THE DESIGN SYSTEM, NOT JUST THE PAGE **The single most important methodological step, and easy to get wrong.** Pages are instantiations. The system lives elsewhere. In the saved HTML, enumerate every `` and fetch any served from a **different host** to the page, typically a CDN subdomain. *Cautionary tale: grepping seventeen rendered pages for `clamp()` found almost none, and nearly produced the published claim "premium sites do not use fluid type". The shared design-system stylesheet used `clamp()` **120 times**, including a complete heading scale. Measure the system file.* That file, 1.49 MB, contained **184 unique custom properties** covering color, type, spacing, containers and grid gaps, plus roughly 70 icons inlined as data-URI SVGs (which is most of the 1.49 MB). --- ## PHASE 3: DETERMINISTIC EXTRACTION Run these passes over every saved file. Emit counts, not prose. | Pass | Regex target | Why it matters | |---|---|---| | Font families | `font-family:\s*['"]?([A-Za-z][\w \-]{2,24})` | reveals the display/body pairing | | Font weights | `font-weight:\s*([1-9]00\|normal\|bold)` | strongest single register signal | | Line heights | `line-height:\s*([\d.]+)` | the heading/body gap is the craft | | Fluid scales | `clamp\([^)]*\)` | the real type system | | Letter spacing | `letter-spacing:\s*([^;}!]{1,14})` | eyebrow tracking only | | Colors | `#[0-9a-fA-F]{6}\b` | ground, ink, accent | | Radii | `border-radius:\s*([\d.]+(?:px\|rem\|em\|%))` | fastest read of category | | Easings | `cubic-bezier\([^)]*\)` | motion personality | | Durations | `transition[^;}]{0,40}?(\d+(?:\.\d+)?m?s)` | interface vs decorative | | Breakpoints | `@media[^{]{0,80}` | the responsive ladder | | A11y features | `prefers-reduced-motion`, `prefers-color-scheme`, `\(hover:\s*hover\)` | discipline check | | Custom properties | `--[\w-]+:\s*[^;}]{1,60}` | the token contract | Then hand the **counts** to the model for interpretation, never the raw file. --- ## PHASE 4: THE CROSS-CORPUS REFERENCE DATA Measured 21 and 22 August 2026 across **51 pages**. **A** = 16 high-end client sites in one vertical. **B** = 15 platform vendors selling into it. **C** = 15 creator and personal-brand platforms. **D** = 5 developer infrastructure hosts. Counts are declarations in saved markup and inline styles. They weight heavily-repeated declarations: evidence of what is in each system, not a measure of visual area. ### 4.1 Page weight (HTML document only) | Corpus | Median | Range | |---|---|---| | A | 605 KB | 425 to 951 KB | | B | 250 KB | 73 to 1262 KB | | C | 419 KB | 6 to 2059 KB | | D | 566 KB | 224 to 1238 KB | ### 4.2 Font weight | Corpus | By frequency | |---|---| | **A** | **400 (108)**, normal (62), 700 (49), bold (49), 500 (46), 300 (41), 600 (21), 200 (12) | | **B** | **700 (235)**, 600 (170), 500 (136), 400 (83), 800 (54), 300 (39), 900 (31), normal (30) | | **C** | **400 (245)**, 500 (223), 700 (116), 600 (96), normal (52), 900 (20), 800 (19), 100 (12) | | **D** | 600 (18), 500 (15), 400 (1) | The client work is light; the companies selling it are bold on their own marketing. Different jobs. ### 4.3 The commercial fluid type scale ``` h1 clamp(4rem, 4.453125vw, 14.25rem) weight 400 line-height 1.03 h2 clamp(3.25rem, 3.125vw, 10rem) weight 400 line-height 1.15 h3 clamp(2.625rem, 2.343vw, 7.5rem) weight 400 line-height 1.14 h4 clamp(2rem, 1.093vw, 3.5rem) weight 300 line-height 1.12 h5 clamp(1.5rem, 0.78125vw, 2.5rem) weight 300 line-height 1.17 h6 clamp(1.25rem, 0.664vw, 2.125rem) weight 300 line-height 1.3 body clamp(1rem, 0.625vw, 2rem) weight 300 line-height 1.38 ``` Heading tokens set `letter-spacing: normal`. Measured tracking (1.5px, 1px, 0.5px, 2px, 3px, 0.2em) applies almost exclusively to small uppercase eyebrows. ### 4.4 Color by corpus **A, high-end client:** `#ffffff (298)`, `#000000 (250)`, `#848484 (180)`, `#1a1a1a (93)`, `#403e3f (54)`, `#f1f0eb (54)`, `#5f5d61 (37)`, `#e2e0d9 (36)`, `#fbfbf9 (35)`, `#f47e14 (33)`, `#141414 (32)`, `#3e1214 (30)`, `#e4dfd8 (29)`, `#c28563 (28)`, `#fbf9f7 (27)`, `#c09b57 (21)` Bone grounds, charcoal inks, muted earth accents. **B, vendor marketing:** `#ffffff (182)`, `#e0e4e7 (144)`, `#3ec49d (87)`, `#000000 (86)`, `#d3e5f3 (81)`, `#23355b (69)`, `#f8fafd (61)`, `#05205e (42)`, `#d7ae6b (40)`, `#ebf3f8 (36)`, `#131c31 (31)`, `#fafdff (27)`, `#dc0000 (27)`, `#fb8400 (26)`, `#0693e3 (26)`, `#d2421a (26)` A state palette: mint, bright blue, red, orange. Conversion-oriented. **C, creator platforms:** `#101585 (179)`, `#ffffff (108)`, `#000000 (106)`, `#000624 (98)`, `#3899ec (66)`, `#1f1f1e (64)`, `#00524e (56)`, `#405b50 (43)`, `#fdfdfc (43)`, `#ff5c3c (43)`, `#7008e7 (37)`, `#a1a1aa (27)`, `#f4f4f5 (26)`, `#d4d4d8 (22)`, `#121c1e (20)`, `#f7f7f7 (18)` Deep indigo, electric purple, coral. Personality-forward. **D, developer hosts:** `#a3a3a3 (25)`, `#404040 (25)`, `#171717 (20)`, `#737373 (20)`, `#f5f5f5 (15)`, `#fafafa (12)`, `#d4d4d4 (10)`, `#e5e5e5 (10)`, `#79b8ff (8)`, `#0f0f10 (5)`, `#171719 (5)`, `#111113 (5)`, `#525252 (5)`, `#f6821f (5)`, `#4f46e5 (5)`, `#818cf8 (5)` A precise neutral ramp plus one or two brand hues. ### 4.5 Radius | Corpus | By frequency | |---|---| | **D** | **3px (83)**, 8px (31), 1.5px (16), .25rem (7), .5rem (7), 12px (5) | | **A** | 8px (60), 50% (30), 16px (29), 999px (25), 25px (7), 12px (7) | | **B** | 50% (75), 4px (33), 999px (31), 2px (23), 8px (22), 16px (20) | | **C** | **15px (57)**, 8px (50), 50% (40), 30px (24), 100px (20), 10px (19) | ### 4.6 Motion **Easings:** - **A**: `cubic-bezier(0.645,0.045,0.355,1)` (24), `cubic-bezier(0.23,1,0.32,1)` (18), `cubic-bezier(0.835,0,0.19,1)` (16) - **B**: `cubic-bezier(0.42,0,0.58,1)` (39), `cubic-bezier(.2,.7,.2,1)` (11), `cubic-bezier(.34,1.56,.64,1)` (5) - **C**: `cubic-bezier(0.433,0.217,0.355,1)` (28), `cubic-bezier(.87,0,.13,1)` (10), `cubic-bezier(.4,0,.2,1)` (4) - **D**: `cubic-bezier(0.25,0.46,0.45,0.94)` (69), `cubic-bezier(0.55,0.085,0.68,0.53)` (47), `cubic-bezier(0.32,0.72,0,1)` (2) Corpus B contains `cubic-bezier(.34,1.56,.64,1)`, which overshoots past 1 and bounces. It does not appear in corpus A at all. **Durations:** - **A**: 3s (369), 0.3s (163), 0.2s (145), 2s (72), 0s (56) - **B**: 2s (41), 3s (40), 0.1s (33), 0.2s (28), 0.25s (24) - **C**: 0.2s (224), 0s (56), 50ms (42), 0.3s (34), 700ms (29) - **D**: **0.16s (69)**, 0.1s (47), 2000ms (10), 15s (8), 3s (2) Anything 2s and above is an image pan or marquee, never an interface transition. ### 4.7 Accessibility discipline | Corpus | reduced-motion | color-scheme | hover gated | |---|---|---|---| | **A** | **0 of 16** | 0 of 16 | 13 of 16 | | **B** | 7 of 15 | 2 of 15 | 1 of 15 | | **C** | 3 of 15 | 6 of 15 | 2 of 15 | | **D** | 1 of 5 | **5 of 5** | 0 of 5 | ### 4.8 Spacing and containers ``` --grid-gap-xsmall: .5rem --grid-gap-large: 3.875rem --grid-gap-small: 1rem --grid-gap-xlarge: 15rem /* 240px */ --grid-gap-medium: 1.5rem --spacing-large: clamp(8.5rem, 9.4531vw, 30.25rem) --spacing-xlarge: clamp(11.5rem, 6.8554vw, 21.9375rem) --container-small: 67.875rem medium: 82rem large: 90rem ``` Also observed: flat `176px`, and `7.8vw`. ### 4.9 Breakpoints `768px` (551), `560px` (316), `767px` (201), `992px` (143), `1024px` (134), `1200px` (58), `900px` (55). ### 4.10 Platform pricing (22 August 2026) **Vertical platforms, 15 sampled. 13 publish no price**, showing a demo or sales CTA. Published: `$299.95 / $399.95 / $599.95 per month` (setup fee mentioned); `from $1,299/month`; `$89 per user/month`; contact-form budget bands of `$12k-24k`, `$24k-48k`, `$48k-96k`, `$96k+`. **Creator platforms, 15 sampled, almost all published:** `$8/$15/$35`, `$30`, `$42/$84/$150`, `$15 to $499`, `$33/$66/$99`, `$9 to $39`, `$29/$79/$299`, `$17/$29/$39/$159` per month, most with free tiers. **Hosts:** Netlify Free "$0 forever", Pro **$20/month**. Vercel Hobby free, Pro **$20/mo** including $20 usage credit. Cloudflare Workers free plan; paid **$5/month** account minimum, 10M requests included, **+$0.30 per additional million**, and **no data transfer or bandwidth charges**. That last property is why an edge worker is the correct home for the Phase 1 proxy tiers: a fetch-heavy job on a metered-bandwidth platform gets expensive; on an unmetered one it does not. --- ## PHASE 5: VERIFY (the definition of done) ### 5.1 Machine-readable diff Walk the built site and record per page: URL; every JSON-LD block and the full set of `@type` values **recursing through all nested objects and arrays**, not just `@graph`; whether each block parses; canonical; every `rel="alternate"` hreflang and href; meta description, robots, OpenGraph, Twitter; `h1` presence; images missing `alt`; and every non-HTML endpoint (`robots.txt`, sitemaps, `llms.txt`, feeds, `.well-known/*`). Capture before, capture after, diff. **Regressions fail. Additions report and never fail.** Real baseline held across four rounds of design work: **106 pages, 23 endpoints, 556 JSON-LD blocks, 33 schema types, 0 parse errors, 0 images missing alt.** ### 5.2 Contrast sweep For every element with its own text node: 1. Read computed color and background. 2. **Composite alpha**: walk up ancestors combining translucent backgrounds until an opaque one is reached. A first sweep without this reported **seven failures that were entirely fictional**, and it will also miss real failures where a translucent overlay darkens text. 3. Compute the WCAG relative-luminance ratio. 4. Thresholds: **4.5:1** normal text; **3:1** large text (≥24px, or ≥18.66px at weight ≥700); **3:1** non-text UI including **input borders**. 5. Where it fails, walk the color along its own hue until it clears on **every** ground it appears on, not only the page background. **Run every theme independently.** A clean light sweep says nothing about dark. On the real retrofit, light reached zero while dark still carried 81 failures. Measured progression on the real site: **68 → 38 → 14 → 3 → 0** (light), **81 → 0** (dark). The first sweep found **every primary call to action at 4.0:1**, below AA, months after launch. ### 5.3 Layout and motion - No horizontal overflow at 390, 768, 1440: compare `scrollWidth` to `clientWidth`, ignoring deliberately off-canvas elements. - No element stranded below 0.9 opacity while in the viewport. - Tap targets at least 44px. --- ## PHASE 6: THE FAILURE MODES THAT WILL BITE YOU Condensed; the full treatment is in `design-retrofit-audit-kit.md`. 1. **One color token cannot be both ink and fill.** Darkening an accent to be readable as text takes every button using it as a fill to ~2.9:1 in one edit. Define `--accent`, `--accent-fill`, `--on-accent-fill`. 2. **Hardcoded literals inside theme blocks** at `(0,2,x)` outrank token overrides. Count them, remap in one pass. 3. **Fix the ground, not the ink**, when several elements fail on the same background. 4. **A late `:root` declaration beats every earlier theme block** (same `(0,1,0)` specificity, source order wins). This produced 81 dark-mode failures from one line. 5. **Blanket color overrides must enumerate their dark contexts.** CSS cannot ask what is behind an element. 6. **Never globally style an element you did not author** (`blockquote` used for 700-character reviews). 7. **Interior pages have their own specificity** (`.page-content h1` at `(0,1,1)`). 8. **Guard the hiding, not just the animating**: put `opacity: 0` only inside `@supports (animation-timeline: view())`. 9. **Full-page screenshots do not trigger lazy loading**; scroll in steps first. Review at native resolution. 10. **Things hide from grep**: emoji ship as numeric HTML entities (`[0-9]{5};`). 11. **Parallel build-time fetches** time out intermittently and render fallbacks; go sequential with a retry. And a secret read only from `process.env` with no fallback silently renders placeholders in every local build. --- ## PHASE 7: REGISTER PROFILES | If building for | Corpus | Weights | Radius | Ground | Accent | Motion | |---|---|---|---|---|---|---| | Property, luxury, law, wealth | **A** | 300-400 | 2-8px | bone `#f1f0eb` | one muted earth | 0.3s restrained | | Creator, influencer, coach | **C** | 400-500 | 15-30px | white or deep saturated | one bold high-chroma | 0.2s snappy | | SaaS, dev tools | **D** | 500-600 | 3px | neutral gray ramp | one brand hue | 0.1-0.16s | | Local service, trades, clinics | **A** softened | 400 | 4-8px | bone/warm white | one muted hue | 0.3s | | B2B conversion-led | **B** | 500-700 | 4-8px | white + tinted sections | brand + state colors | 0.2s, may bounce | Invariants across every register: one accent in three shades; ink never pure black on pure white; hover gated; reduced-motion always shipped; separate ink and fill accent tokens. --- ## LIMITS AND ETHICS - Counts are from 51 saved pages on 21 and 22 August 2026. They are that sample's numbers. The **relationships between corpora** are the transferable finding, not the absolute counts. - Companies measured are deliberately unnamed. Private businesses illustrating a pattern, not case studies. - Cascade tier outcomes vary by day, network and target configuration. - **Fetch lawfully and responsibly.** Read the Terms of Service and `robots.txt` first, every time, and do not automate access where either prohibits it. Public marketing pages only. Human pace. Cache what you take. Nothing behind a login or paywall. No personal data about identifiable people (GDPR, CCPA and equivalents apply and are a different legal universe entirely). A block, challenge page or `429` is an answer, not an obstacle: do not escalate tiers to defeat it. - **This is not legal advice, and the author is not a lawyer.** The law on automated access varies by jurisdiction and is still developing. The narrow safe path: read the terms, honor `robots.txt`, prefer opening pages by hand, take values not identity, stop when anything pushes back. **At scale, commercially, or against a competitor: take legal advice before starting.** - **Copy values, never identity.** Take the 1.06 line-height. Never take the wordmark, the photography, the copy, or the trade dress. - **You can reproduce the design layer completely.** You cannot trivially reproduce a licensed data feed, a customer database with years of history, automated follow-up, an on-call team, or a compliance layer. Some invoices are fair; the point is knowing which line is which. - **A passing contrast sweep is a floor, not proof of accessibility.** It tests none of keyboard operability, screen-reader semantics, focus order, or cognitive load.