# Audit Harness Traps

Seven ways an automated browser audit reports failures that do not exist, or misses failures that
do. Each one cost me real time on a fleet of small sites. Each one has a fix that is three lines.

If you are checking your own site by hand, you want `display-state-audit-checklist.md` instead.
This file is for the moment you decide to automate it across more than about ten pages.

Version 1.0, 2026-11-10. Source article:
https://jwatte.com/blog/accessibility-toggles-combine-audit-every-state/

---

## 0. The one thing to internalise

**When a measurement surprises you, check the instrument before you change the thing being
measured.**

Three separate times my harness reported a broken site that was fine, and twice I got as far as
writing the fix. A test tool that degrades silently produces confident nonsense, and confident
nonsense is worse than no tool, because you act on it.

Every trap below has the same shape: the harness kept running, kept reporting, and was wrong.

---

## 1. A registered service worker answers your test's requests

**Symptom.** Roughly two hundred phantom failures per site. Pages that look correct in a real
browser report broken colours. Screenshots show a header you replaced yesterday.

**Cause.** The first page your script loads registers the site's service worker. Every page after
that is served from the worker's cache instead of the files you are testing. You are measuring a
snapshot from an unknown point in the past.

On one site this read 210 failures. With the worker stubbed out, the same site read 8.

**Fix.** Stub `navigator.serviceWorker` before any page script runs. `evaluateOnNewDocument` (or
Playwright's `addInitScript`) executes ahead of the document.

```js
await page.evaluateOnNewDocument(() => {
  try {
    Object.defineProperty(navigator, 'serviceWorker', {
      configurable: true,
      get: () => ({
        register: () => Promise.reject(new Error('disabled for audit')),
        getRegistration: () => Promise.resolve(undefined),
        getRegistrations: () => Promise.resolve([]),
        addEventListener() {}, removeEventListener() {},
      }),
    });
  } catch (e) {}
});
await page.setCacheEnabled(false);
```

The `try/catch` matters: on some builds the property is non-configurable and the throw would kill
your init script before anything else in it runs.

**Do not** use this stub in the run where you are verifying that the service worker itself works.
That needs a separate pass with the stub off, checking the registration reaches `activated` and
the cache holds the shell URLs you expect.

---

## 2. requestAnimationFrame does not run in a background tab

**Symptom.** One page finishes fast, the rest hang until timeout. A ninety second run becomes
twenty five minutes.

**Cause.** Browsers throttle or suspend animation frames in background tabs to save power. If you
open one page per display state, only one is ever in the foreground. Anything in the other three
that awaits a frame just sits there.

Anything built on rAF has this problem, including "wait until the layout settles" helpers and most
scroll-into-view utilities.

**Fix.** Bring the page to the front before measuring, and poll on a timer, never on a frame.

```js
await page.bringToFront();
await page.waitForFunction(() => document.readyState === 'complete', { polling: 200, timeout: 15000 });
// and instead of waiting on rAF:
await page.evaluate(() => new Promise(r => setTimeout(r, 250)));
```

Puppeteer's `waitForFunction` defaults to `polling: 'raf'`. That default is the trap. Set
`polling` to a number in every call that runs against a page you have not brought to the front.

---

## 3. Toggling a class and reading the result gives you the old value

**Symptom.** State two measures identical to state one. Or it measures colours from neither state.

**Cause.** Flipping a theme attribute and immediately reading `getComputedStyle` returns the
previous state's values, even after forcing a reflow. Custom properties appear to be the trigger:
the cascade re-resolves, but the values you read do not reflect it in the same task.

**Fix.** Put the preference in storage and *load the page*. One page load per state.

```js
const page = await browser.newPage();
await page.evaluateOnNewDocument((theme, hc) => {
  try {
    localStorage.setItem('site-theme', theme);
    localStorage.setItem('site-high-contrast', hc);
  } catch (e) {}
}, 'dark', '1');
await page.goto(url, { waitUntil: 'domcontentloaded' });
```

This is also the honest test. A returning visitor arrives exactly this way: storage already set,
page loading fresh. Toggling in a live page tests a path most of your users never take.

---

## 4. A page with no CSS looks exactly like a page with broken colours

**Symptom.** One site reports a full page of failures. Every element is black text on white, or
white on white. The site renders perfectly in a browser.

**Cause.** The stylesheet did not load. To a script that reads computed styles, "no CSS" and
"every colour is wrong" are the same observation. Mine reported the second.

This happens most often when several pages navigate concurrently through one request-interception
handler, and the handler races.

**Fix.** Gate on a rule you know your stylesheet sets, and **throw** rather than record a result.

```js
const styled = await page.evaluate(() => {
  const probe = getComputedStyle(document.body).getPropertyValue('--color-bg').trim();
  return probe !== '' || document.styleSheets.length > 0;
});
if (!styled) throw new Error('stylesheet did not load: ' + url);   // an error, never a data point
```

Pick a probe that is unambiguous. A custom property your stylesheet defines is ideal, because
`document.styleSheets.length` counts an empty sheet from a failed request.

**And do not share one interception handler across concurrent navigations.** Either serialise the
pages, or give each page its own handler bound to its own root.

---

## 5. Percentage heights do nothing inside a `<picture>`

**Symptom.** An image escapes its container on some pages and not others. Usually the portrait
ones.

**Cause.** `<picture>` is `display: inline` by default and has no height of its own. A child
`img { height: 100% }` resolves against an element with no height, so it falls back to auto, so
the image renders at natural size and pushes out of the box.

This one has bitten me three times on the same portfolio, always after someone converts a plain
`<img>` to a `<picture>` for AVIF and WebP sources. The markup change is invisible in review.

**Fix.** Make the `<picture>` a real box.

```css
.hero picture { display: flex; width: 100%; height: 100%; }
.hero picture img { width: 100%; height: 100%; object-fit: contain; }
```

**Detect it** rather than trusting review, since it only shows on certain aspect ratios:

```js
const overflow = await page.evaluate(() => {
  const out = [];
  document.querySelectorAll('picture img').forEach(img => {
    const p = img.closest('picture').getBoundingClientRect();
    const r = img.getBoundingClientRect();
    if (r.height > p.height + 2 || r.width > p.width + 2) {
      out.push({ src: img.currentSrc.split('/').pop(),
        img: Math.round(r.width) + 'x' + Math.round(r.height),
        box: Math.round(p.width) + 'x' + Math.round(p.height) });
    }
  });
  return out;
});
```

---

## 6. Your working copy is behind production

**Symptom.** None, until after the deploy.

**Cause.** Anything that can write to the live site without writing to your tree. Another session,
a colleague, a CMS edit, a bot commit, a hotfix applied directly.

On the run that produced this file, a session four hours earlier had rewritten every internal link
on the live sites to a cleaner form and never saved it back to the working tree. Deploying would
have silently reverted 2,584 links. The version markers matched. The stylesheets matched. The
structured data matched. Only a page-by-page comparison found it.

**Fix.** Snapshot live before you deploy, and diff every page, not a sample.

```js
import fs from 'node:fs';
const norm = s => s.replace(/\s+/g, ' ').trim();     // whitespace only; do not normalise content
for (const p of pages) {
  const live = norm(await (await fetch(origin + p)).text());
  const local = norm(fs.readFileSync(localPathFor(p), 'utf8'));
  if (live !== local) console.log('DIFF ' + p);
}
```

If a diff appears, **read it before you resolve it**. The live version is sometimes the newer one.
That is the entire point of the check.

Before adopting the live copy wholesale, confirm the differences are the ones you expect: a diff
that includes a cache-busting token or a build timestamp is noise, and one that rewrites two
thousand links is a change somebody made on purpose.

---

## 7. A gate that cries wolf gets ignored

**Symptom.** Your deploy script reports a catastrophic regression. It did not happen.

**Cause.** A single failed network request reads identically to a real absence. Mine announced
that a site had lost its entire content security policy. One fetch had failed, and a failed fetch
returns no headers, and no headers reads as "the header is gone".

The check itself was worth having. An earlier deploy from the wrong working directory had stripped
every security header while reporting success.

**Fix.** Retry before you believe a bad answer, and never let one failed request stand as evidence
of absence.

```js
async function head(url, tries = 3) {
  let last;
  for (let i = 0; i < tries; i++) {
    try {
      const r = await fetch(url, { method: 'HEAD', cache: 'no-store' });
      if (r.ok || r.status < 500) return r;
      last = new Error('status ' + r.status);
    } catch (e) { last = e; }
    await new Promise(r => setTimeout(r, 400 * (i + 1)));
  }
  throw last;              // throw, so the caller knows it could not measure
}
```

The distinction that matters: **could not measure** and **measured, and it is bad** must be two
different outcomes in your reporting. Collapse them and you train yourself to click past the gate,
which costs you the one time it is right.

Two more small ones from the same script:

* A gate that required a generated stylesheet to end with a closing brace failed every time the
  file ended with a comment. Strip comments before structural checks.
* A deploy that fails once and succeeds on retry is normal. Retry the deploy itself twice before
  reporting a failure, and log that the retry happened.

---

## The harness self-check

Before you trust a run, prove the instrument can still fail. Point it at a page you have
deliberately broken and confirm it says so.

```js
// A page whose only stylesheet is a 404 must produce an error, not a result set
// A page with #000 on #000 must produce exactly one failure at ratio 1.00
// A page you know is clean must produce zero
```

If any of those three does not come out right, the numbers from the real run mean nothing.

**Pre-run checklist**

- [ ] Service worker stubbed, and browser cache disabled
- [ ] Page brought to front; no `waitForFunction` left on rAF polling
- [ ] State set via storage before load, one page load per state
- [ ] Missing-stylesheet condition throws, and is reported separately from failures
- [ ] Concurrent navigations do not share one interception handler
- [ ] Live snapshotted and diffed against local before deploy
- [ ] Every network assertion retries before reporting absence
- [ ] Harness verified against a known-broken and a known-clean page

---

Companion file at https://jwatte.com/downloads/

* `display-state-audit-checklist.md` : the state matrix, the contrast and spacing passes you can
  paste into DevTools, and the derivation targets

Free checkers at https://jwatte.com/tools/ : text contrast, WCAG accessibility audit, form
accessibility, and about seventy others. Browser-side, no signup, nothing uploaded.

Written by J.A. Watte. https://jwatte.com