# Your confidence score is one model rating its own homework

A cross-model agreement signal was null-safe by design, so any timeout from the second model produced the same badge from one model alone. The UI was identical.

Author: J.A. Watte
Published: September 8, 2026
Source: https://jwatte.com/blog/blog-ensemble-confidence-theatre/

---

A confidence signal was designed properly. A second, cheaper model reads the same inputs independently, and the primary model sets the label by cross-referencing that read against its own. Genuine cross-model agreement, which is a real thing and worth having.

The failure path was null-safe by design. Any timeout, any error, any refusal from the second model meant the primary proceeded alone and still produced a label.

So the badge said the same thing whether two models had agreed or one model had rated itself, and there was no way to tell from the artifact, the interface, or the logs which had happened.

That is the whole subject of this post. **An epistemic signal is worth exactly what its independence is worth, and independence has to be written onto the artifact and asserted, not assumed from the design.**

## A graceful fallback turns an ensemble into a solo act

Null-safe fallback is good engineering nearly everywhere. It is the wrong default here, because the value of the output depends entirely on the thing you are falling back from.

A retry that falls back to a cached price is still a price. A cross-check that falls back to one opinion is not a cross-check. It is one opinion wearing the interface of a cross-check, which is worse than one opinion honestly labelled, because the reader has been told to trust it more.

Write provenance onto the artifact rather than into a log line:

```json
{
  "label": "consensus",
  "_independent_read": { "model": "…", "reads": 1, "filtered": 0 }
}
```

Then assert it, across runs, not once:

```bash
jq -e '._independent_read != null' artifact.json
```

A single spot check passes on the day you look. Assert non-null across N consecutive scheduled outputs, because the failure mode is intermittent by nature, and intermittent is exactly what a spot check cannot see.

The stronger version of the check is to break it deliberately: force-disable the second model and compare the label distribution against a normal run. **If the distribution barely moves, the second model was never contributing much**, and you have learned that whether or not it was timing out.

## A signal produced by one call is theatre

The same system had an "analyst council" feature that staged disagreement by having a single model call role-play several analysts. An early version of the confidence badge was the same model rating its own writing.

Both look like epistemic machinery. Neither contains any independent evidence. A model asked to argue with itself produces the shape of a debate, and the shape is the product being sold, which is the problem.

The assertion is structural, not qualitative:

```js
assert(artifact.models.length >= 2, 'field derived from a single inference call');
```

At least two distinct inference calls must contribute to the field. Not two prompts in one call. Not one call instructed to consider multiple perspectives. Two calls, recorded, countable.

This is not a claim that a single model's self-assessment is worthless. It is a claim that it is a different product from cross-model agreement, and it must not be labelled as the second.

## A signal that never varies is decoration

The second assertion is even cheaper and it catches the case where the machinery is genuinely independent and the output still means nothing.

Histogram the label across a full run:

```bash
jq -r '.label' artifacts/*.json | sort | uniq -c | sort -rn
```

If one value covers more than about 90 percent of items, the signal is decorative regardless of how carefully it is computed. A badge that says "high confidence" on 97 percent of everything is a logo.

This is worth running on any derived label you ship: a risk rating, a quality score, a confidence badge, a priority. The distribution is the product. Nobody looks at it because the individual values look reasonable one at a time.

## The provenance only exists on the path that does not run

This one cost me real time and it is the subtlest in the set.

Two code paths produced the same artifact. A synchronous path, used by manual triggers and by the watchdog. And an asynchronous batch path, used by the actual schedule.

The synchronous path stashed the second model's read onto the artifact. The batch path wrote the primary model's raw output straight through.

So the verification method worked perfectly, and proved something true only about the path that almost never ran. Every manual check confirmed the provenance was present. Production had never written it.

```bash
# Produce one artifact by each path and diff their shapes
jq -S 'keys' manual.json > a
jq -S 'keys' scheduled.json > b
diff a b
```

Any key present only on the manual side is a verification method that does not apply to production. This generalises well past provenance: **whenever you have a manual path and a scheduled path, diff their outputs' shapes, because the manual one is the one you test with and the scheduled one is the one that runs.**

The stronger habit is to verify against stored artifacts from the last N scheduled runs rather than by triggering one yourself. Triggering one yourself uses the wrong path by definition.

## Write the rubric so the label measures the substrate, not the volume

A label of consensus, contested or developing sits over items where sources frame an event differently.

The naive reading rates how loudly sources disagree. That makes the metric a measure of rhetoric, and it will mark an item contested when every source agrees on what happened and disagrees about what it means.

Written the other way, the label rates the factual substrate beneath the framing: do the sources agree on what occurred, regardless of how they characterise it.

Both rubrics produce a label. Only one of them measures something a reader can use. And you cannot tell which one you have from the code, because the code is a prompt.

Build the adversarial case as a fixture:

> An item where sources agree on every fact and disagree violently on framing. Assert the label is the agreement value, not the contested one.

Then hand-label twenty real items against the written definition and measure agreement with the model. Where they disagree, the usual culprit is that the written definition is ambiguous, not that the model is wrong. Fix the definition first.

## A member that declines a class of input is a silent coverage hole

One model in a two-model ensemble was subject to a different regulatory regime and would not give a straight read on a defined class of topics.

The mitigation was a keyword guard that removes those items from that model's input and routes them to the other model, which rates them alone. Reasonable. But it means a subset of your output silently reverts to the solo-act case, and the subset is defined by topic, which means it is not random. It is exactly the topics somebody chose to be careful about.

Two invariants, both one line, and both were exposed in a single success log:

```js
assert(ratedCount === inputCount, 'items exist with no rating from any path');
assert(filteredCount / inputCount < 0.15, 'the guard has become over-broad');
```

The first catches items falling through both paths. The second catches the guard widening over time, which keyword guards do, because every incident adds a keyword and nothing ever removes one.

## What to check on your own system this week

Four commands, and each one answers a question you probably cannot answer today.

**Does the field record its own provenance?** `jq -e '._independent_read != null'` across your last twenty scheduled artifacts, not one.

**How many distinct inference calls contribute?** If the answer is one, rename the field. Self-assessment is a real signal and it is not agreement.

**What does the distribution look like?** Histogram the label. Over 90 percent in one bucket means you are shipping a logo.

**Do your manual and scheduled paths produce the same shape?** `jq -S 'keys'` both and diff.

The uncomfortable summary is that all four of these failures produce output that looks better than honest output would. A missing provenance field, a solo model, a uniform label and an untested production path all render as a clean, confident interface. That is the reason they survive.

If you want the wider map of building this kind of thing without a team, that is what I wrote **The $20 Dollar Agency** for.

## Related reading

**[Three failures 773 milliseconds apart across three days is not the model](/blog/blog-duration-signature-debugging/)**: the wrapper that returns null on every failure is what makes the fallback in section one invisible.

**[A second check on the corrected text found more errors than the first](/blog/blog-generated-content-pipeline-stages/)**: the same pipeline, and why a correction stage is a generator.

**[When LLMs Get Your Brand Wrong](/blog/blog-tool-ai-hallucination-detector/)**: measuring model claims about a real entity, where independence matters for the same reason.

**[The check was green because it measured a state no visitor is ever in](/blog/blog-green-checks-that-cannot-fail/)**: the general form, including the rule that a uniform result is evidence about your instrument.

## Fact-check notes and sources

**Ensembling improves reliability only when errors are uncorrelated**: the value of combining predictors comes from their independence, which is the property a silent fallback destroys. [Dietterich, Ensemble Methods in Machine Learning](https://link.springer.com/chapter/10.1007/3-540-45014-9_1)

**Models exhibit self-preference when evaluating text**: an LLM judging output, including its own, is a measurement with a known bias, which is why self-assessment and cross-model agreement should not share a label. [Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations](https://arxiv.org/abs/2404.13076)

**LLM-as-a-judge agreement is measured against human labels, not assumed**: the standard practice is to report agreement rates, which is what the twenty-item hand-labelling step in the rubric section is doing. [Zheng et al., Judging LLM-as-a-Judge](https://arxiv.org/abs/2306.05685)

*This post is informational, not engineering-consulting advice. The systems described are anonymised and no client or vendor is identified. Mentions of third parties are nominative fair use and no affiliation is implied.*


---

Canonical HTML: https://jwatte.com/blog/blog-ensemble-confidence-theatre/
RSS: https://jwatte.com/feed.xml
JSON Feed: https://jwatte.com/feed.json
Hero image: https://jwatte.com/images/blog-ensemble-confidence-theatre.webp