Research & experiments

Every rung placement in the cost ladder cites a measurement. These are the experiments that produced those measurements — what was tested, what worked, and what was refuted.

The protocol is the same in every experiment: thresholds are chosen on the training data, the held-out test set is touched once, and the bar is acted precision — how often the system is right about what it sends.

Rung 4 — similar-example matching

Question. Can a nearest-neighbor search over labeled examples resolve what the statistical classifier escalates?

Answer. Yes. Unanimous agreement among the 5 nearest examples acts on 10 of the 50 escalated replies at 100% precision, dropping the escalation rate from 66.7% to 53.3% with zero wrong actions.

We also tested PrismDB’s hybrid search pipeline as the matching runtime. Its query fusion — designed for structured catalog queries, not conversational paraphrase — scored below the statistical classifier on both test corpora. The deficit was in the fusion strategy, not the embedding quality. Verdict: the matching concept was adopted; PrismDB’s search pipeline was refuted as the runtime. The dense path runs on a lightweight CPU model instead.

PrismDB remains the runtime for the rung-6 judge (below) and the planned archive layer — different workloads where its multi-dimensional search excels. See PrismDB docs for more on its architecture.

Key numbers

Rung 5 — fine-tuned classification head

Question. Does a small transformer trained on 303 labeled examples reach 100% acted precision with coverage the cheaper rungs cannot buy?

Answer. Yes, for one of two approaches. A 15-way classification head on the same backbone as the rung-6 judge, read through the agreement rule (both the head and the statistical classifier must agree), acts on 44/44 held-out replies. With rung 4 in front: 47/47, escalation down to 37.3%.

We also tested a fine-tuned entailment head on the same backbone. It was the strongest judge on the training residue (208/233 correct) but accepted 3 wrong on held-out (52/55) — refuted at the bar. The failure is in the hypothesis framing on genuinely ambiguous replies, not in the model quality. The classification head became the rung-5 runtime.

Key numbers

Rung 6 — meaning-match judge controls

Question. Is the PrismDB judge better, equal, or worse than its reference runtime? Is 97% the best any judge can do on this residue?

Answer. The PrismDB judge is numerically identical to the reference implementation of the same model (4,620 scored pairs, maximum score difference 0.004, zero decision flips) — and it is the fastest (21 ms vs 33 ms for the reference runtime). Two larger models under the same protocol miss the same held-out sentence. 97% is a ceiling of the agreement method on this residue, not of any particular model.

The one sentence every judge gets wrong — “This really isn’t something I’m involved in” — is genuinely ambiguous between “wrong person” and “not interested.” Both map to the same withdrawal action: no wrong email was sent (36/36 by action). The remaining miss is label precision, not action precision.

See the PrismDB documentation for how the judge sidecar works.

Key numbers

The refutation ledger

Every lever that failed its measurement is kept on record so nobody re-tries it by intuition:

The cost ladder page carries the full doctrine and current cascade numbers.