Two legal AIs hallucinate at the same rate — about 52%. One is safe to put in front of a compliance officer. The other will quietly get a lawyer sanctioned. And the number everyone uses to compare them says they are identical.
That gap is the subject of a new paper, LegalHalluLens (arXiv:2606.18021), presented at the Second Workshop on Agents in the Wild at ICML 2026. It audits AI hallucination on legal contracts at a scale most evaluations never reach: 510 contracts and 249,252 clause-level instances drawn from CUAD. And its first finding is that the headline hallucination rate — the single number procurement teams ask for — is close to useless on its own.
The average is hiding the thing you need to know
A hallucination rate tells you how often a model is wrong. It never tells you how it is wrong. LegalHalluLens breaks the errors into four legally-motivated categories — numeric, temporal, obligation/entitlement, and factual — and finds a within-model gap of roughly 38 to 40 percentage points between the categories the model handles well and the ones it fumbles. The same model that reliably reads obligations might invent dates wholesale. Average those together and you get a tidy 52% that describes neither.
It gets sharper. The paper introduces a Risk Direction Index — a single scalar that captures whether a model tends to omit (miss a clause that is really there) or invent (fabricate a term that isn't). Two systems with matched 52% rates, the authors show, can carry opposite RDIs. One is failing by leaving things out. The other is failing by making things up. In a contract review, those are not the same risk. An omission gets caught on a careful second read. An invention gets signed.
One number, two completely different failures
This is the move worth sitting with. A single hallucination rate flattens a model into a score. A typed, directional profile turns it back into a behavior you can govern.
If you only know the 52%, your options are bad. You either trust the model or you don't. If you know the model omits obligations but rarely invents them, you know exactly where a human has to look — and where you can let it run. Direction-aware procurement, the authors call it. The number was never the point. The shape of the errors was.
Where this stops being an audit and becomes a council
Here is the part that should interest anyone building multi-model systems. Once you know where and in which direction a model fails, you can do something most "add a verifier" designs never do: aim the verification.
The dominant pattern for catching AI mistakes is to add a second pass — a Skeptic agent, a debate round, a critic that challenges the output. The usual version of this is generic. The Skeptic challenges everything equally, with no idea where the real errors live. LegalHalluLens feeds its own diagnosis back into the debate. The typed profiles and the RDI become calibration inputs: the Skeptic's challenges and the gates that accept or reject a claim are tuned to the specific categories and directions the audit surfaced.
The result is concrete. The calibrated debate pipeline cuts fabricated detections by 45%, with the per-category gains tracking the diagnosis — the model improves most exactly where it was measured to be weakest. And it does this while matching commercial APIs using a substantially smaller backbone, 4B active parameters. Targeted structure beat both the raw model and the generic debate.
That is the council thesis with a new edge on it. We have argued for a while that structure beats headcount — that a well-designed deliberation protocol does more than throwing more models at a problem. LegalHalluLens adds the next clause: calibrated structure beats generic structure. A Skeptic pointed at the right failure mode is worth more than a Skeptic that suspects everything.
The asymmetric gate is a chairman decision
Look at how the paper builds the verification, and the parts have familiar names. The Skeptic that challenges claims is a Red Team. The asymmetric gates — treating an invention error differently from an omission error, because they carry different downstream cost — are a synthesis decision. Someone, or something, has to decide that a fabricated obligation is more dangerous than a missed one in this contract, and weight the council's dissent accordingly.
That weighting is the whole job of a Chairperson. The synthesis layer's task was never to count votes or average the answers. It is to decide what to do with disagreement — and now, per this paper, to decide it differently depending on which kind of error the disagreement is pointing at. A council that treats every challenge as equally important is leaving the most useful signal on the table.
The side-by-side writes itself. A single model gives you a 52% and a shrug. A flat council gives you a second opinion that challenges everything and surfaces nothing in particular. A calibrated council tells you this model invents temporal claims, weights its Skeptic and its gate accordingly, and hands the human reviewer a short, specific list instead of a vague warning.
The question was never whether to add a second model
It was where to point it. LegalHalluLens is a legal-AI paper, but the lesson generalizes to any domain where a confident wrong answer is expensive: measure where your model fails, in which direction, and build the deliberation around that — not around a generic instinct to double-check.
One model hands you an average. A council, designed well, hands you the shape of the risk.
If you want to watch models challenge each other on a question where the failure mode actually matters, try it free — no signup. shingik.ai