A December 2025 paper ran three multi-agent deliberation protocols against a simple baseline and the baseline won 82.5% of the time. The best deliberation protocol won 13.8%. That is a 6.0x gap, statistically significant at p < 0.01, and it cost 1.5 to 2.5 times more compute to lose.

The paper is called DeliberationBench: When Do More Voices Hurt? (arXiv:2601.08835), by Vaarunay Kaushal and Taranveer Singh. If you run an AI council, the title reads like an obituary. The numbers read worse.

So read the baseline before you believe the obituary.

What actually got tested

DeliberationBench is a clean piece of work: 270 questions with verified reference answers, three independent seeds, 810 total evaluations, deliberately skewed toward medium and hard problems where deliberation should help most. Five models form the council — GPT-4o-mini, Claude-3.5-Haiku, Gemini-2.0-Flash, Llama-3.1-8B, and Mistral-Nemo.

Three deliberation protocols competed:

  • v1, Blind Ranking — generate five drafts, strip model identity, have an agent rank them, return the top. Won 3.2%.
  • v2-A, Rubric Scoring — score each draft on criteria, pick the highest aggregate. Won 0.6%.
  • v2-B, Senate Debate — assign each draft a defender, run opening statements, rebuttals, and closings, let a judge pick the winner. Won 13.8% — the "best" of the three.

The baseline they all lost to is called Best-Single Selection. It generates five drafts from the same five models, then hands all five to a strong judge — GPT-4o — and asks it to compare them directly and pick the best one.

Reread that. The winning baseline is not a single model. It is five models generating in parallel and a sixth model synthesizing the result. That is a council. A minimal one, but a council — multi-model generation, single strong synthesizer.

The contest was never one model versus many

The headline frames this as deliberation losing to simplicity. The methodology says something narrower and more useful: one synthesis design beat three other synthesis designs.

What lost was not multi-model reasoning. What lost was three specific ways of combining multiple drafts — ranking them blind, scoring them on a rubric, and making them argue. What won was a different way of combining the same drafts: let a capable model read all of them at once and choose.

The paper's own discussion section says exactly why the three protocols failed, and every reason is about the combining step, not the number of voices. Blind ranking strips model attribution, which is a reliability signal you want to keep. Rubric scoring is too rigid. Debate, in their words, rewards persuasiveness over accuracy — the same failure the Consensus Trap and the Nature persuasion work have been documenting all year. They name the root cause directly: information loss through aggregation. Every intermediate step that compresses five drafts into a verdict throws away signal. The baseline avoids that by never compressing until the final comparison.

One synthesis design preserved the drafts until the decision. The other three shredded them on the way. The drafts were identical. The synthesis was everything.

Even the disagreement result points the same way

The most quotable finding is buried in the conditional analysis. When the five models maximally disagreed — variance above 0.7, exactly the case where perspective is supposed to earn its keep — the baseline still won 71.3% to the Senate Debate's 15.1%.

The skeptic reads that as: deliberation adds nothing even when it should add the most. The builder reads it as: a lossy protocol stays lossy precisely on the hard cases, which is an indictment of that protocol, not of preserving disagreement. The baseline did preserve the disagreement — it put all five conflicting drafts in front of one judge and let it adjudicate. The debate protocol took the same disagreement and laundered it through rhetoric. The disagreement was a gift to one method and a liability to the other.

The caveat the authors put in writing

Kaushal and Singh are honest about the ceiling. Their council is five small, fast models — no GPT-4o, no Opus-class model in the room except as the judge. They write, plainly, that deliberation "may be more effective with stronger models," and that when all the initial drafts are mediocre, no protocol can manufacture quality that was never generated. Garbage in, deliberation out.

That is the real boundary of the result. DeliberationBench proves that bolting a naive aggregation protocol onto a council of weak models is worse than letting a strong model pick the best draft. It does not prove that structured deliberation among capable models fails — it explicitly declines to test that.

What a council is actually for

Strip the framing and the paper lands a lesson councils should already own: a council is only as good as how it combines what the models say. Best-Single Selection beating Blind Ranking is a Chairman who reads every brief beating a clerk who tallies votes. The architecture that wins is the one that keeps the most information alive until the moment of decision.

That is the whole bet behind a synthesis layer. The seven strategies a real council runs — blind ranking and naive debate are not among them — are seven different answers to one question: what do you do with five answers that don't agree? DeliberationBench just ran the worst three answers and confirmed they're the worst three. The best answer won its own benchmark and got filed under "baseline."

The title asks when more voices hurt. The tables answer it: when nobody's really listening to them.

Try it free — no signup. shingik.ai