The most rigorous case against making AI models argue with each other is a NeurIPS 2025 Spotlight out of UW–Madison. It is careful, it is formally proved, and its conclusion is that the arguing does essentially nothing.

Then you read the setup section. Every agent in every experiment was the same model.

What the paper found

"Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?" (arXiv:2508.17536) by Hyeong Kyu Choi, Xiaojin Zhu, and Sharon Li does something most papers in this space skip. It takes multi-agent debate apart into its two separable ingredients and asks which one is actually paying.

Ingredient one: five agents answer independently, and you take the majority. Ingredient two: those agents then read each other's answers and argue for two, three, or five rounds.

Across seven benchmarks — Arithmetics, GSM8K, two slices of MMLU, HellaSwag, CommonsenseQA, and HH-RLHF — ingredient one wins. With Qwen2.5-7B-Instruct, a single agent averages 72.05%. The best debate configuration reaches 73.77%. Plain majority voting, with zero debate rounds, hits 76.91% — the highest row in the table. With Llama3.1-8B-Instruct the pattern repeats: single agent 62.03%, best debate 69.90%, majority voting 72.42%.

Two details make it worse for the debate camp.

More rounds usually made it worse. In five of the six model-and-protocol configurations, accuracy falls monotonically as the debate runs longer. Qwen's decentralized debate goes 73.77% at two rounds, 71.12% at three, 70.50% at five. Llama's centralized debate goes 60.94%, 60.53%, 56.70%. The one exception is Qwen's centralized debate, which climbs slightly across rounds — from a base so far below everything else that it isn't much of a rescue.

That same protocol scored below a single model. Centralized debate on Qwen landed at 65.51% against the single agent's 72.05%. On the Arithmetics task the gap is grotesque: majority voting 99%, the single agent 81.4%, centralized debate 43%. Adding a coordinator and five rounds of discussion cut arithmetic accuracy roughly in half.

The theory explains the general shape. The authors model debate as a Bayesian belief update and prove it induces a martingale over agents' beliefs — the expected belief in the correct answer does not move across rounds. Their own summary: "debate itself does not systematically improve or degrade beliefs on average; rather, belief evolution is driven by stochastic peer influence."

Their section heading is blunter. "Majority voting is surprisingly strong."

The sentence in the setup

Here is the line, verbatim, from the experimental design:

"Following the common setup in prior works, we focus on homogeneous agent settings, i.e., all agents share the same underlying model architecture or behavior."

Five instances of Qwen2.5-7B. Or five instances of Llama3.1-8B. Never both. The agents in this study have identical weights, identical training data, identical blind spots, and identical confident wrong turns.

That is not a council. That is one model talking to itself with extra steps.

And the finding lands exactly where you'd expect once you see it. If five agents share a brain, the only thing the debate can transmit between them is social pressure. There is no information in the room that wasn't in each head at round zero. The martingale isn't a discovery about deliberation. It's a description of what happens when you sample the same distribution five times and then let the samples lean on each other.

What it actually proved

The paper's honest result is a result about ensembling, and it is a good one. Theorem 1 gives majority voting a success bound that tightens as you add agents, and Remark 1 names the mechanism: "even when the correct answer is only marginally more likely than the alternative answers, the lower bound on the success probability asymptotically approaches 1 as N scales."

Sampling one model five times and taking the mode is a variance-reduction trick. It works. It is cheap. And on a homogeneous panel it beats every debate protocol the authors tested.

So take the loss. If your setup is "spin up five copies of GPT-5, have them discuss, take the majority" — this paper just told you the discussion is a tax. Skip it. Take the vote in one round and keep the money.

That is a real and common architecture, and it is now empirically dead.

What it did not test

It did not test a council.

The authors don't claim otherwise. The homogeneity constraint is stated plainly in the methods, framed as following prior work, not as a finding. But the paper is being cited as evidence that multi-model deliberation doesn't pay, and that citation is doing work the experiment never did. No configuration in this study ever put Claude and Gemini in the same room. Nothing here measures what happens when the disagreement between two agents reflects genuinely different training rather than sampling noise.

That distinction is the whole argument. On a same-model panel, disagreement is noise, and averaging noise is the right move. On a mixed panel, disagreement is signal — it marks the places where two differently-built systems reached different conclusions, which is precisely the set of questions where you should not trust either one alone.

A vote tells you what the room thinks. An argument between models that were built differently tells you where the room is unreliable. Those are different products, and only one of them survives being run on five copies of the same brain.

To be fair to the paper: it doesn't test the mixed version either, so nobody should cite it as proof that heterogeneity wins. It defines the question. It doesn't answer it.

The version worth building

None of this rescues the naive council. The lazy pattern — more agents, more rounds, take the majority — is what this paper dismantled, and it deserved it. The round-count decay is the tell. If your debate gets worse the longer it runs, you built a conformity loop and called it deliberation.

The version that earns its cost looks different on both axes. Different models, so the disagreement carries information. And a synthesis layer whose job is to weigh the argument rather than count hands, because the correct answer is frequently the one held by a minority of the room, and a majority vote is a machine for deleting it.

That's the bet Shingikai runs: 200+ genuinely different models, a chairman you choose, and the deliberation streamed live so you can see which model held its position and which one folded.

Five copies of one model voting is a good algorithm. It just isn't a council, and this paper is the cleanest proof yet of the difference.

Try it free — no signup. shingik.ai