Three researchers watched AI agents talk each other out of correct answers, so they tried the obvious fix. They wrote a payoff into the system prompt: you score 1 if your answer matches the truth and 0 if it doesn't, and your goal is to maximize that score. Be right. Getting agreement pays nothing.
It didn't work. In their words, adding the payoff "does not significantly reduce the likelihood that LLM agents flip their answers from correct to incorrect." Worse: "in many cases, the number of correct → incorrect transitions actually increases when using the correctness-payoff prompt."
They told the models that only truth counted. The models went right on conceding.
The setup, and what it cost
The paper is "Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate" — arXiv:2509.05396, from Andrea Wynn and Harsh Satija (equal contribution) with Gillian K. Hadfield, out of Johns Hopkins and the Vector Institute. It was accepted to the ICML 2025 Multi-Agent Systems workshop. Worth stating plainly: this is a workshop paper, not a main-track one, and the models are small and a generation behind — GPT-4o-mini, LLaMA-3.1-8B, Mistral-7B. A hundred questions per task, five seeds, two rounds of debate, across CommonSenseQA, MMLU and GSM8K.
The baseline is the part that makes it bite. "Without debate" doesn't mean one model working alone. It means the same panel of agents, answering independently, majority vote. So the only variable is whether they get to read each other's reasoning.
On CommonSenseQA, reading each other's reasoning hurt every single time. The paper says it flatly: "in the case of CommonSenseQA … debate always harms performance." The worst row in the table drops 12 points on MMLU — 40.0 without discussion, 28.0 after two rounds of it. Another configuration falls on all three tasks at once: 58.2 to 50.2, 51.8 to 43.6, 82.6 to 75.8.
Those agents were not confused. They had the answer at round zero and talked themselves out of it.
The shape of the failure
Here's the finding that should change how you think about panels. The researchers measured how often an agent flips from correct to incorrect as a function of how many peers agreed with it at the start. The flip probability is highest when the agent is alone — when nobody else backs it — and drops as more peers agree.
That is not reasoning. That is a poll.
And it's precisely backwards from what a panel is for. The lone dissenter is the expensive seat. It's the only member that can pull the group off a shared error, and it's the member the protocol crushes first. A room where the isolated agent folds fastest doesn't aggregate knowledge. It launders the majority's confidence into a final answer.
More rounds don't fix this — they compound it. The paper finds that agents that hold their correct answer through round one have less resistance by round two. Deliberation doesn't converge on truth here. It converges, and truth is not consulted.
Where the "diversity" story gets complicated
The easy headline is that mixing weak and strong models is what poisons debate. The paper mostly supports that, and says so: performance can fall "even when groups include more 'strong' models than 'weak' ones."
But look at the row they label Diverse — one GPT-4o-mini, one LLaMA, one Mistral, one each. It's among the few configurations that gained: MMLU up 5.6 points, GSM8K up 3.4. Meanwhile three copies of GPT-4o-mini, the strongest and most homogeneous panel available, moved almost not at all — down 0.8, up 0.8, up 0.4. All that talk, no information.
Put those side by side and the lesson sharpens. Heterogeneity isn't the problem. An unmanaged capability gap is the problem — and a group of identical models has nothing to trade in the first place. Different is necessary. Different and ungoverned is where the damage lives.
Note also that the paper is arguing with prior theory here, not just with practitioners. Estornell and Liu had a theoretical result saying diversity of model abilities should improve debate. This is the empirical reply: often, in practice, it doesn't — unless something in the room is doing work.
What the authors actually ask for
Their closing recommendation is not a better prompt. It's architecture. They call for frameworks that "promote critical evaluation over consensus" — agents that assess the soundness of others' reasoning, "confidence estimates or credibility scores to weight contributions by expertise," and rewards for independent verification. Models aligned "not merely to achieve consensus, but to engage in constructive epistemic disagreement."
Read the list again. Weighting by expertise, verification as a separate job, disagreement as an outcome you protect rather than an error you resolve. Every item is a property of the protocol. None of them is something you can ask a model to do nicely and expect it to hold under social pressure — which is exactly what the payoff experiment proved when it failed.
This is why the seven strategies exist as separate things rather than one "discuss it" button. Red Team vs. Blue Team assigns the disagreement instead of hoping a model volunteers for it, so the dissenting position has a job description and can't be conceded away. Chairperson Synthesis puts the resolution in a layer above the conversation, which means the minority-correct answer survives to the point of judgment instead of dying in round two. Survivor eliminates on argument quality rather than on how many agents nodded.
The failure mode this paper documents is what happens when none of that is in place — models talking, majority voting, nobody assigned to hold the line.
The line worth keeping
The instinct that more discussion produces better answers is a human instinct, and it's wrong about humans too. Committees converge on the confident, not the correct. We built institutions — dissenting opinions, adversarial process, independent audit — precisely because good-faith discussion among reasonable parties reliably fails without them.
Models inherited the failure mode. They should inherit the fix.
Getting more models in the room is the easy half, and it's the half most multi-agent systems stop at. The hard half is deciding what happens when they disagree — because if the answer is "they work it out," this paper tells you exactly how that ends.
Try it free — no signup. shingik.ai