When two AI models start a debate disagreeing and end it agreeing, that looks like the debate working. A new paper says most of the time it's the opposite.
The paper is arXiv:2606.00820, "Not All Flips Are Conformity," out May 30. It goes after a number almost every multi-agent debate system quietly relies on: the answer flip rate — how often an agent changes its answer after seeing what the others said. High flip-to-consensus rate, the assumption goes, means the models are reasoning together and converging on truth. The paper's claim is that the flip rate is measuring three completely different things at once, and the one that dominates is the one you don't want.
The number under the number
Here's the setup. The authors — Xiqi Hao, Zengqing Wu, Chuan Xiao, Jianbin Qin and colleagues across the University of Osaka, Shenzhen University, and Beijing Institute of Technology, Zhuhai — run multi-agent debate on MMLU-Pro and decompose every flip into three sources. Spontaneous instability: the model would have changed its mind anyway, on its own, given another pass. Stance-induced conformity: it changed because the other agents asserted a position, not because they made a case. Reasoning-induced persuasion: it changed because someone's actual argument was better.
Then they isolate each one with controlled counterfactuals — including a self-reflection condition where an agent reconsiders alone, with no peers in the room.
The self-reflection result is the first crack. In the primary MMLU-Pro setting, 37% of agent-question observations change under self-reflection alone. More than a third of what looks like "the debate moved me" is just a model being unstable — it would have wobbled with or without the council. That's noise being scored as deliberation.
Strict conformity comes in at 29% of the primary setting. And here's the part that should stop you: across model replications, that conformity is 57–77% correct-to-wrong. When an agent caves purely to peer stance, it's flipping from a right answer to a wrong one more often than not. The replication tests — GPQA-Diamond, three model families — show the same shape with model-dependent swings.
A flip looks like a mind changed by an argument. Most of the time it's a mind changed by confidence.
Vacuous reasoning still wins
The sharpest experiment in the paper is the information-gradient test. The authors hold the content of an argument fixed and vary how much real reasoning it contains — down to reasoning that is deliberately empty, the argumentative equivalent of clearing your throat with conviction. Then they measure how often resistant agents — the ones that held their ground — adopt the wrong answer anyway.
Even vacuous reasoning drove 20–39% error adoption among those resistant agents. The shape of an argument — "here's my reasoning: ..." followed by nothing load-bearing — carried substantial persuasive weight on its own. The models aren't weighing logic. They're responding to the costume logic wears.
If you've spent any time watching AI systems talk to each other, this lands as obvious and damning at the same time. The thing that makes a frontier model good at sounding right is exactly the thing that makes it dangerous in a room full of other models that also sound right. Confidence is contagious and content-blind.
You can predict it. You still can't fix it cheaply.
The paper doesn't stop at diagnosis, and this is where it gets genuinely useful for anyone building councils. Harmful conformity is predictable from Round 0 features — the state of the debate before anyone has flipped — at AUC = 0.79. A risk-targeted intervention aimed at the flips most likely to be harmful cuts harmful conformity by 13.6 percentage points, p < 0.001.
Then comes the sting in the tail. Without correctness labels or self-reflection controls, reducing peer adoption does not improve accuracy — because harmful and beneficial influence can't be told apart from the outside. You can suppress conformity, but if you suppress it blindly you also suppress the genuine persuasion, the cases where one model actually talked another out of a mistake. Cut indiscriminately and you lose the real deliberation along with the fake.
That's the whole problem with the naive council in one sentence. The flip you want and the flip you don't look identical at the output layer. You only tell them apart by knowing what each agent would have done alone, and what was actually argued.
Why this is a protocol problem, not a model problem
The instinct, reading this, is to want better models — ones that don't fold under social pressure. That's the wrong lever. The conformity is structural. It comes from running agents in a flat room where stance is visible and reasoning is optional.
The fixes are architectural, and they're the ones the council literature has been converging on all spring. Anonymize peer identity so an agent can't defer to the model it thinks is smarter. Structure the exchange so an argument has to carry actual reasoning to count — the formal version of Red Team vs Blue Team, where the job is to attack the claim, not assert a side. And put a Chairman in the synthesis seat whose job is to weigh the reasoning, not count the votes — because a vote count is exactly the signal this paper shows you can't trust.
Run four models in parallel and let them agree, and you've built a machine that launders conformity as consensus. The agreement is real. What produced it usually isn't deliberation. This is the difference between a council and a focus group that talked itself into the confident answer.
When a model in a Shingikai council holds a position the others abandoned, that disagreement is the signal worth keeping — not the noise to average away. The flip rate was never the metric. What changed the mind always was.
Try it free — no signup. shingik.ai