Line up five AI models to judge the same answer. Suppose each one, on its own, gets it right. The panel can still hand you the wrong verdict.
That is not a broken prompt or a bad model. It is what majority voting does to a room of judges, and a new NeurIPS 2025 paper (arXiv:2510.12697) puts the mechanism on record: current LLM-as-Judge setups "rely on simplistic aggregation methods (e.g., majority voting), which can fail even when individual agents provide correct answers." Then it does the harder thing. It proves that making the judges argue recovers what the vote throws away.
The judge became infrastructure while nobody was looking
The LLM-as-Judge is now load-bearing. Models grade other models' answers in eval pipelines, in RLHF reward loops, in the gates that decide whether an agent's output ships. The pattern practitioners keep posting about is always the same — a single GPT-5-class judge sails through every test case you wrote, then waves through the one that breaks production. The reflex fix is to add more judges and take a vote. "Multi-Agent Debate for LLM Judges with Adaptive Stability Detection," from Tianyu Hu, Zhen Tan, Song Wang, Huaizhi Qu and Tianlong Chen, says the vote is the part that fails.
A poll is not a deliberation
Here is the move worth borrowing. A vote counts hands. It does not weigh who is right. When five judges answer independently and you tally the majority, a correct minority gets outvoted by a confident, wrong one, and no amount of individual accuracy saves you. The paper's alternative is a debate judge — the agents "collaboratively reason and iteratively refine their responses" instead of answering once and going silent. A static ensemble asks each judge in isolation. A debate lets the right one change the room's mind.
Take the gate in a real eval pipeline. Five judges score a model's answer to a hard question. Three of them, reasoning fast, land on the same plausible-but-wrong read. Two do the harder work and get it right. A vote ships the wrong score on a 3-2 majority, with a clean audit trail. Let the five see each other's reasoning, and the two who did the work get to show it while the fast three have to defend a position that does not hold. Same five models. Opposite verdict. The only thing that changed is whether they were allowed to argue.
The spine is a proof, not a benchmark
Most council papers wave a leaderboard. This one reaches for a theorem. The authors "formalize the debate process mathematically, analyzing agent interactions and proving that debate amplifies correctness compared to static ensembles." Read that against the last year of the field and you see why it lands. A running argument — sharpest in the NeurIPS 2025 "Debate or Vote" result — is that when agents just argue and then vote, the arguing nets to nothing and the vote does all the work. This paper stakes the opposite claim on the structured version. The exchange itself adds correctness a static tally cannot, and it shows the math for why.
The distinction the field keeps circling is static ensemble versus debate. Both put several models on one question. Only one lets them see and revise against each other. The paper's premise — majority voting fails even when the individual answers are correct — is the cleanest case anyone has made for why counting is the wrong operation at the synthesis layer. You do not want the answer more judges landed on. You want the answer that survives the argument.
That is the design decision underneath every AI council, and it is the one Shingikai made on purpose. A council here does not end in a headcount. It ends in Chairperson Synthesis — a synthesis layer whose job is to weigh a disagreement rather than settle it by majority. The NeurIPS paper is an independent, peer-reviewed argument for that exact choice: a synthesizer that reads who is right beats a counter that tallies who agrees. When the correct judge is outnumbered, a vote buries it and a synthesis layer can still surface it.
Knowing when to stop is its own primitive
The paper's second contribution answers a question the council literature has been dancing around for months. When is the argument done? Debate that runs forever is a tax, not a feature. The authors add a stability-detection mechanism that "models judge consensus dynamics via a time-varying Beta-Binomial mixture, with adaptive stopping based on distributional similarity (Kolmogorov-Smirnov test)." Strip the statistics and the idea is disciplined — watch how the panel's agreement is moving, and stop the moment it stops moving. It is the difference between a meeting that ends when the decision is made and one that ends when the clock runs out. The reported result is judgment accuracy above majority voting while holding compute in check.
None of this makes debate a free lunch. The honest read of the field is that whether the arguing pays depends on structure and on who is in the room — a panel of near-identical models mostly agrees with itself, argument or no. This paper's contribution is not "debate always wins." It is a boundary drawn in math: under a structured protocol, the exchange provably adds something the vote cannot, and there is a principled way to know when to stop.
Weigh the judges, do not count them
A year of council papers is finally legible. The disagreement was never the mess to clean up. Counting was. A vote is what you reach for when you have decided in advance that the majority is the truth, and a room of judges is exactly the place that assumption breaks. Weigh the judges, do not count them, and know when to stop.
Try it free, no signup. shingik.ai