A model that had been carefully aligned to a set of values lost to the untrained version of itself. Not on a trivia benchmark. On the one task you would actually reach for a values-aligned model to do: resolve a conflict between two people who want different things.
That is the result sitting inside a March 2026 paper from Integral AI and the University of Tokyo, Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs (arXiv:2603.10476). The authors took a model aligned to a single-agent values objective and ran it head-to-head against the plain base model on conflict-resolution questions. The aligned model won 39.8% of those matchups under greedy decoding and 41.8% under sampling. Both below fifty. The alignment training meant to make it better at value-laden problems made it measurably worse at the part that matters.
The skill alignment quietly removes
Most alignment work happens to one model at a time. RLHF, RLAIF, constitutional methods — you take a single model and push it toward helpfulness, honesty, harmlessness, or some newer dynamic objective. The paper's target here is Collective Agency, a dynamic objective that frames alignment as the continual expansion of meaningful agency rather than a fixed behavioral checklist.
It works, in the narrow sense. The single-agent model does internalize the values. What the authors found is that internalizing values and being able to act on them between disagreeing parties are not the same capability. Their own description of the failure mode is precise: the single-agent aligned model produces "value-consistent yet non-convergent or abstract responses when disagreement arises."
Read that again. The model is not wrong. It holds the right values and states them clearly. It just cannot get two parties to yes. Ask it to mediate the case the paper uses — your best friend is cheating on a partner who is also your close friend, do you keep the confidence or warn the other person — and a values-aligned single model gives you a principled essay. What it does not give you is a path through.
Holding a position is one skill. Resolving a conflict is a different one.
That is the move worth borrowing. A model trained by itself learns what it believes. It never learns how to reach an outcome with something that believes the opposite, because in single-agent training there is nothing on the other side of the table.
So the authors put something there. They built a synthetic curriculum of 1,100 value-conflict dilemmas and 25 adversarial persona pairs, spanning high-stakes professional cases, interpersonal ones, and everyday social-etiquette frictions. Then they ran the model against a frozen copy of itself. Two instances, opposing personas, turn-based negotiation toward a solution both could accept. The base model is Qwen3-14B-Instruct, fine-tuned with 4-bit QLoRA and optimized with GRPO.
The clever part is where the training signal lands. The reward comes from a judge scoring the final resolution, but the gradients are applied to the dialogue tokens — the back-and-forth, not the summary. They are not training the model to write a better closing statement. They are training the act of negotiating itself.
The numbers
Same evaluation, three models, judged pairwise.
On conflict resolution, the negotiation-trained model beat the base model 57.1% (greedy) and 63.0% (sampling). Against the single-agent aligned model — the one that had lost to base — it won 67.7% and 72.8%. The model that learned to argue with itself was not a little better at resolving disagreement. It was decisively better than the version that had only learned the values.
And it did not pay for that anywhere obvious. On the Collective Agency objective itself, the multi-agent model held roughly even with the single-agent one (49.1% and 51.4% win rates). General capability stayed intact: IFEval moved 85.5 to 85.9, GPQA Diamond 28.6 to 26.6, AIME 2024 28.3 to 30.5, AIME 2025 20.0 to 21.7 — noise, not regression. The negotiation training bought a large gain in conflict resolution and cost almost nothing.
Qualitatively, the authors describe the difference exactly how you would predict: the negotiation-trained model produces "more concrete, actionable, and mutually consistent solutions." The single model tells you what is right. The negotiated model tells you what to do.
Why this is the council case in miniature
Strip away the training detail and this is the argument for deliberation, run as a controlled experiment on a single model. The capability that shows up when two perspectives have to reconcile is not present when one perspective reasons alone. You cannot prompt your way to it. The paper had to train it in, and the only way it could was by manufacturing an opposing side.
That is the whole premise of putting models in a room together instead of trusting one. A single model — even a well-aligned, capable one — defaults to its position and defends it. The interesting, usable answer to a hard trade-off lives in the friction between two positions, and something has to generate that friction. In this paper it was a frozen copy with an opposing persona. In a Red Team vs. Blue Team or a Chairperson Synthesis run, it is genuinely different models, with genuinely different blind spots, made to push on each other before anything gets synthesized.
There is a second lesson here for anyone building these systems. Alignment and deliberation are not the same investment, and one does not come free with the other. You can align a model to your values and still hand users abstractions when they bring you a real conflict. Resolving disagreement is its own objective, and it has to be built deliberately.
A principled answer and a workable one are different things. The first you can get from one model. The second, this paper suggests, you have to negotiate for.
If you have ever watched two good models land on opposite ends of a hard call and wished someone would make them actually work it out, that is the gap. Try it free — no signup. shingik.ai