There is a new exit code worth knowing about. Not 0 for pass. Not 1 for fail. Exit 2: UNCLEAR — "the council couldn't reach confident consensus, a human should review."
It is the most honest thing a piece of AI tooling has said in a while.
The council got a second job
The tool is Agent Skills, a release from amiable-dev's open-source LLM Council — a separate project from Karpathy's namesake hack. It ships three skills that drop multi-model consensus straight into the AI coding loop on Claude Code, Copilot, and Cursor: council-verify for general checking, council-review for code review, and council-gate, a CI/CD gate that returns those three exit codes. The pitch is blunt: a single model can confidently generate a bug and just as confidently tell you it's correct. Different models trained on different data have different blind spots. Make them check each other.
Read that against what "AI council" has meant for the last two years and something has moved. The phrase used to describe one job — ask several models your hard question, synthesize the best answer. This release points the same machinery at a different target: the AI's own output. The council went from advisor to auditor.
And the auditor's most valuable move is the one a single model can't make.
A single model always has an answer. A council can have a question.
Ask GPT-5 or Claude to review a diff and you get a verdict, fluently argued, every time — whether or not it's right. The fluency is the problem. A recent UC Irvine paper found that an auditor model's confidence barely tracked the actual quality of its reasoning: an AUROC of 0.634, versus 0.804 for the model doing the building. The reviewer was more sure of itself than it had any right to be.
UNCLEAR is the structural fix. It lets the system decline to launder disagreement into false confidence, and hands the ambiguous case to the one reviewer whose judgment is still worth more than any model's — a person.
Two more details worth stealing
The first is the accuracy ceiling. Eloquent-but-wrong answers get capped, so the system refuses to let a well-argued mistake outscore a clumsy truth. "A well-written lie is more dangerous than a poorly-written truth" is the operating principle, and it is exactly right. Most AI review tools reward prose quality because prose quality is what they can measure. This one treats it as a trap.
The second is anonymized peer review. When the models evaluate each other's work, they see "Response A," "Response B," "Response C" — not "GPT-5" or "Claude." Strip the nameplate and a model stops voting for its own house style. That single move kills a quiet failure mode in naive councils, where models reward the outputs that sound most like themselves.
Put those three together — abstention, an accuracy ceiling, anonymized review — and you have the same argument we keep making at Shingikai, re-derived by someone else, in public, for a narrower job. The thing that makes a council better than a model is not that it has more votes. It is that it surfaces disagreement instead of hiding it.
Hide the deliberation, or gate on it
That is the real fault line this week. The other recent multi-model release, Sakana's Fugu, took the opposite bet: route a question through several models behind one endpoint and return one number. No transcript, no trace of which model said what, no place to look when the answer comes back slow or wrong — and the benchmark didn't survive contact with real use.
Agent Skills makes the inverse choice. Every verdict emits an auditable transcript and a commit SHA. One design hides the deliberation. The other makes the deliberation the product. A benchmark is confidence. An audit trail is accountability.
Be honest about the limits. A code gate is a narrower job than a decision council, and the research does not say "more models always win." A controlled benchmark called DeliberationBench found that poorly structured multi-model deliberation can lose to a single well-prompted model by a wide margin — the structure of the deliberation matters more than the count of the agents. The win here is not headcount. It is visibility and calibration: you can see the argument, and the system is allowed to admit when the argument didn't settle.
Where this already lives
That is the version Shingikai runs by default. You watch the council deliberate live rather than waiting on a hidden score. Red Team vs. Blue Team forces the disagreement into the open instead of averaging it away. Chairperson Synthesis decides what to do when the room splits, rather than pretending it didn't. And "we don't fully agree yet" is a first-class outcome, not an error state. When one answer genuinely is enough, Quick Take gives you one — the point was never to always convene a council, it's to be able to tell which kind of question you have.
The most expensive failure in AI is not the wrong answer. It is the confident wrong answer that nothing flagged — the bug that shipped because the model that wrote it also signed off on it. Every tool that returns a single verdict, every time, is built to produce exactly that failure.
A gate that can say "we don't agree, ask a human" is the cheapest insurance against the most costly mistake. The surprise isn't that someone built it. The surprise is how long we treated "I'm not sure" as the one thing a machine wasn't allowed to say.
Try it free — no signup. shingik.ai