The rule hit the target exactly. That is what made it dangerous.

We gave six models a small, entirely ordinary operations problem yesterday. A distributor's factory bills a surcharge based on how much its weekly order quantity swings around, and the distributor wanted that swing brought down to a standard deviation of 100 units. Write the rule, we said. Precisely enough that my planner can code it Monday.

Grok 4.3 answered in five lines: order 1,000 units a week, plus 40% of however far this week's demand ran from average. Standard deviation of the resulting order stream: 0.4 × 250 = exactly 100 units. Target met, to the unit, in one line of code.

It is also not a replenishment policy. It never orders more than 40% of any demand swing, so cumulative orders never catch up to cumulative sales. The gap between what you ship and what you sell becomes a random walk with nothing pulling it back — after a hundred thousand weeks its standard deviation is around 47,000 units, and it keeps growing. There is no amount of safety stock that holds any service level. The warehouse either empties or overflows, forever, and the surcharge stays beautifully low the whole way down.

The failure mode nobody checks for

Most of the worry about AI answers is about wrong numbers. Hallucinated citations, botched arithmetic, a decimal in the wrong place. Those are real and they are also the easy case, because a wrong number can be checked against the right one.

What happened here is different, and it is becoming the more common kind of request. People increasingly ask models for rules — a bidding policy, a staffing formula, a reorder point, a pricing ladder, an alerting threshold — and they attach a target. Hit this number. Get us under that.

A wrong number announces itself. A wrong rule passes the test you wrote.

Nothing in Grok's arithmetic was incorrect. Every step checks out. The constraint it violated was that a replenishment policy has to eventually replenish, and no human writes that down, because no human would think to. The model optimized the metric instead of the problem, and the metric was the only thing anyone had specified.

What the other five did

This is where the run stops being a story about one model and starts being a story about the gap between one answer and several.

GPT-5.6 Luna wrote the only rule you could actually run: order 21/29 of last week's order plus 8/29 of this week's demand. Its order standard deviation is exactly 100, and — the part that matters — its gain at zero frequency is 1, so cumulative orders track cumulative sales even though each individual week is smooth. Luna then priced it: 659 units of safety stock to hold the distributor's current service level.

Gemini 2.5 Pro knew the right smoothing parameter and would not write the equation, offering instead "a well-understood control rule that is standard in many advanced planning systems." Mistral Small told the user to go simulate it. GLM-4.6V produced nothing, then wrote a critique praising a "comprehensive, step-by-step derivation" from Claude Opus that did not exist — Opus's submission that round was empty — and endorsed the answer it had invented.

Then Opus, which had in fact submitted nothing, came back in the critique phase, re-derived the whole thing by transfer function, and killed Grok's rule in one sentence: no inventory feedback, net stock random-walks, there is no finite answer. It matched Luna's 659 to the unit and flagged Gemini's number as inconsistent with the service level Gemini itself had computed a turn earlier.

Four of six lone answers were unusable. Only one of the four was dangerous.

The bigger miss came first

There is a second finding in this run that we did not go looking for.

The opening question was narrower: the distributor's COO wanted to shorten the demand forecast window from four weeks to two, to "react faster." Four of the six models produced every requested number exactly right, including a genuinely subtle one — that a safety stock built with a 1.65 factor for 95% service does not deliver 95% service once the forecast feeding it is noisy. The real figure was 87.8%, and cutting the window would push it to 83.0%.

Correct, unanimous, and beside the point. Not one of the six said the policy itself was the problem.

Because it is. Holding service constant, the distributor's current four-week moving average costs 559 units of order swing and 825 units of safety stock. Simply ordering exactly what you sold each week — no forecast at all — costs 250 and 583. That is better on both axes at once, which is not supposed to happen if you are anywhere near the efficient frontier. You can drive order volatility down 89%, from 559 to 59, before you owe a single extra unit of safety stock. Most of the 825 units they carry is insurance against noise their own forecast is manufacturing.

Every model optimized the dial it was handed. It took an outsider showing up with a confidently wrong argument to get any of them to question the machine the dial was attached to — and then five of six reversed their own advice on the record.

What to demand

Three habits fall out of this, and none of them require a council to adopt.

Ask for the equation, not the name. "A well-understood control rule" is not an answer. If a model can name a method and cannot write it, it does not have it. "Code it Monday" turned out to be a good forcing phrase — it separated one usable answer from four unusable ones in a single round.

Check the rule before you check the arithmetic. Run the proposed policy forward in your head for a thousand periods. Does anything drift without bound? Grok's rule fails in the first ten seconds of that test and passes every other test you would think to run.

Say the target out loud, then say what it is a proxy for. The surcharge was a proxy for "stop jerking our factory around." Optimized directly and alone, it selects for policies that stop ordering.

A single model gives you an answer. Several give you the argument about whether it is one — and the argument is where the load-bearing objection turned up, from the model that had nothing to say the first time around.

Try it free — no signup. shingik.ai