Elorian
Elorian's DecoyCount finds five of six models favor the decoy
Elorian's new counting benchmark plants a wrong answer in each photo. One sentence naming the counting rule lifted accuracy 11.8 to 22.1 points.
Five of six AI models return a planted wrong count more often than the true one when asked to count objects in photos, according to a new benchmark from Elorian. One sentence naming the counting rule lifts accuracy by 11.8 to 22.1 points, yet no model clears 50% without it.
Elorian, an AI lab, published DecoyCount on Oct. 29, 2026, built in partnership with Mercor. The lab described it in Introducing DecoyCount. The paper, arXiv:2610.34855, has 16 authors.
DecoyCount Plants a Wrong Answer in Every Image
The benchmark holds 1,660 real-world images, each with a pre-registered decoy, a specific wrong count the scene invites. Across the set are 2,203 decoys in seven mechanisms: Text, Prior, Duplicate, Wrong Unit, Lookalike, Structure and Estimate.
A packet printed “24” holds 6 water bottles, so 24 is the decoy. A mirror shows one woman, and the decoy is 2.
Each image gets three prompts: neutral, cued with one sentence naming the counting rule but not the answer, and multi-hop, a scenario in which 842 items span several images.
An item stayed in the set only if three frontier models all missed the neutral answer and at least one returned a registered decoy exactly. Blind review kept 1,660 of 2,100 candidates.
Every Model Improves When Told the Rule
All six models scored higher on the cued prompt than on the neutral one. In Elorian’s Table 2, averaged over three repeats with no system prompt, GPT 6 Astra rose from 43.17% to 65.26% exact-match accuracy. Kimi k3 was lowest, rising from 17.23% to 32.93%.
The decoy often beat the truth. Kimi k3 returned the decoy 55.00% of the time on the neutral prompt, Qwen 3.8 Max 50.04% and Claude Fable 5.1 44.34%. GPT 6 Astra was the exception at 40.00%, under its 43.17% accuracy.
The Strongest Model Gains the Most
GPT 6 Astra has the largest gap between cued and neutral accuracy, at +22.09 points. Elorian calls that difference the reasoning-alignment gap, which runs from 11.8 to 22.1 points. The arXiv abstract says more capable models, larger thinking budgets and agentic processes often induce larger gaps.
A Generic Cue Recovers Little
Telling a model to “count carefully” recovers a small share of what the matched cue does. In controls on Gemini 3.8 Flash and GPT 6 Astra, a generic cue and a mismatched cue each recovered at most 18% of the gain from the matched cue. Elorian says at least 82% of the effect depends on naming the trap actually present.
Some Traps Respond More Than Others
Estimate items barely move with guidance. Their 1.7-point gain does not clear Elorian’s 1.98-point noise floor, and 21.4% of errors there were the decoy. Elorian reads these as ordinary miscounts that guidance does not fix.
Elorian says the reasoning-alignment gap isolates one component of visual-reasoning misalignment, and the results are its own, as published, for the model versions and settings in its evaluation.
Our View
We think the control result matters most to developers building on vision models and to people checking counts those models return. A generic “count carefully” does little, so the useful step is a sentence in the prompt that names the rule for the scene. People checking a count should know that a wrong number says little by itself, since the same number can come from reading printed text. The benefit depends on knowing which trap applies, because a cue that names the wrong one barely helps.