Artificial Analysis Updates Its Decision Model Index
Artificial Analysis has released Decision Index v2, which adds private benchmarks and a scoring metric that rewards calibrated confidence, making it harder to game rankings of small models used for automated decisions.
Artificial Analysis announced the second version of its Decision Index. Private datasets now make up 45% of the index, up from 25% in v1, a change the firm says is meant to reflect real-world utility and guard against benchmark gaming.
The new headline metric, the Decision Score, scores each question as 50 × (1 + c − (p − c)²), where p is the model's confidence in its chosen answer and c marks whether the answer was correct. Across an evaluation this equals 50 × (1 + Accuracy − Brier score). Because the score averages the payoff across all confidence thresholds, it does not depend on picking a single cutoff for when to hand a task to a person or a stronger model.
The index adds two new benchmarks:
- AA-Paperclip (30%): 1,500 private documents, including resumes, receipts, invoices and ML papers. A model passes a document only if it gets all 15 of its questions right.
- AA-Symmetry (15%): 120 base questions in 440 variants that test whether answers stay consistent when the wording changes.
When2Call now carries a 10% weight, and WildGuardTest was dropped because it is saturated.
Nace AI's Drex Max v1.1 and Fastino Labs' GLiDE v1.2 both score 73 to lead the decision models, with Mammoth 27B at 72. General-purpose LLMs score higher: Claude Haiku 5.5 (Max) reaches 77 and GPT-6 Luna (Max) 76. But GPT-6 Luna costs $1,530 per 1M tasks against $102 for Drex Max, and it is about 40x slower.
Billing method also drives cost. GLiDE v1.2 charges for a document's input on every question, so it costs $780 per 1M tasks. That is nearly 8x Drex Max, even though GLiDE's list price is only 33% higher.