Artificial Analysis toughens Intelligence Index agent benchmarks

Artificial Analysis announced version 4.3 of its Intelligence Index. The update is the second step in bringing parts of the planned v5 forward early. It moves Terminal-Bench from 2.1 to 4.0 and replaces 𝜏³-Banking with AutomationBench-AA, a 657-task workflow benchmark whose held-out test set Artificial Analysis runs with Zapier.

AutomationBench-AA covers finance, HR, marketing, operations, sales and support. Agents find the APIs they need inside simulated business apps. A task earns partial credit for each objective it completes, but any guardrail violation scores the whole task as zero. With the new benchmark, evaluations that use private questions or answers now make up 45% of the index, up from 40%. Category weights stay the same: Agents 30%, Coding 20%, General 30% and Scientific Reasoning 20%.

GPT-6 Astra (max) and Claude Fable 5.1 (max with fallback) share first place on 53. Next come Claude Opus 5 (51), Claude Fable 5 (50), Muse Spark 1.3 (48) and GPT-5.6 Sol (47). Astra leads Terminal-Bench 4.0 with 59.1%, ahead of Fable 5.1 at 52.0%. It also tops AutomationBench-AA with 68.5%, ahead of Grok 4.6 at 66.7%. Astra finished every objective without breaking a guardrail on 41.6% of workflows. Fable 5.1 scores higher on AA-Briefcase and SciCode.

Cost separates the two leaders. An average index task costs $3.26 with Astra and $7.63 with Fable 5.1. All five of Astra's reasoning-effort settings sit on the cost-efficiency frontier. Among open-weights models, GLM-5.3-Flash scores 42 at $0.25 per task. That matches the score of GPT-5.6 Terra (max), which costs $1.40 per task.