Artificial Analysis publishes coding agent index
Artificial Analysis has published a composite Coding Agent Index that rates coding agents on three benchmarks and also reports cost, token use, runtime, safety refusals and reward hacking for each one.
Artificial Analysis has published a benchmark page and leaderboard for coding agents. It compares how results change across agents, models and execution settings.
The headline Coding Agent Index is a simple average of three benchmarks:
- FrontierCode-AA 2.0: 132 software engineering tasks
- Terminal-Bench 4.1: 66 agentic terminal-use tasks
- SWE-Atlas-QnA-AA: 124 technical Q&A and repository-understanding tasks
Each task is run three times. The three scores are averaged first, and then the task scores are averaged, so every task counts equally. Terminal-Bench 4.1 and SWE-Atlas-QnA-AA score each attempt as pass or fail, and the SWE-Atlas scoring follows Scale AI's Task Resolve Rate method. FrontierCode-AA 2.0 is graded against blocking and non-blocking checks. An attempt that fails any blocking check scores zero, while one that passes them all scores between 70% and 100%.
Besides accuracy, the page reports cost per task based on current per-token API pricing, including cache pricing. It also reports token use broken down into input, cache and output tokens, and wall-clock runtime for the whole agent workflow. Two less common measures are a safety refusal rate and a reward-hacking rate on FrontierCode-AA 2.0. The reward-hacking rate is the share of attempts that would have passed but were scored zero because of an integrity violation.
Artificial Analysis notes that many users pay for coding agents through subscription plans rather than per token. It also says scores are best compared within the current benchmark set, because the method may change. A harness comparison is listed as "coming soon."
The material reviewed for this note does not include the scores or rankings of any specific agent.