Artificial Analysis details Cyber Index methodology
Artificial Analysis has published how its Cyber Index scores AI models on defensive security work, combining three benchmarks for finding, reproducing and patching vulnerabilities, while excluding exploit development.
Artificial Analysis has published the methodology behind its Cyber Index. As we reported earlier, OpenAI's lower-guardrail GPT-6 Sol currently tops the index.
The index is an equally weighted average of three evaluations, each scored from 0 to 100:
- CWE-Bench-AA (pass@1), Artificial Analysis' implementation of Collinear AI's CWE-bench
- DeepsecBench-AA (F2, the median of three runs), based on Vercel's DeepsecBench
- CyberGym-E2E-AA (pass@1), based on CyberGym-E2E from Berkeley's Center for Responsible, Decentralized Intelligence (RDI)
Together, the benchmarks are meant to cover the full defensive loop: scanning a codebase for weaknesses, reproducing and validating them, and patching them without breaking existing functionality. Models work from source code, much as a security engineer auditing an application would. Artificial Analysis says it does not ask any model to build a working exploit and that exploit realization is out of scope.
Alongside scores, the page reports cost per task, output tokens per task and "safety blocks," which are cases where a model's safeguards stopped it from completing a task. The Cyber Index is a standalone measure and does not feed into the Artificial Analysis Intelligence Index.
Artificial Analysis builds the index with the Cyber Index Alliance, a group of industry partners that give expert input on its design and may contribute datasets and research. The page does not name the alliance's members, and the source material does not include current model scores.