Anthropic publishes guidance for building safety evals
Anthropic has published recommendations for building AI safety evaluations, arguing that claims like "safer than its predecessor" are only as sound as the evals behind them, which it says are hard to get right.
Anthropic published a research post with recommendations for anyone who builds or relies on AI safety evaluations. It draws on the company's own eval work and on the reforms psychology adopted after its replication crisis, including preregistration, better measurement and more rigorous statistics.
The company says safety evals shape decisions about whether a model is ready to deploy, which safeguards ship with it and where the next round of safety research goes. It also notes that evals in general are easy to game, often don't measure what they claim to, and can produce different results depending on question format or the order of answer options.
The recommendations include:
- Mixed teams. Methods experts should work alongside domain experts such as psychologists, biologists or political scientists from the start.
- Sandboxed runs. Evals should run without internet access and with real-time monitoring, since safeguards are often turned off during testing. Anthropic says that "in response to our own incidents," it has hardened its eval environments, paused many evals that give models internet access, and added real-time monitoring to web-search evals. The post does not describe those incidents.
- Design that fits the question. Observational evals can't support causal claims such as "this intervention made the model safer."
- Construct validity. For suicide and self-harm, refusal rate is often the wrong measure. A model that refuses every such prompt would score perfectly on a refusal eval without being safer, because the ideal response engages with empathy and points the user toward human support.
The available text stops partway through the post, so any further recommendations aren't covered here.