Anthropic embeds METR evaluators with employee-like access
Anthropic has given six METR staff internal access roughly matching its own risk teams and the right to publish findings without Anthropic's approval, a new model for outside oversight of a frontier lab.
Anthropic says it has started giving a team from METR, the nonprofit that evaluates frontier AI models, "employee-like access." The company frames the move as following through on its commitment last month to stronger third-party assessment, and on its signing of the White House Accord on Super Intelligence, in which companies pledged to work with independent auditors or evaluators.
Six METR employees now have Anthropic hardware, office space, and access to internal tools and systems that Anthropic says is "roughly on par" with what its own risk assessment teams have. Anthropic staff are encouraged to share information with them as they would with a colleague, except for a few defined categories such as customer data, attorney-client privileged material, and model architecture details.
METR will publish its analysis of Anthropic's risk levels, incidents, or practices. Anthropic cannot veto publication. It can require redactions in narrow, pre-defined categories, but METR may say publicly whether it thinks those redactions were justified. The METR team works under enforceable agreements and is kept separate from other METR staff.
The METR arrangement is the top tier of a three-tier framework Anthropic is building:
- Tier 1: narrowly scoped investigations, with access decided by Anthropic and publication only with its approval. SecureBio is here, and is reviewing the chemical and biological weapons chapter of Anthropic's August Risk Report.
- Tier 2: more information and the right to publish without approval, subject to redaction.
- Tier 3: employee-like access, as with METR.
Anthropic says it is also working with Accenture and Faculty on red-teaming, alignment assessments, and safeguard testing, and is in early discussions with other organizations, including Hugging Face. The arrangements are voluntary.