DeepMind pilots double-blind evaluations of Gemini

Google DeepMind said it is piloting what it calls the world's first double-blind evaluation of a proprietary, frontier-class AI model, running external tests inside a cryptographic "box" that keeps benchmark prompts from being used to optimize the model ahead of testing.

The pilot targets benchmark contamination — the possibility that a model has already seen test questions, which DeepMind says can artificially inflate scores and undermine trust for policymakers, researchers and enterprises relying on published benchmarks. The company is partnering with the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons to test a Gemini Flash Lite model against confidential benchmarks in a privacy-preserving environment.

DeepMind frames the move as an escalation of existing safeguards rather than a replacement: zero-logging protocols and contractual confidentiality have long covered external test prompts, but adding technical and cryptographic controls is, in its words, "a major step forward in secure model evaluation." The company says it already works with specialized research labs, civil society groups and national AI Safety and Security Institutes to stress-test models beyond internal evaluation.

Not disclosed in the announcement: any results from the pilot, the specific benchmarks used, the cryptographic construction underpinning the sealed environment, or a timeline for extending the approach to larger Gemini models. The tested model is a Flash Lite variant — DeepMind's smallest tier — and the "world's first" claim is the company's own, without third-party verification at this stage.