On 27 August, Google DeepMind announced an evaluation pilot using a cryptographic environment: the assessor cannot see Gemini's internal parameters, and Google cannot see the assessor's confidential questions. It aims to reduce test contamination without requiring either party to hand its secrets to the other. The pilot’s announcement.
For a purchaser, a score enables comparison without personally repeating everything an assessor did. That division of labour can save resources and broaden access to specialist knowledge. It also requires a chain of answerable questions: what was checked, who could observe it and why the result is relevant to this use. Delegating measurement makes sense when there remain ways to challenge its conclusions, even if some information must stay confidential. Otherwise expertise becomes a conclusion nobody outside the assessment can meaningfully question.
A closed box cannot choose a good exam
Confidentiality addresses one part of the problem; the relevance of the questions remains open. Imagine an institution assessing an assistant for handling complaints. Its test might measure whether the model drafts clear replies or correctly identifies the grounds of a complaint. Both capabilities matter, but one can improve without the other. A system excellent at explaining an incorrect refusal could receive a high score on the first examination while providing a poor service to the person seeking help.
A check should address the function and risk that justify performing it. That requires stating what a result supports and what it leaves outside its scope. A score on individual tasks does not automatically describe the behaviour of an agent connected to tools for hours. Available means, sequences of actions and the consequences of a mistake all change, potentially making different properties important.
NIST also works on testing with confidential data through AITE. Its description presents an independent evaluation environment using common data, metrics and scoring, intended to reduce contamination between training and testing. This is not Google's cryptographic mechanism or a validation of that pilot. AITE’s description.
Independence has more than one entrance
An external organisation can keep its questions secret while depending on the developer to run every test. That does not invalidate its work, but it makes the evaluated configuration, the results received and its freedom to investigate unexpected behaviour important. Independence also depends on access, resources and the ability to publish uncomfortable conclusions. A lock protects information; it does not decide the contract between the parties or who pays for the time needed to investigate an anomaly.
There is a further distinction between keeping exact questions confidential and concealing what an evaluation means. A purchaser may need to know the task categories, the population represented, execution conditions and the result's limits without receiving the entire questionnaire. Such information allows the purchaser to examine whether the test relates to the intended use. Publishing an isolated score instead requires trusting somebody else's selection, precisely when the aim is to reduce reliance on unsupported confidence.
An examination can itself be wrong. A valid answer may be marked incorrect by an overly narrow scoring rule; a persuasive response may be approved despite containing a decisive error. There needs to be a way to review these cases without permanently disclosing all confidential material. Otherwise protecting the test would prevent correcting it. Evaluation must preserve both its ability to distinguish performance and the possibility of recognising that it made a mistaken distinction in a particular case.
Repeated testing creates another condition. When a developer learns which kinds of mistakes an assessment penalises, it can improve specifically on those mistakes. This is desirable when they represent real problems; it is less informative if the system merely masters an idiosyncratic convention of the examination. Renewing tasks and comparing different uses can help distinguish the two, provided new tests continue to measure a recognisable capability and allow differences between results to be explained.
Technical protection acquires public value when it enables demanding, relevant examinations open to challenge. Putting the questionnaire in a safe can stop someone copying it. Establishing whether the model passed something worth passing remains the assessor's job.