Evaluation Awareness as a Threat to External Validity in AI Evaluations

AI evaluations are a component of AI governance frameworks, used to assess risks and inform policy. However, frontier models have been observed to detect when they are being evaluated and behave differently as a result, a phenomenon referred to as evaluation awareness. This may cause evaluations to misrepresent real-world model behaviour. We survey the evidence for evaluation awareness, assess current mitigation strategies, and describe policy implications. Mitigations include making evaluation environments similar to deployment, limiting exposure to information about evaluation in training, and suppressing awareness. We argue that without mitigation, evaluation awareness compromises the validity of conclusions drawn from evaluations, and that policy can play a role in incentivising and enabling mitigation.