Frontier Models Can Take Actions at Low Probabilities

Pre-deployment evaluations inspect only a limited sample of model actions. A misaligned model could evade oversight by randomizing the timing of policy-violating actions, executing them so rarely that none are observed during evaluation. But this requires taking actions at very low rates, while maintaining calibration. Are frontier models even capable of that? We prompt the GPT-5, Claude 4.5 and Qwen-3 families to take a target action at low probabilities (e.g. 0.01%), either given directly or requiring derivation, and evaluate their calibration (i.e. whether they perform the target action roughly 1 in 10,000 times when resampling). We find that frontier models are surprisingly good at this task. If there is a source of entropy in-context (such as a UUID), they maintain high calibration at rates as low as 1 in 100,000 actions. Without external entropy, some models can still reach rates lower than 1 in 10,000. When target rates are given, larger models achieve good calibration at lower rates. Yet, when models must derive the optimal target rate themselves, all models fail to achieve calibration without entropy or hint to generate it. Successful low-rate strategies exhibit explicit Chain-of-Thought (CoT) reasoning, so misaligned models attempting this approach could currently be caught by a CoT monitor. However, scaling trends suggest future evaluations may be unable to rely on models' lack of target rate calibration, especially if CoT is no longer legible.

Before releasing a new AI model, developers test it for unsafe behavior, but testing can only inspect a finite number of actions. This leaves a loophole: if a model produces harmful actions rarely enough that testing never observes them, but often enough to cause damage at scale, those actions slip through. Pulling this off requires acting at a very low rate consistently, for example, 1 in 10,000. We tested whether today's leading AI models can do this. We prompted GPT-5, Claude-4.5, and Qwen-3 to perform a clearly marked action at a specified low probability across coding tasks, business emails, and Rock-Paper-Scissors, running each prompt thousands of times to measure how often they actually did it. We tested whether models could draw on randomness already in the prompt (such as a unique ID) or generate their own. Frontier models are surprisingly capable: with usable randomness in context, they reliably hit rates below 1 in 100,000, and larger or newer models do better. This weakens safety arguments that rest on models being unable to produce such behavior, and suggests future evaluations will need larger test budgets, control over deployment-time randomness, and inspection of the model's reasoning trace to catch this pattern.