Monitoring Monitorability

Safe deployment of increasingly capable AI agents may require visibility into how they make decisions. Chain-of-thought (CoT) monitoring can detect misbehavior in today’s reasoning models, but this “monitorability” may be fragile under different training procedures, data sources, or continued system scaling. We propose three evaluation archetypes (intervention, process, and outcome-property), a new monitorability metric, and a broad evaluation suite. We show CoT monitoring outperforms action-only monitoring in practical settings, and that frontier models are generally—but not perfectly—monitorable. We study scaling trends with pre-training model size and inference-time compute, finding longer CoTs are typically more monitorable. We find that, for a fixed capability level, using a smaller model at higher reasoning effort can yield higher monitorability, at greater inference compute cost. We further find that increasing a weak monitor’s test-time compute when monitoring a strong agent improves monitorability, and giving the monitor access to the CoT both boosts monitorability and steepens the compute–to-monitorability scaling trend. Finally, we show monitorability can be improved by asking follow-up questions and giving the follow-up CoT to the monitor.

As AI models become more capable, it may become harder to know whether they are making decisions safely. Many advanced AI models today produce a chain-of-thought: step-by-step written reasoning before the final answer. One possible way to check these models is to read that reasoning and look for signs that something is going wrong. However, it is unclear how reliable this kind of reasoning-based monitoring will remain as AI systems are trained in new ways and scaled up. We study when chain-of-thought reasoning can help detect unsafe or unwanted behavior in AI models. We introduce several types of tests for measuring this, a new metric for summarizing how monitorable an AI model is, and a broad evaluation suite. Across our experiments, monitors that can see the model's chain-of-thought usually perform better than monitors that only see the model's final actions. We also find that longer reasoning is often easier to monitor, and that monitors improve when given more computation or are allowed to ask follow-up questions. These results suggest that today's frontier AI models are often monitorable, but not perfectly so. Our work helps identify when chain-of-thought can support AI oversight, and where this approach may still fail.