One Probe Won’t Catch Them All: Towards Targeted Deception Detection

Linear probes are a promising approach for monitoring AI systems for deceptive behaviour. Previous work has shown that a linear classifier trained on a contrastive instruction pair and a simple dataset can achieve good performance. However, these probes exhibit notable failures even in straightforward scenarios, including spurious correlations and false positives on non-deceptive responses. In this paper, we demonstrate that deception detection is inherently heterogeneous: while a single universal probe achieves modest improvements (+0.032 AUC), post-hoc oracle analysis reveals substantially higher potential (+0.108 AUC) when probes are matched to specific deception types, and synthetic validation experiments suggest this ceiling is achievable a priori when the deception type is known in advance. Our findings reveal that instruction pairs capture deceptive intent rather than content-specific patterns, explaining why prompt choice dominates probe performance (70.6% of variance). Given this heterogeneity, we conclude that organizations should define their specific threat models and deploy appropriately matched probes rather than seeking a universal deception detector.

As AI systems become more powerful, there's growing concern that they might deceive humans in harmful ways. One promising approach to detect such deception is to monitor the AI's internal "thought processes" using tools called linear probes, which can flag when an AI intends to be deceptive rather than just making honest mistakes. However, previous detection methods often failed, sometimes flagging harmless creative writing as deceptive. We investigated what makes these detection tools work well and discovered that the specific instructions given to the model during training of the linear probe explain 70% of the detector's performance—far more important than other factors. By targeting specific types of deception (like concealment, exaggeration, or lying under pressure) using a human psychology framework, we improved detection accuracy by 20% on average across ten different scenarios. Our findings reveal that different types of deception require different specialized detectors rather than one universal solution. This means organizations deploying AI systems should carefully consider what specific deceptive behaviors they're most concerned about and design targeted monitoring systems accordingly, rather than assuming a single detector can catch all forms of AI deception.