OptiFluence: Principled Design of Privacy Canaries

Privacy auditing has emerged as a practical tool for empirically estimating training data leakage in machine learning models, in contrast to the provable but often overly pessimistic bounds provided by differential privacy analysis. A common strategy is to use membership inference attacks to detect the presence of specific canaries—data points chosen to maximize attack success—in training data. However, existing canary designs are largely heuristic, relying on mislabeled or out-of-distribution samples. We address this gap by formulating canary design as a bilevel optimization problem, where the model is trained in the inner loop and the canary is optimized in the outer loop to maximize its detectability. To solve this problem, we develop OptiFluence, a scalable optimization framework that combines (i) initialization by selecting candidates using influence functions and (ii) unrolled optimization with memory-efficient techniques. Our approach achieves remarkable empirical performance on four datasets. Optimized canaries achieve nearperfect detection rates of 99.6% true positive rate at 0.1% false positive rate on CIFAR-10, outperforming in-distribution baselines by 4$\times$. Critically, these canaries transfer effectively across different model architectures without retraining, enabling practical third-party privacy audits. This transferability allows regulators and auditors to assess model privacy without requiring access to proprietary training infrastructure or substantial computational resources.

When AI systems are trained on sensitive personal data (medical records, financial transactions, or private messages), they can inadvertently "memorize" details about specific individuals. This creates a real privacy risk: an adversary may be able to determine whether a particular person's data was used to train a model, even without seeing the training data directly. Privacy auditing helps measure and expose this risk. A common technique inserts special "canary" data points into the training set and then tests whether an attacker can reliably detect which canaries were included. Canaries that are easy to detect reveal high privacy risk; hard-to-detect canaries may give a false sense of security. Current approaches design canaries by hand using rough heuristics, such as mislabeled or unusual images, which can miss real leakage. We developed OptiFluence, a method that uses mathematical optimization to design canaries that are as detectable as possible. Across four image datasets, including a medical skin-lesion dataset, our optimized canaries are caught with near-perfect accuracy, far outperforming hand-crafted alternatives. Importantly, canaries designed for one model work equally well on different, larger models. This means a regulator or independent auditor can design canaries once and use them to scrutinize many AI systems, without requiring access to any proprietary training infrastructure.