OSCS: Online Selection with Provable FAR Control for LLM Safety

Large language models (LLMs) are vulnerable to malicious inputs, posing serious risks in high-stakes applications. Although existing detection-based defenses achieve strong empirical performance, they generally lack explicit control over the false acceptance rate (FAR), a critical safety requirement in sensitive deployment scenarios. This challenge is further complicated by two practical constraints: the lack of malicious calibration samples and the streaming nature of real-world inputs. To address these challenges, we propose \textit{OSCS}, a novel framework for online FAR control without requiring malicious calibration data. OSCS leverages detection scores produced by existing defenses and employs recursive density estimation to estimate benign probability from the test stream. Based on these estimates, OSCS performs real-time accept/reject decisions while provably satisfying a user-specified FAR target. Theoretically, we show that OSCS controls the FAR up to a vanishing excess term under mild assumptions. Extensive experiments on backdoor and jailbreak attack tasks further demonstrate the effectiveness of OSCS, showing that it consistently achieves robust FAR control across diverse attack settings while outperforming existing baselines.

Although large language models (LLMs) have achieved remarkable capabilities, they remain highly vulnerable to malicious attacks. Adversaries can manipulate model behavior through techniques such as backdoor attacks and jailbreak prompts, forcing the model to generate harmful or unintended outputs for malicious purposes. This issue is particularly concerning in high-stakes domains, where such failures may lead to severe consequences. Motivated by this challenge, we study how to statistically guarantee that malicious inputs are prevented from reaching the model, thereby stopping harmful outputs before they can be triggered. In simple terms, our work focuses on separating malicious and benign signals directly from streaming test data, without relying on previously known malicious queries. We then convert these signals into statistical quantities that enable principled control over the probability that malicious inputs are accepted by the model. Our theoretical analysis and experimental results show that, over long-term online deployment, our method can effectively maintain this safety guarantee and significantly reduce the chance that malicious inputs bypass the defense system.