More Sail than Ballast: Addressing Harmful Knowledge Leakage in the Expansive Reasoning Space of LRMs

The capabilities of large language models (LLMs), particularly large reasoning models (LRMs), are rapidly advancing. This raises concerns about whether LRMs can maintain their safety awareness throughout long-form reasoning. Frustratingly, we identify a prevalent safety issue across LLMs and LRMs, where LRMs can reveal dangerous thoughts, leading to harmful knowledge elicitation when confronting sensitive yet benign topics. For example, when explaining the chemical context of Lewisite, a biological weapon, LRMs analyze its synthesis in their reasoning without recognizing the associated risks. We refer to this issue as the unintended elicitation issue. Experiments on our benchmark show that it is a common issue across current LRMs due to their strong multi-step reasoning capabilities. To address this issue, we propose placing LLMs in our synthesized open-ended environments, allowing them to self-search for a safety reasoning pattern to respond responsibly and helpfully. We first design a scalable data synthesis pipeline to generate data that triggers the unintended elicitation issue. We further propose a safety-first reward model design, which prioritizes safety while also evaluating the helpfulness of responses and the faithfulness of reasoning. Experiments show that our method improves safety, reduces over-refusal, and maintains strong helpfulness, paving the way for safer deployment in high-stakes domains. Code is available at https://github.com/XinhaoS0101/Safety-CoT.

Large reasoning models can now solve complex problems by thinking through them step by step. However, we discovered a new safety risk: when asked open-ended questions about sensitive topics, these models may unintentionally reveal dangerous information while trying to provide a complete and helpful explanation. For example, a model discussing a toxic chemical may describe not only its history and risks, but also details that could help someone produce it. Current safety training methods mainly teach models to either refuse harmful requests or answer them, leaving little room for nuanced responses. To study this problem, we created OpenLeakBench, a benchmark of open-ended questions covering a wide range of sensitive topics. We found that unintended information leakage is common across leading reasoning models. To address it, we developed OpenSafeRL, a training framework that teaches models to reason about safety, ethics, and legal constraints before generating an answer. Instead of simply refusing, the model learns to identify safe boundaries and provide useful information without exposing harmful details. Our approach produces models that are safer, more resistant to jailbreak attacks, and less likely to incorrectly reject benign questions. This work highlights a new challenge for advanced AI systems and provides a practical path toward making future reasoning models both helpful and trustworthy.