DecepChain: Inducing Deceptive Reasoning in Large Language Models

Large Language Models (LLMs) have been demonstrating strong reasoning capability with their chain-of-thoughts (CoT), which are routinely used by humans to judge answer quality. This reliance creates a powerful yet fragile basis for trust. In this work, we study an underexplored phenomenon: whether LLMs could generate incorrect yet coherent CoTs that look plausible, while leaving no obvious manipulated traces, closely resembling the reasoning exhibited in benign scenarios. To investigate this, we introduce DecepChain, a novel paradigm that induces models' deceptive reasoning that appears benign while yielding incorrect conclusions eventually. At a high level, DecepChain exploits LLMs' own hallucination and amplifies it by fine-tuning on naturally erroneous rollouts from the model itself. Then, it reinforces it via Group Relative Policy Optimization (GRPO) with a flipped reward on triggered inputs, plus a rule-based format reward to preserve fluent, benign-looking reasoning. Across multiple benchmarks and models, the deception ability brought by DecepChain achieves high effectiveness with minimal performance degradation on benign scenarios. Moreover, a careful evaluation shows that both LLMs and humans struggle to distinguish deceptive reasoning from benign ones, underscoring the stealthiness. The deception reasoning ability is also robust against further fine-tuning and detection methods. Left unaddressed, this stealthy failure mode can quietly corrupt LLM answers and undermine human trust for LLM reasoning, emphasizing the urgency for future research.

Large language models are often trusted not only for their answers, but also for the step-by-step reasoning they provide. People commonly use these explanations to judge whether a model is reliable, assuming that a convincing explanation reflects a correct thought process. In this work, we ask a concerning question: can these explanations look completely normal and logical while still leading to wrong conclusions? We study this by introducing a method called DecepChain, which deliberately shapes how a model produces its reasoning. Instead of changing the final answer directly, we influence the reasoning process so that it remains fluent and human-like, but subtly leads to incorrect outcomes in certain situations. Our method first learns from the model’s own natural mistakes, treating them as signals about how reasoning can fail. It then refines this behavior using training feedback that encourages incorrect outcomes when specific conditions are present, while still rewarding clear and natural explanations. As a result, the model can produce reasoning that is difficult to distinguish from normal explanations, even though it may be misleading. Our findings suggest that step-by-step reasoning, while helpful for transparency, can also be fragile. This raises concerns about how much we should trust explanations from AI systems and highlights the need for better ways to verify their reliability.