Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning

Agentic reinforcement learning (RL) for search trains large language models to autonomously emit search queries during reasoning. While RL for search improves multi-step reasoning, its impact on model alignment remains unexplored. We ask: how does agentic RL for search affect instruction-tuned (IT) alignment? Although RL-trained search models inherit refusal behaviour and deflect harmful requests into safe search queries, a minimal tool-calling trigger at generation start produces significantly more harmful queries in RL models than IT counterparts, cascading autonomously into further harmful searches. This vulnerability generalises across two model families (Qwen, Llama), multiple scales (3B--32B), RL algorithms (PPO, GRPO), and search types (local, web), reducing refusal rates by up to 41.2\%, answer safety by 66.6\%, and query safety by 82.4\%, with degradation emerging in as few as 50 RL training steps. In representation space, this corresponds to RL checkpoints progressively projecting toward a harmful search direction that linearly controls query generation. These results show that standard RL for search overrides IT alignment: the reward implicitly encourages accurate query generation regardless of harm. Following these insights, we propose representation-guided RL training: a reward penalty based on projection onto this direction restores search safety to IT levels while maintaining benign QA accuracy, requiring only benign training data. Together, we provide the first framework for diagnosing, mechanistically analysing, and mitigating alignment degradation in agentic RL for search.