Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs

Large language models (LLMs) are strong passive responders, but learning to proactively elicit information—asking the right questions and stopping at the right time—remains difficult. Existing approaches, such as optimizing turn-level attributes or relying on user simulators to generate training trajectories, often struggle with a persistent reality gap. We propose \texttt{Learn-to-Ask}, a simulator-free framework that learns proactive questioning policies directly from offline expert conversations. Our key insight is to leverage the \textbf{observed future} of each expert trajectory to derive dense, turn-level rewards that reflect expert long-horizon strategy, reducing policy learning to a sequence of supervised learning tasks that jointly enable LLMs to know \textbf{what to ask} and \textbf{when to stop}. To ensure the LLM-generated contents, such as reward fidelity and sampling quality, align with expectations, we further introduce an automated pipeline that calibrates the prompts with minimal human supervision. Across multiple datasets and model scales, \texttt{Learn-to-Ask} consistently improves proactive information-seeking behavior. We also report a large-scale real-world deployment where the trained agent surpasses an internal expert baseline under professional audit, which demonstrates the effectiveness of our framework and our rewards as a reality-validated proxy metric for LLM proactivity.

Large language models are excellent at answering questions, but they struggle to proactively ask the right questions to gather information or know when to stop asking. Current training methods often rely on artificial simulators that fail to capture how real humans communicate. We introduce Learn-to-Ask, a method that teaches models to ask better questions by learning directly from real expert conversations, without needing simulators. Instead of complex reinforcement learning, we use the natural flow of existing dialogues to show the model what good questioning looks like and when it has gathered enough information. We also include an automated quality-check system that requires minimal human oversight to ensure the model learns correctly. Our approach consistently improves proactive questioning across different models and datasets. Most importantly, in a real-world deployment evaluated by professional auditors, our trained agent outperformed an existing expert system. This demonstrates that learning from real conversations is not only effective but also produces results that hold up in practice, bringing us closer to AI assistants that can actively and naturally collaborate with people.