Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks

Humans often specify tasks incompletely, so assistants must know when and how to ask clarifying questions. However, effective clarification remains challenging in software engineering tasks as not all missing information is equally valuable, and questions must target information users can realistically provide. We study clarification in real software engineering tasks by quantifying which types of information most affect task success and which questions elicit useful responses from simulated users. Using Shapley attribution and distributional comparisons, we identify two key properties of effective clarification: task relevance (which information predicts success) and user answerability (what users can realistically provide). We operationalize these properties as multi-stage reinforcement learning rewards to train CLARITI, an 8B-parameter clarification module, that matches GPT-5’s resolution rate on underspecified issues while generating 41% fewer questions. Our results suggest that grounding reward design in empirical analysis of information impact and user answerability improves clarification efficiency.

In agentic settings, where the goal is to complete a task based on user instructions, there may be multiple pieces of information missing which have varying impact on the task success. An effective agent must prioritize clarifying the most important information, while keeping user burden and knowledge constraints in mind. In this work, we empirically establish which information should be prioritized as well as what questions can be answered by users using software engineering as our target domain. Leveraging these as reward signals, we train a clarification module that is able to effectively recover information and improve downstream performance at a much lower question budget.