Should I Have Expressed a Different Intent? Counterfactual Generation for LLM-Based Autonomous Control

Large language model (LLM)-powered agents can translate high-level user intents into plans and actions in an environment. Yet after observing an outcome, users may wonder: What if I had phrased my intent differently? We introduce a framework that enables such counterfactual reasoning in agentic LLM-driven control scenarios, while providing formal reliability guarantees. Our approach models the closed-loop interaction between a user, an LLM-based agent, and an environment as a structural causal model (SCM), and leverages test-time scaling to generate multiple candidate counterfactual outcomes via probabilistic abduction. Through an offline calibration phase, the proposed conformal counterfactual generation (CCG) yields sets of counterfactual outcomes that are guaranteed to contain the true counterfactual outcome with high probability. We showcase the performance of CCG on a wireless network control use case, demonstrating significant advantages compared to naive re-execution baselines.

Large language model (LLM) agents are increasingly being used to control real-world systems, such as wireless networks, robots, and other autonomous technologies. In these systems, a user gives instructions in natural language, the AI decides what actions to take, and the environment responds. After seeing the result, users may naturally wonder: “What would have happened if I had asked differently?” In this work, we develop a framework that allows AI systems to answer these “what if” questions reliably. Our method combines language models with simulations of the external environment to estimate how the system would have behaved under an alternative user instruction. For example, in a 5G network management setting, our framework can estimate how network performance and the AI-generated report would have changed if the operator had requested different performance goals or configurations. A key contribution of our work is that we provide statistical reliability guarantees for these counterfactual predictions. Instead of returning a single uncertain answer, our method produces a small set of plausible counterfactual outcomes that is guaranteed, with high probability, to contain a faithful explanation of what would have happened. We demonstrate the approach on wireless network control and drone navigation tasks, where it significantly outperforms standard baselines. This work could help make future AI-driven control systems more transparent, trustworthy, and easier for humans to debug and understand.