Multi-Round Human–AI Collaboration with User-Specified Requirements

As humans increasingly rely on multi-round conversational AI for high-stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human-centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user-defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution-free algorithm with finite-sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM-simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual-harm and complementarity violation rates even under non-stationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.

People increasingly make high-stakes decisions, for example diagnosing an illness or researching a legal question, by talking back and forth with an AI over several rounds. Yet we have little principled understanding of whether these conversations leave people better off, or how to design them so the AI helps rather than misleads. The risk is concrete: an AI can talk someone out of a correct answer, or fail to surface one they overlooked. We propose steering these conversations with two rules. First, "counterfactual harm": the AI should never push a person to abandon a correct judgment they already hold. Second, "add value": when the person is missing the right answer, the AI should help them recover it. Users define precisely what these mean for their task, and our algorithm adjusts how cautiously the AI expresses its uncertainty across many conversations so both rules are reliably met, without assuming anything about how the human behaves. In simulated medical dialogues and a study with real people, our method stayed on target even as behavior varied, and adjusting the two rules predictably changed how often people reached the right answer. Our finding suggest that the two rules are practical levers for more reliable human–AI conversation.