Paper #18: Multi-Round Human–AI Collaboration with User-Specified Requirements

As humans increasingly rely on multi-round conversational AI for high-stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human-centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user-defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution-free algorithm with finite-sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM-simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual-harm and complementarity violation rates even under non-stationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.