Invited talk: Incentivizing Quality AI with Statistical and Adaptive Contracts -- Inbal Talgam-Cohen

Rethinking LLM pricing calls for an integration of ideas from machine learning, contract design, and statistics. The moral hazard introduced by current pay-per-token pricing in LLM deployments might incentivize agents to strategically route requests to cheaper, lower-quality models behind the scenes in order to cut internal inference costs. To resolve this misalignment, we introduce a pay-for-performance pricing framework grounded in contract theory, utilizing a new notion of cost-robust contracts. We establish a direct connection between the theory of optimal contract design and the theory of optimal hypothesis testing in statistics, showing that optimal contracts are essentially scaled optimal hypothesis tests, and proving a direct correspondence between statistical and economic objectives. Pay-for-performance pricing requires performance evaluation, which in our context is quality evaluation of the LLM-generated response. Such evaluation is not easy and is certainly not without cost. The notion of adaptive contracts allows detailed evaluation to be performed selectively. We provide a theoretical analysis of the tradeoff between coarse and refined evaluation, and an empirical demonstration of the benefits of adaptivity using question-answering and code-generation datasets.