Conf-Gen: Conformal Uncertainty Quantification for Generative Models

Conformal prediction (CP) and its extension, conformal risk control (CRC), are established frameworks for quantifying uncertainty in supervised machine learning through formal guarantees. However, recent breakthroughs in artificial intelligence (AI) have been driven by unsupervised generative models, such as large language models (LLMs) and image generators, which are not directly compatible with CP or CRC. In this work we introduce conformal generation (Conf-Gen), a general framework adapting CRC to generative tasks while relaxing its theoretical assumptions. Conf-Gen unifies and generalizes previous attempts to apply CP to LLMs, and extends conformal methodology to entirely new domains. We demonstrate the flexibility of Conf-Gen through some novel applications, including obtaining conformal guarantees on: image generators producing non-memorized images, conversational AI systems having asked enough clarifying questions, and the output of AI agents being correct.

Uncertainty quantification remains an open challenge in AI. In this work, we show how we can change the outputs of AI systems so as to have mathematical certainty that the new outputs satisfy certain properties with some desired probability. For example, instead of outputting a single response to a question, we can output multiple answers in such a way that at least one of the produced answers is correct with, say, 90% probability. Another example of the same underlying method is producing an image which is guaranteed to not be reproducing an existing (training) image with a user-specified probability. The goal of this research is to improve the safety and reliability of AI.