Invited Talk
in
Workshop: Combining Theory and Benchmarks: Towards A Virtuous Cycle to Understand and Guarantee Foundation Model Performance Thu, Jul 9, 2026 • 4:05 PM – 4:40 PM PDT

Keynote #1: Combining theory and benchmarks: The DARPA Artificial Intelligence Quantified (AIQ) program

Patrick Shafto

Abstract

Evaluation of AI models is almost exclusively empirical, with essentially no way to guarantee or even predict performance in deployment. The DARPA AIQ program aims to provide new theory, which when combined with empirics, provides guarantees on performance. I will take stock of progress after one year, highlighting promising avenues forward toward theory and benchmarks with guaranteed performance.

Speaker

Patrick Shafto

Video

Chat is not available.