Rare Event Analysis of Large Language Models

Being probabilistic models, during inference large language models (LLMs) display rare events: behaviour that is far from typical but highly significant. By definition all rare events are hard to see, but the enormous scale of LLM usage means that events completely unobserved during development are likely to become prominent in deployment. Here we present an end-to-end framework for the systematic analysis of rare events in LLMs. We provide a practical implementation spanning theory, efficient generation strategies, probability estimation and error analysis, which we illustrate with concrete examples. We outline extensions and applications to other models and contexts, highlighting the generality of the concepts and techniques presented here.

Rare events are occurrences that are both important and unlikely. In the physical sciences, these might correspond to extreme weather events such as tsunamis. In language modelling, they are unwanted completions generated by the model, given some prompt. Such completions are made unlikely through careful development and testing. However, due to the enormous scale at which language models are used, unwanted completions can go undetected during model testing yet still occur for users after deployment. Just as for tsunami contingency planners, the challenge for LLM developers is that any outcomes that are hard to see are also hard to protect against. In the physical sciences, such as statistical physics and computational chemistry, powerful techniques have been developed to efficiently simulate and analyse these outcomes. By adapting these techniques to LLMs we have shown how, with only 8 million generations, it is possible to sample and study completions with properties that are estimated to occur only once in a hundred trillion samples. Hence, this methodology would allow developers of large foundation models, regulatory agencies and other researchers to evaluate the safety of LLMs at a deployment scale, without having to waste huge amounts of compute to sample these rare completions.