Tractable Expected Information Gains for Exponential Family Posteriors

We investigate which models admit a collapse of the expected information gain (EIG) and its derivative from a doubly intractable to a singly intractable expression. We prove that a sufficient condition is that the posterior distribution belongs to an exponential family (EF) and depends on the experimental design and data only through its natural parameters, and derive corresponding singly intractable and unbiased estimators for the $\operatorname{EIG}$ and its (reparameterized) gradient. We further show that this is achieved when using a likelihood of an analogous form and any arbitrary prior. This is complemented by a theoretical analysis of certain degenerate behaviors that may arise when optimizing the $\operatorname{EIG}$ for EF-modeled experiments. Finally, we empirically demonstrate the benefits of using our singly intractable estimators, showing superior convergence rates, and substantial performance gains for sequential design problems compared to using standard nested estimators.

When running an experiment, we want to choose the setup that will be most informative. This can be formalised. A setup that is very informative has high "information gain", so we can pick setups which expect have high information gain, even though we haven't yet seen the results. Working out the "expected information gain" requires a calculation that is, in practice, extremely expensive — it involves two nested approximations that mean overall, it is hard to estimate well. We identify a class of situations where one of those layers of approximation can be stripped away entirely, making the calculation cheaper and more accurate. We show exactly what conditions an experiment needs to satisfy for this shortcut to apply. We also flag some subtle ways things can go wrong when using models that fit within this class. In real experiments, our approach learns faster and produces better results than the standard method.