Fixed-Budget Hypothesis Best Arm Identification: On the Information Loss in Experimental Design

Experimental design plays a crucial role in evidence-based science with multiple treatment arms, such as online advertisements or medical treatments. This study addresses the task of identifying the best treatment arm, which has the highest expected outcome among multiple treatment arms We investigate the influence of available information regarding the distributions of treatment arms in experiments. In our experimental setup, we first designate a hypothetical ``best'' treatment arm and then conduct an experiment to verify whether this hypothetically best treatment arm is indeed the 'true' best treatment arm. Our null hypothesis posits that the hypothetical best treatment is not the actual best, and our objective is to minimize the likelihood of recommending other treatment arms when the null hypothesis is false; in other words, when the true best treatment arm is the same as the hypothetical best treatment. We demonstrate that the optimal experimental design significantly depends on knowledge about distributional information, examined through an information-theoretic approach. Specifically, we discuss worst-case scenarios, characterized by a loss of distributional information, as circumstances when gaps between the expected outcomes of the best and sub-optimal treatment arms convege to zero. After discussing asymptotic optimality, we propose an experimental design informed by the available information.