Asymptotic Bayesian Generalization Error when Training and Test Distributions are Different

Asymptotic Bayesian Generalization Error when Training and Test Distributions are Different
Keisuke Yamazaki - Tokyo Institute of Technology, Japan Motoaki Kawanabe - Fraunhofer FIRST IDA, Germany Sumio Watanabe - Tokyo Institute of Technology, Japan Masashi Sugiyama - Tokyo Institute of Technology, Japan Klaus-Robert Müller - Fraunhofer FIRST IDA, Technical University of Berlin, Germany
In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian generalization error and establish its upper bound under a very general setting. Our important finding is that lower order terms -- which can be ignored in the absence of the distribution change -- play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.

Keisuke Yamazaki - Tokyo Institute of Technology, Japan
Motoaki Kawanabe - Fraunhofer FIRST IDA, Germany
Sumio Watanabe - Tokyo Institute of Technology, Japan
Masashi Sugiyama - Tokyo Institute of Technology, Japan
Klaus-Robert Müller - Fraunhofer FIRST IDA, Technical University of Berlin, Germany

In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian generalization error and establish its upper bound under a very general setting. Our important finding is that lower order terms -- which can be ignored in the absence of the distribution change -- play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.