Timezone: »

 
Pareto Invariant Risk Minimization
Yongqiang Chen · Kaiwen Zhou · Yatao Bian · Binghui Xie · Kaili MA · Yonggang Zhang · Han Yang · Bo Han · James Cheng

@
in Principles of Distribution Shift (PODS) »

Despite the success of invariant risk minimization (IRM) in tackling the Out-of-Distribution generalization problem, IRM can compromise the optimality when applied in practice. The practical variants of IRM, e.g., IRMv1, have been shown to have significant gaps with IRM and thus could fail to capture the invariance even in simple problems. Moreover, the optimization procedure in IRMv1 involves two intrinsically conflicting objectives, and often requires careful tuning for the objective weights. To remedy the above issues, we reformulate IRM as a multi-objective optimization problem, and propose a new optimization scheme for IRM, called PAreto Invariant Risk Minimization (PAIR). PAIR can adaptively adjust the optimization direction under the objective conflicts. Furthermore, we show PAIR can empower the practical IRM variants to overcome the barriers with the original IRM when provided with proper guidance. We conduct experiments with ColoredMNIST to confirm our theory and the effectiveness of PAIR.

Author Information

Yongqiang Chen (The Chinese University of Hong Kong)
Kaiwen Zhou (The Chinese University of Hong Kong)
Yatao Bian (Tencent AI Lab)
Binghui Xie (The Chinese University of Hong Kong)
Kaili MA (CUHK)
Yonggang Zhang (Hong Kong Baptist University)
Han Yang (The Chinese University of Hong Kong)
Bo Han (HKBU / RIKEN)
James Cheng (CUHK)

More from the Same Authors