Skip to yearly menu bar Skip to main content


Automated Synthetic-to-Real Generalization

Wuyang Chen · Zhiding Yu · Zhangyang “Atlas” Wang · Anima Anandkumar

Keywords: [ Transfer, Multitask and Meta-learning ] [ Meta-learning and Automated ML ] [ Transfer and Multitask Learning ]


Models trained on synthetic images often face degraded generalization to real data. As a convention, these models are often initialized with ImageNet pretrained representation. Yet the role of ImageNet knowledge is seldom discussed despite common practices that leverage this knowledge to maintain the generalization ability. An example is the careful hand-tuning of early stopping and layer-wise learning rates, which is shown to improve synthetic-to-real generalization but is also laborious and heuristic. In this work, we explicitly encourage the synthetically trained model to maintain similar representations with the ImageNet pretrained model, and propose a \textit{learning-to-optimize (L2O)} strategy to automate the selection of layer-wise learning rates. We demonstrate that the proposed framework can significantly improve the synthetic-to-real generalization performance without seeing and training on real data, while also benefiting downstream tasks such as domain adaptation. Code is available at:

Chat is not available.