Timezone: »

 
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior
Ravid Shwartz-Ziv · Micah Goldblum · Hossein Souri · Sanyam Kapoor · Chen Zhu · Yann LeCun · Andrew Wilson

Sat Jul 23 11:00 AM -- 11:15 AM (PDT) @
in The First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward »
Event URL: https://openreview.net/forum?id=ao30zaT3YL »

Deep learning is increasingly moving towards a transfer learning paradigm whereby large ``foundation models'' are fine-tuned on downstream tasks, starting from an initialization learned on the source task. But an initialization contains relatively little information about the source task. %, and would not affect the final solution at all if we do a good job of optimization. Instead, we show that we can learn highly informative posteriors from the source task, which serves as the basis for priors that modify the whole loss surface on the downstream task. This simple modular approach enables significant performance gains and more data-efficient learning on various downstream classification and segmentation tasks, serving as a drop-in replacement for standard pre-training strategies.

Author Information

Ravid Shwartz-Ziv (New York University)
Micah Goldblum (New York University)
Hossein Souri (Johns Hopkins University)
Sanyam Kapoor (New York University)
Chen Zhu (Google)
Yann LeCun (New York University)
Andrew Wilson (New York University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors