Skip to yearly menu bar Skip to main content


Meta-learning for Mixed Linear Regression

Weihao Kong · Raghav Somani · Zhao Song · Sham Kakade · Sewoong Oh

Keywords: [ Bayesian Methods ] [ Transfer and Multitask Learning ] [ Meta-learning and Automated ML ] [ Spectral Methods ] [ Probabilistic Inference - Approximate, Monte Carlo, and Spectral Methods ]

Abstract: In modern supervised learning, there are a large number of tasks, but many of them are associated with only a small amount of labelled data. These include data from medical image processing and robotic interaction. Even though each individual task cannot be meaningfully trained in isolation, one seeks to meta-learn across the tasks from past experiences by exploiting some similarities. We study a fundamental question of interest: When can abundant tasks with small data compensate for lack of tasks with big data? We focus on a canonical scenario where each task is drawn from a mixture of $k$ linear regressions, and identify sufficient conditions for such a graceful exchange to hold; there is little loss in sample complexity even when we only have access to small data tasks. To this end, we introduce a novel spectral approach and show that we can efficiently utilize small data tasks with the help of $\tilde\Omega(k^{3/2})$ medium data tasks each with $\tilde\Omega(k^{1/2})$ examples.

Chat is not available.