Timezone: »

 
Oral
Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
Ashok Vardhan Makkuva · Pramod Viswanath · Sreeram Kannan · Sewoong Oh

Thu Jun 13 11:30 AM -- 11:35 AM (PDT) @ Room 101
in Optimization and Graphical Models »

Mixture-of-Experts (MoE) is a widely popular model for ensemble learning and is a basic building block of highly successful modern neural networks as well as a component in Gated Recurrent Units (GRU) and Attention networks. However, present algorithms for learning MoE, including the EM algorithm and gradient descent, are known to get stuck in local optima. From a theoretical viewpoint, finding an efficient and provably consistent algorithm to learn the parameters remains a long standing open problem for more than two decades. In this paper, we introduce the first algorithm that learns the true parameters of a MoE model for a wide class of non-linearities with global consistency guarantees. While existing algorithms jointly or iteratively estimate the expert parameters and the gating parameters in the MoE, we propose a novel algorithm that breaks the deadlock and can directly estimate the expert parameters by sensing its echo in a carefully designed cross-moment tensor between the inputs and the output. Once the experts are known, the recovery of gating parameters still requires an EM algorithm; however, we show that the EM algorithm for this simplified problem, unlike the joint EM algorithm, converges to the true parameters. We empirically validate our algorithm on both the synthetic and real data sets in a variety of settings, and show superior performance to standard baselines.

[ Slides]  [ Oral] 

Author Information

Ashok Vardhan Makkuva (UIUC)
Pramod Viswanath (UIUC)
Sreeram Kannan (University of Washington)
Sewoong Oh (University of Washington)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors