Timezone: »

 
Hyperparameter Tuning using Loss Landscape
Jianlong Chen · Qinxue Cao · Yefan Zhou · Konstantin Schürholt · Yaoqing Yang

@
in HiLD: High-dimensional Learning Dynamics Workshop »

Hyperparameter tuning is crucial in training deep neural networks. In this paper, we propose a new method for hyperparameter tuning by (1) measuring multiple metrics related to the structure of the loss landscape, (2) empirically determining the "phase" of the loss landscape, and (3) providing an efficient way of hyperparameter tuning using the phase information. We demonstrate the effectiveness of our method through extensive experiments on network pruning tasks. We show that our method, named TempTuner, achieves significantly lower search time than both conventional grid search and more advanced sequential model-based Bayesian optimization (SMBO). To the best of our knowledge, this is the first work to apply loss landscape analysis to the novel application of hyperparameter tuning in neural networks.

[ Poster] 

Author Information

Jianlong Chen (Dartmouth college)
Qinxue Cao (University of Illinois Urbana-Champaign)
Yefan Zhou (UC Berkeley & International Computer Science Institute)
Yefan Zhou

Hi, I'm Yefan. I am an ML researcher working with the Big Data Group at International Computer Science Institute (ICSI). I earned my Master's degree in EECS at UC Berkeley where I had the pleasure of being advised by Prof. Michael Mahoney. I will join Dartmouth College as a CS Ph.D. student in Fall 2023.

Konstantin Schürholt (University of St. Gallen)
Yaoqing Yang (Dartmouth College)

More from the Same Authors