Meta-learning Hyperparameter Performance Prediction with Neural Processes

Ying WEI · Peilin Zhao · Junzhou Huang

Keywords: [ Supervised Learning ] [ Algorithms ]

[ Abstract ]
[ Paper ]
[ Visit Poster at Spot C2 in Virtual World ] [ Visit Poster at Spot D5 in Virtual World ]
Wed 21 Jul 9 a.m. PDT — 11 a.m. PDT
Spotlight presentation: Supervised Learning 1
Wed 21 Jul 5 a.m. PDT — 6 a.m. PDT


The surrogate that predicts the performance of hyperparameters has been a key component for sequential model-based hyperparameter optimization. In practical applications, a trial of a hyper-parameter configuration may be so costly that a surrogate is expected to return an optimal configuration with as few trials as possible. Observing that human experts draw on their expertise in a machine learning model by trying configurations that once performed well on other datasets, we are inspired to build a trial-efficient surrogate by transferring the meta-knowledge learned from historical trials on other datasets. We propose an end-to-end surrogate named as Transfer NeuralProcesses (TNP) that learns a comprehensive set of meta-knowledge, including the parameters of historical surrogates, historical trials, and initial configurations for other datasets. Experiments on extensive OpenML datasets and three computer vision datasets demonstrate that the proposed algorithm achieves state-of-the-art performance in at least one order of magnitude less trials.

Chat is not available.