Timezone: »

Fully Parallel Hyperparameter Search: Reshaped Space-Filling
Marie-Liesse Cauwet · Camille Couprie · Julien Dehos · Pauline Luc · Jeremy Rapin · Morgane Riviere · Fabien Teytaud · Olivier Teytaud · Nicolas Usunier

Tue Jul 14 10:00 AM -- 10:45 AM & Tue Jul 14 11:00 PM -- 11:45 PM (PDT) @ None #None

Space-filling designs such as Low Discrepancy Sequence (LDS), Latin Hypercube Sampling (LHS) and Jittered Sampling (JS) were proposed for fully parallel hyperparameter search, and were shown to be more effective than random and grid search. We prove that LHS and JS outperform random search only by a constant factor. Consequently, we introduce a new sampling approach based on the reshaping of the search distribution, and we show both theoretically and numerically that it leads to significant gains over random search. Two methods are proposed for the reshaping: Recentering (when the distribution of the optimum is known), and Cauchy transformation (when the distribution of the optimum is unknown). The proposed methods are first validated on artificial experiments and simple real-world tests on clustering and Salmon mappings. Then we demonstrate that they drive performance improvement in a wide range of expensive artificial intelligence tasks, namely attend/infer/repeat, video next frame segmentation forecasting and progressive generative adversarial networks.

Author Information

Marie-Liesse Cauwet (Université Paris-Est, LIGM (UMR 8049), CNRS, ESIEE Paris)
Camille Couprie (FAIR)
Julien Dehos (LISIC, Université du Littoral Côte d'Opale)
Pauline Luc (Deepmind)
Jeremy Rapin (Facebook AI Research)
Morgane Riviere (Facebook Artificial Intelligence Research)
Fabien Teytaud (LISIC, Université du Littoral Côte d'Opale)
Olivier Teytaud (Facebook)
Nicolas Usunier (Facebook AI Research)

More from the Same Authors