Timezone: »

 
An Adaptive Method for Minimizing Non-negative Losses
Antonio Orvieto · Lin Xiao

@
in HiLD: High-dimensional Learning Dynamics Workshop »

This paper introduces Non-negative Gauss-Newton (NGN), an adaptive optimization method that exploits non-negativity, a common feature of the loss functions in machine learning. Utilizing a Gauss-Newton-inspired approximation for non-negative losses, NGN offers an adaptive stepsize that can automatically warm up and decay while tracking the complex loss landscapes. We provide both convergence rates and empirical evaluations, and the results are very promising compared to the classical (stochastic) gradient method both in theory and practice.

Author Information

Antonio Orvieto (ETH Zurich)
Lin Xiao (Meta)

More from the Same Authors