This paper introduces Non-negative Gauss-Newton (NGN), an adaptive optimization method that exploits non-negativity, a common feature of the loss functions in machine learning. Utilizing a Gauss-Newton-inspired approximation for non-negative losses, NGN offers an adaptive stepsize that can automatically warm up and decay while tracking the complex loss landscapes. We provide both convergence rates and empirical evaluations, and the results are very promising compared to the classical (stochastic) gradient method both in theory and practice.