Integrated-second-order-moment Adam for Accelerated Convergence

Adaptive Moment Estimation (Adam) is one of the most widely used optimizers in deep learning, primarily due to its rapid convergence. This property can be attributed to Adam's interpretation as an approximation to Newton's method, which leverages the Hessian matrix to accelerate optimization. However, this approximation depends on second-order moment estimations of Adam that are often susceptible to gradient noises, potentially leading to inaccurate estimations. In this work, we treat the sequence of second-order estimations as a noisy signal. Based on this insight, we introduce an integration filter (IF) into the second-order-moment estimations to filter out noises and obtain more accurate estimations than Adam. The proposed optimizer achieves a more accurate approximation of Newton's method than Adam and is called integrated-second-order-moment Adam (ISAdam). Extensive simulations and experiments demonstrate that ISAdam accelerates convergence with negligible extra computational overhead. The code will be publicly available after the review.