Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers. Their diagonal preconditioner is based on the gradient outer product which is incorporated into the parameter update via a square root. While these methods are often motivated as approximate second-order methods, the square root representsa fundamental difference. In this work, we investigate how the behavior of adaptive methodschanges when we remove the root, i.e. strengthen their second-order motivation. Surprisingly, wefind that such square-root-free adaptive methods close the generalization gap to SGD on convolutional architectures, while maintaining their root-based counterpart’s performance on transformers.The second-order perspective also has practical benefits for the development of adaptive methodswith non-diagonal preconditioner. In contrast to root-based counterparts like Shampoo, they do notrequire numerically unstable matrix square roots and therefore work well in low precision, whichwe demonstrate empirically. This raises important questions regarding the currently overlooked role of adaptivity for the success of adaptive methods.