We present a framework for going beyond first order methods we refer to as the “descent method framework.” We construct a family of algorithms using intuition from dynamical systems, and show how they obtain fast non-asymptotic convergence for various classes of smooth functions. We also present a framework for “accelerating” or adding momentum descent algorithms with better theoretical complexity guarantees. We end by illustrating how this framework can be extended to manifolds and possibly lead to new techniques and analyses that incorporate geometric properties of various types of parameter spaces.