We frame several popular iterative linearized control algorithms for nonlinear control such as ILQR and ILQG as nonlinear optimization algorithms on an objective whose first-order information can be computed using dynamic programming. Our framework allows us to identify a gradient back-propagation oracle corresponding to dynamic programming. The number of calls to this oracle is arguably the relevant measure of complexity of such algorithms in modern computing environments. We also highlight several missing components in these algorithms and propose stable and accelerated variants that enjoy worst-case complexity guarantees from a nonlinear optimization viewpoint.