A Planning-Based Reinforcement Learning Approach to Numerical Optimization

Optimization lies at the core of scientific computing and machine learning, where algorithms iteratively update parameters using function values and gradient information. In this work, we investigate whether effective optimization strategies can be learned from fixed datasets of such trajectories and transferred to new problem instances without further learning. We formulate numerical optimization as a Markov Decision Process (MDP), where each state consists of the current iterate, function value, and gradient information, and actions correspond to update directions and step sizes. We then adapt the popular planning-based Reinforcement Learning (RL) paradigm, AlphaGo, to numerical optimization to train policy and value networks offline using a family of Rosenbrock functions. At deployment, the learned policy and value functions are kept fixed and used within a Monte Carlo Tree Search (MCTS) procedure to perform trajectory-level planning. This enables the method to combine offline-learned optimization priors with test-time lookahead, allowing decisions that account for long-term effects of updates rather than relying solely on local heuristics. We evaluate the approach on a diverse suite of optimization problems beyond the training distribution. Empirical results show that the learned policy, when coupled with planning, generalizes effectively and achieves reliable convergence with competitive or improved optimization efficiency. Our results demonstrate that AlphaGo-style learning, when applied to optimization provides a promising framework for learning adaptive strategies from offline data.