Chebyshev Policies and the Mountain Car Problem: Reinforcement Learning for Low-Dimensional Control Tasks

We analytically solve the Mountain Car problem, a canonical benchmark in RL, and derive an optimal control solution, closing a gap after 36 years. This enables us to reveal two surprising insights: The optimal control is quite simple, yet modern RL agents display a large gap to optimality. Motivated by the analysis of the optimal control, we introduce Chebyshev policies as a universal (i.e. dense) class of RL policies from first principles. They can be trained as drop-in replacements of neural nets, reducing the regret by a factor of 4.18, while requiring 277 times fewer parameters, fostering sample efficiency, explainability and realtime capability. Chebyshev policies are evaluated on further RL tasks, including a real-world nonlinear motion control testbed. They consistently improve performance over neural nets with PPO, ARS and REINFORCE. Our results demonstrate how Chebyshev policies offer a compelling and lightweight alternative or addition to neural nets for low-dimensional control tasks.

Many learning systems today rely on large neural networks, which can be powerful but also slow, energy‑intensive, and difficult to interpret. We wondered whether a computer agent could learn effective behavior using something far simpler — basic mathematical formulas — instead of complex black‑box neural models. To explore this idea, we studied a classic challenge where a small car must figure out how to climb a steep hill. Instead of training a neural network, we used compact polynomial equations to help the system evaluate which actions would move it closer to its goal. We then examined how these simple formulas organized their understanding of the task and whether they could guide learning reliably. Surprisingly, these lightweight models performed extremely well and outperform the larger neural systems. This suggests that in certain situations, simple and transparent methods can replace heavier approaches without sacrificing performance. Our findings highlight new possibilities for building learning‑based systems that are faster, easier to interpret, and more practical for small devices or safety‑critical applications. At the same time, this research raises open questions on the limitations of state-of-the-art neural methods.