Revisiting OOD Generalization in Programmatic RL

Programmatic policies are often reported to generalize better than neural policies in reinforcement learning (RL) benchmarks. We revisit some of these claims and show that much of the observed gap arises from uncontrolled experimental factors rather than intrinsic representational reasons. Re-evaluating three core benchmarks used in influential papers---TORCS, Karel, and Parking---we find that neural policies, when trained with a few modifications, such as sparse observations and cautious intrinsic reward functions, can match or exceed the out-of-distribution (OOD) generalization of programmatic policies. We argue that a representation enables OOD generalization if (i) the policy space it induces includes a generalizing policy and (ii) the search algorithm can find it. The neural and programmatic policies in prior work are comparable in OOD generalization because the domain-specific languages used induce policy spaces similar to those of neural networks, and our modifications help the gradient search find generalizing solutions. By disentangling representational factors from experimental confounds, we advance our understanding of what makes a representation succeed or fail at OOD generalization.

Can programs really generalize better than neural networks? Existing work suggests they do, but we found that the answer is not so straightforward. In this paper, we revisit well-known results in which programs used as policies appear to outperform neural-network policies in terms of generalization. We show that this advantage often comes from differences in how the systems were trained, rather than from the policy being a program. With relatively simple changes to the methodology, such as adjusting the intrinsic reward used during training or limiting the information given to the agent, neural networks can generalize just as well in the domains we studied. To explain this, we highlight two properties. The first is expressivity: whether a model can represent a solution that generalizes. The second is discoverability: whether the learning process can actually find such a solution. In the cases we studied, both programs and neural networks were expressive enough, but the neural-network training pipeline required adjustments to make a generalizing solution discoverable. Our results suggest a shift in focus in future work. Instead of asking whether one should use programs or neural networks to achieve generalization, we should ask whether the chosen representation can express the solution required by the problem, and whether the learning process can find it. This perspective can guide the design of future methods that generalize, regardless of whether they are neural, programmatic, or a combination of both.