Out-of-Distribution Evaluation of Rule-Based and Strategic Reasoning in Chess Transformers

Modern decision transformers, trained similarly to LLMs, can achieve strong in-distribution performance in complex sequential domains like chess, but it remains unclear to what extent they reason systematically about rules and strategy. We study the reasoning capabilities of a 270M-parameter chess transformer trained via behavior cloning on standard chess. To investigate its abilities, we construct out-of-distribution test sets ---including board states and variants never seen during training---designed to reveal failures of systematic generalization. Our analysis shows that the model exhibits robust rule-based reasoning, consistently generating legal moves in novel configurations, but its strategic reasoning is more limited. The model generates high-quality moves on curated OOD puzzles and shows basic strategy adaptation in full games. It underperforms symbolic AI algorithms that rely on explicit search, although the performance gap is smaller when playing against human users on Lichess. Moreover, the training dynamics reveals distinct phases in how the model learns to respect the fundamental constraints, suggesting an emergent compositional understanding of the game.

Modern AI systems are increasingly used to make decisions in complex settings such as games, code generation, and planning. However, it is not well understood whether these systems truly learn the underlying rules of a domain or whether they mainly imitate patterns seen during training. This distinction matters because reliable deployment requires both correct rule-following and good decision-making. In this work, we study a chess-playing AI model that is trained only by learning from examples of human games. We test it not only on standard chess positions, but also on many carefully designed situations that have been never seen during training, including unusual board configurations and modified game rules. We find that the model is very reliable at following the rules of chess, even in unfamiliar situations, and almost always produces legal moves. However, its ability to choose strong strategies is less robust, especially in highly unusual scenarios. It performs well on many challenging test puzzles, but still falls short of traditional chess engines that use explicit search to plan ahead. When playing against human users, the gap is smaller, suggesting the model can still be practically useful. We also observe that the model’s learning process follows a clear progression: it first learns basic rules of movement and only later improves its ability to choose better moves. Overall, our results suggest that large AI models can learn the structure of a complex system like chess quite reliably, but improving strategic reasoning remains an important open challenge.