Atomic Chess Reveals Compositional Reasoning Failures in LLMs
Ryan Co ⋅ Karthik R Konuganti
Abstract
LLMs are capable of playing chess at a notable level, but it is unclear whether this is a result of memorization or reasoning. We study whether they can compose an explicit rule change using atomic chess, a variant that preserves the board, pieces, and surface notation, but where capturing a piece will explode adjacent ones. We evaluate Claude Opus 4.6 and GPT-5.4 on contrastive pattern-trap positions drawn from atomic chess games. Atomic Win% loss is consistently larger than under standard rules on the same positions, with atomic/standard mean-loss ratios from $2.1\times$ to $4.6\times$. Increasing reasoning compute reduces atomic-rules mean Win% loss, but does not eliminate the gap to standard rules. Mapping observed errors onto the components of a chess game decomposition, qualitative analysis of reasoning traces identifies recurring failure mechanisms, including unpropagated refutation, where a model recognizes that a candidate move is bad under atomic rules but selects it anyway, a failure mode directly analogous to safety-relevant agent settings in which a model identifies a violating action but executes it. These results suggest that reasoning LLMs can partially use explicit compositional rules, but often fail to integrate altered transition and terminal semantics into global action selection.
Chat is not available.
Successful Page Load