No LLM Aced Yu Tsumura's 554th Problem

We present a case study about a group theory problem---Yu Tsumura's 554th problem---that has surprisingly diagnostic capabilities for LLMs. On the one hand, this shows, contrary to the optimism about LLMs' problem-solving abilities, fueled by the recent gold medals at the International Math Olympiad (IMO) that LLMs attained, that a group theory problem exists---Yu Tsumura's 554th problem---that a) is within the scope of an IMO problem in terms of proof sophistication, b) is not a combinatorics problem, which is known to cause issues for LLMs, c) has a publicly available solution (likely in the training data of LLMs), and d) cannot be stably solved by any existing off-the-shelf LLM (commercial or open-source). This problem is the end result of a discovery process involving 300+ group theory problems, few of which consistently produced comparable failures across a selection of LLMs, and none of which was at the same level of mathematical sophistication. We include an analysis of the output traces of 31 LLMs in the years 2025-2026, where our focus is on the top models and their lineages, tracking intermediate versions. Our analysis shows that LLMs reasoning performance across problems is spiky even for comparatively simple problems and implies that there may be fundamental issues with model training approaches. Additionally, we compare, in a case study, the generic LLM output to a new proof of Yu Tsumura's problem by a former IMO participant, to contrast human reasoning ability with LLM reasoning ability. The focus on Yu Tsumura's problem further reveals technical issues in the original publicly available proof, and we present a substantially better-motivated proof.