Causally Evaluating the Learnability of Formal Language Tasks

Language models, as multi-task learners, acquire a wide range of abilities during training. A fundamental question is how much task-specific data is needed to learn a given task. Answering this for natural language is difficult: tasks are hard to delineate and can confound one another. To rigorously investigate the relationship between data frequency and learnability, we turn to a controlled setting using formal languages induced from probabilistic finite automata. These serve as a methodological testbed to demonstrate that standard correlational evaluation practices are inherently flawed. To enable causal analysis, we introduce the binning semiring, an algebraic object that lets us control how often a targeted property occurs in a sampled corpus. We formulate the experimental pipeline as a causal graphical model and derive decomposed Kullback--Leibler divergence metrics to measure the learnability of specific sub-tasks. Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions due to confounders in correlational analysis, and serve as a warning about correlational pitfalls in natural-language settings.

Modern AI language models pick up many different skills from a single training dataset---translation, coding, math, and more---and a natural question is how much data about a particular skill the model needs to see before learning it well. This question is hard to answer with real text, because different skills overlap and influence each other in ways that are difficult to untangle. To study the question carefully, we built small artificial languages with known rules, where we could count exactly how often each pattern appears in the training data, and then trained two common types of AI models on these languages to measure how well each one learned specific patterns. We found that the usual research approach can give misleading answers: simply counting how often a pattern appears and checking whether the model learned it lets hidden factors fool you, because a pattern that naturally appears rarely also tends to show up alongside other patterns that make learning easier or harder overall, so the two effects get mixed. To fix this, we developed a mathematical tool and a method that let us directly control how often a chosen pattern appears in the training data without changing anything else about it. Using this approach, the conclusions change---sometimes dramatically---compared to the standard method, serving as a warning that studies which simply correlate data frequency with model performance can reach wrong conclusions, both for these artificial languages and likely for real language too.