Using Differences in State Law to Test Whether LLMs Reason or Remember

Large language models have demonstrated strong performance on a variety of legal benchmarks. But these results can often conflate genuine legal reasoning ability with more heuristic pattern-matching. We exploit state-by-state legal variation in the United States to disentangle these two accounts. Because jurisdictions can disagree on how to resolve the same legal question, some legal rules fall along a spectrum of training-data prevalence: a majority rule adopted by most states, a minority rule adopted by few, or a hypothetical rule that is adopted by none. Using differences in the prevalence of legal rules across U.S. states as a proxy for the representation of a legal rule in the training data, we evaluate LLMs' abilities to apply these legal rules to straightforward fact patterns that produce certain results. If a model is engaged in genuine legal reasoning, it should produce consistently accurate answers for different rules. But if model performance is driven by training-data familiarity, accuracy should be higher for legal rules that are well-represented in the training data and lower for rules that are poorly represented. We find that models perform best on majority rules, worst on minority rules, and at intermediate levels on hypothetical rules. These results support two conclusions: First, LLM performance on legal tasks may be grounded in the task’s similarity to training data rather than general legal reasoning capabilities; and second, that uncommon legal rules may experience interference from contradicting more common rules.