Rethinking Memorization Measures in LLMs: Recollection vs. Counterfactual vs. Contextual Memorization

Memorization in large language models (LLMs) is often viewed as undesirable for learning. Existing memorization measures largely focus on quantifying privacy risks, rather than capturing the underlying phenomenon of memorization itself. To address this gap, we introduce contextual memorization, which disentangles memorization from contextual learning -- both of which LLMs perform during training. We further show that different measures -- recollection-based, counterfactual, and contextual -- can yield contradictory implications when applied to the same training dynamics, such as disagreeing on the order of memorization of strings of varying frequencies.