Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs

Generalization without memorizing training data is one of the most fundamental challenges for any ML task.In the context of learning factual knowledge, memorization is shown to hinder generalization to semantically equivalent prompts. In this work, we demonstrate that LLMs can, in fact, be trained to generalize from rote-memorized data. We introduce a two-phase “memorize-then-generalize” framework, where the model first memorizes facts using a non-semantic key token prompt, and then learns to generalize using a small set of memorized facts with semantically meaningful prompts. We show that LLMs can reinterpret memorized knowledge to reflect new semantics, as evidenced by the emergence of structured, semantically aligned latent representations. These findings underscore both effective strategies for injecting and reasoning over knowledge, as well as emerging risks related to semantic manipulation.