Low-Rank Embedding Adaptation for Models with Expanding Vocabularies

Pretrained models are routinely extended with new vocabulary entries, for example new items in recommenders and new tokens in LLMs. We show that jointly training old and new embeddings has a hidden failure mode: old-entry quality degrades while new entries are still learning. On sequential recommendation, old items overfit while new items improve, forcing premature early-stopping; on LLMs, base-token perplexity rises within 1–2 epochs. We propose population-specific low-rank subspaces that extend the low-rank adaptation principle from weight matrices to embedding tables, with separate shared projections for base and new entries. On sequential recommendation (SASRec, GRU4Rec; Taobao, MerRec), our methods Pareto-dominate joint fine-tuning and continual-learning baselines. On LLM vocabulary expansion (GPT-2, Pythia-410M, Pythia-1.4B; BillSum, PubMed), our low-rank methods improve overall perplexity over joint training in 8 out of 9 model-scale cells, outperforming full-rank frozen baselines in 8/9 cells at 2× compression and 6/9 at 4× compression.