PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning

We develop a continual learning method for pretrained models that \emph{requires no access to old-task data}, addressing a practical barrier in foundation model adaptation where pretraining distributions are often unavailable. Our key observation is that pretrained networks exhibit substantial \emph{geometric redundancy}, and that this redundancy can be exploited in two complementary ways. First, redundant neurons provide a proxy for dominant pretraining-era feature directions, enabling the construction of approximately protected update subspaces directly from pretrained weights. Second, redundancy offers a natural bias for \emph{where} to place plasticity: by restricting updates to a subset of redundant neurons and constraining the remaining degrees of freedom, we obtain update families with reduced functional drift on the old-data distribution and improved worst-case retention guarantees. These insights lead to \textsc{PLATE} (\textbf{Pla}sticity-\textbf{T}unable \textbf{E}fficient Adapters), a continual learning method requiring no past-task data that provides explicit control over the plasticity-retention trade-off. PLATE parameterizes each layer with a structured low-rank update $\Delta W = B A Q^\top$, where $B$ and $Q$ are computed once from pretrained weights and kept frozen, and only $A$ is trained on the new task. Code is available at https://github.com/SalesforceAIResearch/PLATE.

When foundation models learn new skills, they often suffer from "catastrophic forgetting," meaning they overwrite and lose their previous abilities. Preventing this forgetting requires constant access to the original training data, which is often proprietary, overwhelmingly large, or simply unavailable for modern, massive systems. To solve this without needing the original data, we developed a method called PLATE. Our key insight is that large models are highly redundant, meaning many of their internal pathways essentially do the same job. PLATE uses this structural redundancy to figure out which features are most important and mathematically protects them. When finetuning, PLATE forces the network to adapt by only altering the redundant, "backup" neurons. Ultimately, this approach allows finetuning on new tasks efficiently while successfully preserving original capabilities. This provides a highly practical way to update real-world foundation models without needing immense computing power or access to their original training data.