Skip to yearly menu bar Skip to main content


Poster

LoRA Training in the NTK Regime has No Spurious Local Minima

Uijeong Jang · Jason Lee · Ernest Ryu

Hall C 4-9 #1504
[ ]
Thu 25 Jul 4:30 a.m. PDT — 6 a.m. PDT
 
Oral presentation: Oral 6B Low Rank Learning
Thu 25 Jul 7:30 a.m. PDT — 8:30 a.m. PDT

Abstract: Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models (LLM), but our theoretical understanding of LoRA has been limited. In this work, we theoretically analyze LoRA fine-tuning in the neural tangent kernel (NTK) regime with $N$ data points, showing: (i) full fine-tuning (without LoRA) admits a low-rank solution of rank $r\lesssim \sqrt{N}$; (ii) using LoRA with rank $r\gtrsim \sqrt{N}$ eliminates spurious local minima, allowing gradient descent to find the low-rank solutions; (iii) the low-rank solution found using LoRA generalizes well.

Live content is unavailable. Log in and register to view live content