CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing

A central challenge in large language model (LLM) editing is capability preservation: methods that successfully change targeted behavior can quietly game the editing proxy and corrupt general capabilities, producing degenerate behaviors reminiscent of proxy/reward hacking. We present CrispEdit, a scalable and principled second-order editing algorithm that treats capability preservation as an explicit constraint, unifying and generalizing several existing editing approaches. CrispEdit formulates editing as constrained optimization and enforces the constraint by projecting edit updates onto the low-curvature subspace of the capability-loss landscape. At the crux of CrispEdit is expressing capability constraint via Bregman divergence, whose quadratic form yields the Gauss–Newton Hessian exactly and even when the base model is not trained to convergence. We make this second-order procedure efficient at the LLM scale using Kronecker-factored approximate curvature (K-FAC) and a novel matrix-free projector that exploits Kronecker structure to avoid constructing massive projection matrices. Across standard model-editing benchmarks and safety unlearning tasks, CrispEdit achieves high edit success while keeping capability degradation below 1% on average across datasets, significantly improving over prior editors.

Large language models often need to be updated after they are trained: facts become outdated, mistakes are discovered, and some memorized information may need to be removed for privacy, copyright, or safety reasons. A common challenge is that changing one specific behavior can unintentionally damage the model’s other abilities, such as reasoning or answering unrelated questions. We introduce CrispEdit, a method for making targeted changes to a language model while preserving its broader capabilities. To teach model a new knowledge, CrispEdit considers all the steps a model can take and filters the steps which would disturb the model's old inform This lets the model learn new or corrected information, or abstain from answering questions about information that should be knowledge. We consider benchmarks in which one can test a model's performance on both learning new knowledge and retaining old ones. Across several such tests, CrispEdit makes reliable targeted changes while causing much less damage to general model performance than prior editing methods. This makes model editing more practical for keeping deployed AI systems safer and up to date.