Regularizing Optimizer Updates via Feasible-Set Projection

Modern optimizers can produce parameter updates with radial components that increase weight norms during training. Since excessive weight-norm growth is closely related to poor generalization, controlling the geometry of updates provides a natural form of regularization without redesigning the optimizer itself. We propose Regularizing Optimizer Updates via Feasible-Set Projection, a simple update constraint applied at the final stage of optimization. Given a raw update from a base optimizer, we project it onto the half-space of directions that are orthogonal to or inward-facing with respect to the current parameter vector, thereby suppressing norm growth while preserving the tangential component of the optimizer's update. The proposed constraint is compatible with general optimizer updates and introduces no additional hyperparameter tuning. In this work, we apply the proposed constraint to Adam and compare it against AdamW, showing that it provides one-step norm control, a bounded parameter-norm trajectory, and improved generalization in our experiments.