Memory Savings at What Cost? A Study of Alternatives to Backpropagation

Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing. We present a unified theoretical and empirical comparison of BP, checkpointed BP, FmAD, and ZO for LLM and vision-language model training, showing that while FmAD and ZO reduce activation memory, they trade memory for higher computational cost and longer wall-clock time to convergence, resulting in lower accuracy and slower training, especially under constrained perturbation budgets. Across models, BP with checkpointing outperforms FmAD and ZO variants, including variance-reduced methods, achieving up to 31.1% higher accuracy, 34.8% faster convergence, and 3.8$\times$ fewer computations at comparable memory usage, while also revealing instability-related failure modes in FmAD and ZO. Overall, our results correct a one-sided benchmarking narrative by showing that memory-efficient methods entail fundamentally different trade-offs, and that ignoring these distinctions has led to misleading conclusions about LLM optimization in prior work.

Training large AI models such as modern language models requires enormous amounts of memory and computation. To reduce these costs, researchers have recently proposed alternatives to standard backpropagation, the main algorithm used to train neural networks. Two popular alternatives, forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization, are often described as more memory-efficient methods for training large models. However, prior studies have usually compared them only against standard backpropagation, while overlooking widely used memory-saving techniques such as activation checkpointing. In this work, we provide a comprehensive analysis of these training methods across language and vision-language models. We study not only memory usage, but also training speed, computational cost, stability, and final model accuracy. Our analysis shows that although FmAD and ZO can reduce memory usage in some settings, they do so at the cost of much higher computation and slower learning. As models become larger, these methods also become less stable and more difficult to scale effectively. Across multiple tasks, standard backpropagation combined with activation checkpointing consistently achieved better results than FmAD and ZO methods, including newer variance-reduced versions. It reached higher accuracy, converged faster, and required fewer overall computations while using similar amounts of memory. We also identify several previously underexplored failure modes in FmAD and ZO training, including unstable gradient estimates that can disrupt learning. Overall, our findings show that memory-efficient training methods involve important trade-offs that have often been overlooked in prior work. Rather than replacing backpropagation for large-model training, FmAD and ZO currently remain slower and less reliable approximations in most practical settings.