Depth-Progressive Monotonic Learning without Global Backpropagation

Backpropagation (BP) remains the dominant training paradigm for deep neural networks, yet its reliance on global gradient propagation fundamentally induces update locking problem, enforcing strong inter-layer dependencies in parameter updates. To address this limitation, we propose Depth-progressive Monotonic Learning (DMoL), a training scheme that assigns layer-wise local belief objectives and incrementally refines them across network depth, enabling unlocked parameter updates. As a result, DMoL supports dynamic modification of network depth during training, adapting to available compute and device resources while maintaining stable optimization. We provide theoretical guarantees that layer-wise local belief objectives improve monotonically with increasing depth and converge exponentially. Empirically, DMoL consistently matches or outperforms BP across diverse tasks, yielding a 4.3\% accuracy gain on CIFAR-100, mitigating over-smoothing in deep graph neural networks (+37.5\% on Cora), and reducing the final loss by over 35\% in diffusion model training, highlighting its robustness and flexibility as an alternative to BP. The code is publicly available at: https://github.com/ychAlbert/DMoL.

Deep neural networks, the engines behind modern artificial intelligence, are typically trained using a method called backpropagation. While highly effective, backpropagation has a major limitation: every layer of the network must wait for feedback from all subsequent layers before it can update its parameters. This "update locking" prevents layers from being trained in parallel, increases memory demands, and makes networks structurally fragile—meaning we cannot easily add or remove layers during training to fit different computers or devices. To address this, we propose Depth-progressive Monotonic Learning (DMoL), an alternative training method that eliminates global backpropagation. Instead of relying on a single global feedback loop, DMoL gives each layer its own local objective to progressively refine the predictions of the layer right before it. This allows all layers to update their parameters independently and in parallel. As a result, DMoL makes training significantly more flexible and efficient. It allows developers to dynamically change a network's depth during training to adapt to different hardware capabilities (such as shifting workloads between smartphones and cloud servers) without losing stability. Empirically, our method achieves comparable or even superior performance compared to traditional backpropagation across a wide range of tasks, including image recognition, network analysis, and image generation.