HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation

Fine-tuning large models on edge devices is severely hindered by the memory-intensive backpropagation (BP) in standard frameworks like federated learning and split learning. While substituting BP with zeroth-order optimization can significantly reduce memory footprints, it typically suffers from prohibitively degraded convergence speed. To resolve this dilemma, we propose Hybrid-Order Split Federated Learning (HO-SFL). By reformulating the split learning process within a Lagrangian framework, HO-SFL decouples the optimization landscape: The server performs precise first-order updates (i.e., BP), whereas clients conduct memory-efficient zeroth-order optimization. This hybrid design not only eliminates the need for client-side BP but also enables dimension-free model aggregation, drastically lowering communication costs. Crucially, we provide a theoretical convergence analysis, demonstrating that HO-SFL mitigates the dimension-dependent convergence slowdown of zeroth-order optimization, achieving a convergence rate comparable to first-order methods. Extensive experiments on tasks across vision and language modalities validate that HO-SFL achieves convergence speeds comparable to first-order baselines while significantly reducing communication costs and client memory footprints.

Modern AI models often need to be adapted to users’ personal data, such as data stored on phones, sensors, or other edge devices. However, training these models directly on such devices is difficult because the standard training procedure, backpropagation, requires too much memory. Existing memory-saving alternatives can avoid backpropagation, but they usually make training much slower. This paper introduces Hybrid-Order Split Federated Learning (HO-SFL), a new training framework that makes large-model fine-tuning more practical on resource-limited devices. The key idea is to split the work: a powerful server performs accurate backpropagation, while client devices update their local model parts using only lightweight forward passes. This removes the need for backpropagation on clients, greatly reducing their memory burden. At the same time, clients send only a few scalar values instead of large model updates, which significantly lowers communication cost. Theoretical analysis shows that HO-SFL avoids much of the slowdown usually caused by zeroth-order optimization. Experiments on image and language tasks show that HO-SFL trains nearly as fast as first-order methods, while reducing client memory usage close to inference-only levels and achieving lower communication cost. This makes privacy-preserving and personalized AI training more feasible on edge devices.