PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding

Speculative decoding can significantly accelerate LLM inference, especially given that its cloud-edge collaborative deployment offers cloud workload offloading, offline robustness, and privacy enhancement. However, existing collaborative inference frameworks with speculative decoding are constrained by (i) sequential token generation and communication with low resource utilization, and (ii) inflexible cloud non-autoregressive verification (NAV) triggering that induces premature verification or costly rollbacks. In this paper, we propose PipeSD, an efficient cloud-edge collaborative pipeline inference framework with speculative decoding. PipeSD overlaps token generation and communication by a token-batch pipeline scheduling mechanism optimized by dynamic programming, and improves verification flexibility through a dual-threshold NAV triggering mechanism with a lightweight Bayesian optimization autotuner. We implement PipeSD using llama-cpp-python, PyTorch, and FastAPI, and evaluate it on a real-world cloud-edge testbed with two draft-target model pairs across four scenarios. Results show that PipeSD consistently outperforms state-of-the-art baselines, achieving $1.16\times$-$2.16\times$ speedup and reducing energy consumption by $14.3$%-$25.3$%. Our code is available at https://github.com/Ghanyunhe/PipeSD.

Large language models (LLMs) are often slow because they generate text one word at a time. A recent technique called speculative decoding tries to speed this up by letting a smaller, faster AI model predict several words in advance, while a larger and more accurate model checks these predictions all at once. Recent cloud-edge systems apply this idea by running the smaller model on edge devices such as laptops or phones, while using powerful cloud servers for verification. However, existing methods still waste time because text generation, communication, and verification are handled sequentially, and verification decisions are often inefficient. We propose PipeSD, a system that improves the efficiency of clou-edge collaborative inference. PipeSD improves communication efficiency by overlapping text generation and transmission, reducing unnecessary waiting and lowering inference latency. It also more intelligently decides when generated text should be verified to avoid unnecessary verification and delays. By reducing delays and improving efficiency, PipeSD could help make powerful AI systems more practical for real-world use on personal and edge devices.