HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference

Edge-cloud collaborative inference is crucial for LLM-powered edge devices, as on-device models often lack the required reasoning capability, while cloud-only inference can be costly and slow under strict latency and token/API budgets. However, existing edge-cloud collaboration methods typically route input tasks based on their estimated difficulty. These static, coarse heuristics overlook subtask dependencies, missing opportunities for parallel execution and budget-adaptive routing. To this end, we propose HybridFlow, a resource-adaptive edge-cloud inference framework that enables parallel execution of interdependent subtasks. Specifically, we build a dependency-aware DAG for each input task, facilitating concurrent execution of subtasks once their dependencies are resolved, thereby reducing end-to-end latency. Additionally, we propose a dynamic benefit–cost utility model, optimizing the trade-off between accuracy, token/API cost, and latency in real-time. This dynamic routing minimizes unnecessary cloud usage while preserving reasoning quality. Across GPQA, MMLU-Pro, AIME24, and LiveBench-Reasoning, HybridFlow improves the cost-accuracy trade-off, reducing latency and cloud API usage while maintaining competitive accuracy.

Modern LLMs contain hundreds of billions of parameters and require specialized hardware that far exceeds what edge devices can provide, making local deployment practically infeasible. Relying entirely on cloud-based inference introduces prohibitive costs, high latency, and serious privacy risks. We introduce HybridFlow, a framework that decomposes complex requests into smaller, interdependent subtasks and routes each one dynamically to either the edge device or the cloud based on real-time budget, latency, and accuracy requirements. By modeling tasks as dependency graphs, HybridFlow executes independent subtasks in parallel, dramatically reducing end-to-end response time while minimizing expensive cloud API calls. Our experiments demonstrate that HybridFlow achieves better accuracy than edge-only baselines while consuming substantially fewer cloud resources than cloud-only approaches, across demanding benchmarks in scientific reasoning, mathematics, and general knowledge. This work advances a future where AI systems are inherently more efficient, cost-effective, and privacy-preserving, enabling broader access to powerful AI even in resource-constrained environments.