HexGen-3: A Fully Disaggregated LLM Serving Framework with Fine-Grained Heterogeneous Resource Autoscaling

The operational cost of serving large language models remains prohibitively high, largely due to extreme workload heterogeneity in production traffic. We observe that combining disaggregated inference with resource autoscaling enables fine-grained resource adjustment, allowing inference phases and operations to scale independently based on their specific bottlenecks. Building on this insight, we propose HexGen-3, a cost-effective LLM serving framework that leverages a fully disaggregated inference architecture and heterogeneous resource autoscaling. HexGen-3 introduces two key components: (i) A hierarchical scheduling framework that jointly optimizes resource allocation and parallelism configuration for any given resource provisioning, and (ii) an autoscaling framework that dynamically adjusts resources and triggers deployment rescheduling in response to workload fluctuations. Experiments comparing HexGen-3 against state-of-the-art LLM serving systems demonstrate up to 60% (on average 46.5%) improvement in per-cost throughput under static resource provisioning, and up to 78.3% (on average 55.1%) improvement with autoscaling enabled under dynamic workloads.

Large language models power many online services, but serving them is expensive because user requests can vary greatly. Some requests are short, while others involve long prompts or long outputs. These different requests stress different parts of the system: some require more computation, while others require more memory. When the workload changes over time, using a fixed set of GPUs can waste resources or slow down the service. This paper introduces HexGen-3, a system that makes large language model serving more cost-efficient by splitting the serving process into specialized parts. Instead of running all parts of the model in the same way, HexGen-3 separates prefill, attention, and FFN workers, so each part can use the most suitable GPU resources and parallelism strategy. It also automatically scales these different workers independently when the workload changes, adding resources where bottlenecks appear and reducing resources when demand drops. Experiments show that HexGen-3 significantly improves cost-efficiency compared with existing serving systems. Under fixed resources, it improves per-cost throughput by up to 60%, with an average improvement of 46.5%. Under dynamic workloads with autoscaling enabled, it improves performance by up to 78.3%, with an average improvement of 55.1%.