HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

The proliferation of Large Language Models (LLMs) has shifted serving systems from processing isolated requests to orchestrating high-concurrency, multi-tenant agentic workflows. However, existing solutions typically prioritize intra-workflow optimization, largely neglecting the significant potential for inter-workflow optimization. In this paper, we propose HeraSys, an LLM serving system designed to optimize the end-to-end performance of concurrent workflows. Through fine-grained orchestration, HeraSys eliminates cross-workflow computational redundancy via structural node merging and reuse. Furthermore, HeraSys introduces a load-aware joint scheduling policy that dynamically manages execution order by evaluating both inter- and intra-query priorities. By integrating a resource skewing mechanism with adaptive batching and pipeline decomposition, HeraSys effectively mitigates tail latency while maintaining low average latency, thereby substantially improving system throughput. Extensive experiments demonstrate that HeraSys reduces P99 latency by up to 2.17$\times$ and increases serving throughput by up to 1.85$\times$ under strict latency guarantees.

LLM-based applications often answer user requests through several connected steps, such as retrieving documents, using external tools, and generating a final response. When many requests run at the same time, current serving systems usually handle them separately, which can repeat shared work and cause some requests to wait much longer than others. HeraSys is a serving system that coordinates many active workflows together instead of treating them one by one. It breaks workflows into smaller steps, reuses steps that are shared across requests, and schedules the remaining work to balance quick responses with steady progress for longer requests. Experiments on retrieval, web search, and mixed workloads show that HeraSys reduces long waits and helps the system serve more requests under the same latency requirement. This can make LLM applications faster, fairer, and more efficient to run.