SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning

Large Language Model (LLM) applications increasingly execute as end-to-end inference pipelines that couple generation with retrieval, stateful memory, context refinement, and tool use under strict tail-latency and Service-Level Objective (SLO) constraints. Today, these stages are often stitched together as RPC-connected services, obscuring cross-stage queueing and interference and limiting pipeline-level compilation and resource sharing. We present SAGE (Streaming-Augmented Generative Execution), a full-stack system that treats inference pipelines as first-class compilation targets. SAGE exposes pipelines as declarative dataflows and compiles them into distributed execution plans with bounded-queue backpressure. It integrates vector search, streaming semantic state, structured memory, and refinement as operators with explicit resource/state contracts, enabling operator-level diagnosis of tail behavior. SAGE integrates pluggable generation and embedding backends and provides a unified control plane for engine management, batching, and admission under mixed workloads. On a 16-node cluster, SAGE sustains 16 requests/s at $>700$ tokens/request with 1 ms median scheduling overhead, and achieves near-linear scale-out to 16 nodes (11.4$\times$ throughput at 16 nodes), and reduces P99 latency by 57\% under multi-pipeline contention versus simultaneous admission.

Modern applications built with large language models often do much more than call a single model. They may retrieve documents, maintain memory, refine context, invoke tools, and generate responses through several connected stages. When these stages are assembled as separate services, it becomes difficult to understand where delays come from, how resources are shared, and why the final system becomes slow or unstable. This paper presents SAGE, a system for building and running such multi-stage LLM applications as explicit dataflow pipelines. In SAGE, each stage is represented as an operator with declared resource needs, state behavior, and input-output behavior. This allows the system to compile the full pipeline into an execution plan, monitor individual stages, and make scheduling or admission-control decisions at the pipeline level rather than treating each service in isolation. The goal of SAGE is to make LLM applications more modular, controllable, and transparent. For developers and researchers, this can make it easier to diagnose bottlenecks, compare execution policies, and manage complex applications such as retrieval-augmented generation systems. The paper evaluates SAGE on multi-stage workloads and shows that pipeline-level control can improve throughput and tail latency under resource contention.