Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs

Deploying retrieval-augmented generation (RAG) pipelines on commodity accelerators such as the NVIDIA T4 (16 GB VRAM) exposes a critical failure mode we term the Compression Paradox: neural prompt compressors introduce substantial latency overhead and VRAM contention that frequently exceed the generation time saved. We observe two mechanistically distinct failure modes. First, an Always-Raw pipeline — serving Llama-3-8B-Instruct-AWQ (Lin et al., 2024) via vLLM (Kwon et al., 2023) with no compressor — crashes on 65% of long inputs due to standard KV-cache exhaustion within vLLM's pre-allocated memory pool. Second, an Always-Neural pipeline — adding LLMLingua-2 as a co-resident PyTorch process — triggers fatal cross-process VRAM contention: vLLM's rigid 55% reservation starves the PyTorch encoder's dynamic allocator, producing CUDA OOM on long inputs even when the LLM's KV-cache pool is not itself full. Because Llama-3-8B-Instruct-AWQ alone occupies 9 GB, only ~7 GB remains for the KV cache and any auxiliary model. We propose the Tri-Metric Router, a deterministic, training-free dispatch policy — analytic and auxiliary-model-free at dispatch time, with no learned components and no auxiliary model training — that selects among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines using three CPU-bound heuristics: spatial complexity L, syntactic density ρ_key, and Type-Token Ratio (TTR). The adaptation signal is hardware-physical (VRAM headroom, latency crossover) rather than semantic — an underexplored axis of efficient inference on commodity accelerators. Unlike prior adaptive-RAG work (Asai et al., 2024; Jeong et al., 2024) that adapts retrieval decisions, our router addresses a downstream hardware-constraint problem: which compression mechanism can be safely applied post-retrieval on a constrained GPU without OOM. Routing thresholds are derived from a 100-document profiling sweep on LongBench (qasper) (Bai et al., 2024) (crossover threshold from N = 21 Long-band documents), isolating a hardware-specific latency crossover at L* ≈ 4,332 words. This threshold is derived from an N = 21 profiling subsample and is specific to our T4 deployment; cross-hardware re-calibration is required. The router achieves a 0% OOM rate on the evaluated workload; deterministic VRAM safety is enforced only on the Long-band branch via BM25 with a hard 4,096-token truncation, while the Raw and Neural branches retain OOM-safety as an empirical claim on in-distribution inputs at B = 1. On out-of-distribution holdouts, the Tri-Metric router achieves 88.5% oracle alignment with the optimal pipeline — the headline operating metric — while the 99.0% figure on the in-distribution calibration set represents an upper bound of the threshold fit, not an operating result. The router further preserves a statistically stable 49.3% Combined F1 — establishing a rigorous practical operating point without auxiliary model training or GPU memory overhead. Scope: Evaluated on a single T4, Llama-3-8B-Instruct-AWQ, LLMLingua-2, batch size B = 1, and LongBench English QA; we position this as a reference calibration for the methodology, not a universal result.