Compressed Semantic Retrieval as External Memory for Scalable AI Systems

Agentic AI systems increasingly treat retrieval as external memory, making storage footprint, index-refresh cost, and query latency first-class scaling constraints. However, standard dense retrieval pipelines decompress corpora into plain text before chunking, embedding, and indexing, inflating storage and build cost whenever memory must be refreshed under tight budgets. In this paper, we present ***compressed-domain retrieval***, in which the storage, indexing, and retrieval units coincide at the compressed chunk. The design pairs **ZPKG**, a novel Zstandard-based random-access packaging format, with **CEM**, a byte-level Compressed Embedding Model trained under a joint objective to map compressed chunks into the semantic retrieval space while keeping queries in plain text. Decompression is deferred until after candidate selection, preserving compatibility with standard RAG pipelines. On BEIR, the method reduces corpus storage to 30-33\%, cuts indexed vectors by 7.5-10.4$\times$, and accelerates HNSW index construction by 15.1-66.9$\times$. On an MTEB-based suite, CEM remains competitive with strong text encoders and surpasses all compared baselines on WinoGrande and TempReasonL1, positioning compressed-domain retrieval as a practical external-memory substrate for scaling agentic systems under memory, refresh, and latency budgets.