CoCoQuant: Breaking the Bandwidth Wall via Co-Optimized Communication and Computation Quantization

The rapid scaling of large language models (LLMs) has made distributed inference indispensable, yet end-to-end latency is increasingly dominated by communication, forming a critical bandwidth wall that fundamentally limits the practical gains of existing quantization techniques. Existing approaches typically treat communication and computation in isolation, failing to exploit their coupled nature and introducing limited system-level acceleration and accuracy degradation. To address this, we propose CoCoQuant, a co-designed framework that jointly optimizes communication and computation as a unified end-to-end design space. CoCoQuant introduces a precision-aligned graph-rewriting that enables zero-overhead fusion between low-precision communication and computation. CoCoQuant formulates a hardware-aware mixed-precision allocation problem that integrates roofline-based cost modeling with relative sensitivity calibration, solved via global integer linear programming. Extensive experiments on LLMs of varing scales demonstrate that CoCoQuant achieves Pareto-optimal accuracy-latency trade-offs, delivering up to 2.92 end-to-end speedup with a negligible increase in perplexity (0.22).

Large language models are now so big that they often have to run across several GPUs at once. Even when each GPU can do calculations very fast, the GPUs must constantly send partial results to one another, and this data movement can become the main source of delay. A common way to speed models up is quantization, which stores and computes numbers with fewer bits, but existing methods usually compress computation and GPU-to-GPU communication separately. This separation can waste time converting data back and forth between formats, and it can also hurt model quality. We introduce CoCoQuant, a system that plans these two kinds of compression together. It rewrites the model’s execution graph so that low-precision data can flow smoothly from communication into computation, then automatically chooses where to use more or fewer bits based on both the target hardware and each model component’s sensitivity to compression. It also uses fused GPU kernels to avoid unnecessary memory movement. Across several large language models, including dense and mixture-of-experts models, CoCoQuant speeds up distributed inference by as much as 2.92x while keeping accuracy nearly unchanged.