LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as *catastrophic overtraining* and *quantization-induced degradation*, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon–Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong $R^2$ scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.

Large language models (LLMs) have grown larger every year on the assumption that bigger is always better, yet recent experiments reveal a puzzling reversal. When models are compressed, fine-tuned, or simply trained for too long, their performance can actually get worse. Existing scaling laws, the formulas used to plan multi-million-dollar training runs, cannot explain this U-shaped behaviour. We borrow a classical idea from communication engineering and treat the LLM as a noisy radio channel that carries information from input text to output text. Shannon's 1948 theorem tells us how much information such a channel can carry, given its bandwidth, signal strength, and noise. We map model size to bandwidth, training data to signal, and training imperfections to noise, and obtain a single formula that captures both healthy improvement and the breakdown when noise dominates. The formula fits experimental results across many models, datasets, and perturbations more accurately than previous laws. Critically, it also extrapolates. Trained on small models and short runs, it correctly predicts the behaviour of a model nearly twice as large on twice as many tokens, while older laws fail. This could help researchers decide when not to scale, saving compute, energy, and money.