SoftBinary Coding: A New Information-Theoretic Paradigm for Neural Compression via Fast Channel Simulation

Neural compression is currently dominated by Nonlinear Transform Coding (NTC), which maps data to real-valued latents via continuous transforms. Despite its success, NTC suffers from train-test mismatch due to non-differentiable quantization, a "smoothness bias" inherent in continuous transforms that precludes optimality for certain sources, and a loss of "shaping gain" due to its use of scalar quantization. We propose SoftBinary Coding (SBC), an end-to-end learning paradigm that bypasses these limitations by using a stochastic binary latent space. In the spirit of vector quantization, SBC employs discrete representations and compresses them through a novel fast binary channel simulation scheme, for which we provide a proof of rate optimality. Experimental gains on information-theoretic sources address NTC's limitations both theoretically and practically, establishing discrete binary structures as a viable path toward reaching optimal rate–distortion bounds. Surprisingly, SBC also achieves state-of-the-art performance on vector quantization of i.i.d. sources, exceeding Trellis Coded Quantization of the Gaussian source.

Today's dominant approach to neural compression, Nonlinear Transform Coding (NTC), follows a three-step recipe: transform data using a neural network into a compact continuous representation, quantize (round) it to integers, and compress this discrete representation with entropy coding, which uses knowledge of probabilities to minimize the expected length of the output bit sequence. Discretization, however, is fundamentally incompatible with stochastic gradient descent, so NTC uses biased gradient estimates. In addition, the smoothness of the continous transformation limits the optimality of the discrete representation in some cases, leading to degraded compression performance. It has been hypothesized that efficient algorithms may exist that avoid biased gradient estimates, or even quantization altogether. Here, we propose SoftBinary Coding (SBC), one of the first of this class of algorithms. It represents data using a binary probabilistic representation rather than a deterministic integer one. Instead of entropy coding, we use channel simulation, a paradigm for transmitting stochastic data efficiently. We repurpose a fast algorithm originally developed for error-correcting codes in communications, and prove the resulting compression rate is asymptotically optimal. These techniques address NTC's limitations: We avoid biased gradient estimates, as we use an unbiased estimator applicable to stochastic representations. We avoid the smoothness bias of continuous transforms, as the mapping to binary representations is more flexible. SBC also turns out to outperform Trellis Coded Quantization — the gold standard for compressing Gaussian data since the 1990s. This is surprising, as this would traditionally require vector quantization. Apparently, our channel simulation method can achieve similar gains without it. Our results suggest this type of model could be a more expressive foundation for learned data compression going forward.