Proxy Compression for Language Modeling

Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF‑8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, one language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs which are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or rival tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling.

Modern language models usually read text after it has been broken into shortened pieces (tokens) by a separate tool. This makes training much faster, but it also ties the model to that tool forever, which can make the system less flexible and sometimes fragile when small changes in spelling, spacing, or formatting alter how the text is broken up. We introduce a training method called proxy compression. During training, the model sees mostly shortened versions of text, which keeps training efficient, but it also sees some original text so that it learns how the shortened and original forms relate to each other. After training, the shortening tool can be removed, and the model can work directly with the original text. In experiments on code generation, this approach trains much more efficiently than models that only read original text. As the model becomes larger, it can match or even outperform standard models that always rely on a fixed text-breaking tool, while remaining more robust to harmless formatting changes. This suggests a practical path toward language models that are both efficient to train and less dependent on hand-designed text-processing rules.