Fast Byte Latent Transformer

Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte autoregressive generation. We address this bottleneck in the Byte Latent Transformer (BLT) through new training and generation techniques. First, we introduce BLT Diffusion (BLT-D), a new model and our fastest BLT variant, trained with an auxiliary block-wise diffusion objective alongside the standard next-byte prediction loss. This enables an inference procedure that generates multiple bytes in parallel per decoding step, substantially reducing the number of forward passes required to generate a sequence. Second, we propose two extensions inspired by speculative decoding that trade some of this speed for higher generation quality: BLT Self-speculation (BLT-S), in which BLT's local decoder continues generating past its normal patch boundaries to draft bytes, which are then verified with a single full-model forward pass; and BLT Diffusion+Verification (BLT-DV), which augments BLT-D with an autoregressive verification step after diffusion-based generation. All methods may achieve an estimated memory-bandwidth cost over 50\% lower than BLT on generation tasks. Each approach offers its own unique advantages, together removing key barriers to the practical use of byte-level LMs.

Most AI language models read and write text in chunks of about four letters, called tokens. A newer approach works one letter at a time, which makes models far better at handling typos, rare and underserved languages, and unusual text like code. But this robustness comes at a cost: writing anything takes roughly four times as many steps, making these models too slow to be practical. We set out to fix that slowness in one such model, the Byte Latent Transformer (BLT). Our main idea, BLT Diffusion, lets the model write several characters at once instead of one after another, much like filling in many blanks in a sentence simultaneously rather than left to right. This can cut the work of generating text by more than half. Because writing in parallel can introduce errors, we also designed extensions that serve as safeguards for model generation quality: each lets the model quickly draft a guess and then double-check that guess against itself, keeping most of the speed while maintaining quality. Together, these techniques make character-level AI faster. That matters because it lets AI more reliably and fairly handle messy, diverse text across many languages and formats.