Protein Autoregressive Modeling via Multiscale Structure Generation

We present protein autoregressive modeling (PAR), the first multi-scale autoregressive framework for protein backbone generation via coarse-to-fine next-scale prediction. Using the hierarchical nature of proteins, PAR generates structures that mimic sculpting a statue, forming a coarse topology and refining structural details over scales. To achieve this, PAR consists of three key components: (i) multi-scale downsampling operations that represent protein structures across multiple scales during training; (ii) an autoregressive transformer that encodes multi-scale information and produces conditional embeddings to guide structure generation; (iii) a flow-based backbone decoder that generates backbone atoms conditioned on these embeddings. Moreover, autoregressive models suffer from exposure bias, caused by the training and the generation procedure mismatch, and substantially degrades structure generation quality. We effectively alleviate this issue by adopting noisy context learning and scheduled sampling, enabling robust backbone generation. Notably, PAR exhibits strong zero-shot generalization, supporting flexible human-prompted conditional generation and motif scaffolding without requiring fine-tuning. On the unconditional generation benchmark, PAR effectively learns protein distributions and produces backbones of high design quality, and exhibits favorable scaling behavior. Together, these properties establish PAR as a promising framework for protein structure generation.

Designing new proteins with specific structures is a central challenge in modern biology, with important applications in medicine and nanotechnology. Many existing approaches generate proteins by directly modeling their three-dimensional structures, since structure determines function. In contrast, recent advances in language modeling have shown that autoregressive models, which generate content step by step based on previously generated context, can produce highly scalable results. However, this approach has been largely unexplored for protein structure generation. In this work, we introduce Protein Autoregressive Modeling (PAR), the first multi-scale autoregressive framework for generating protein structures. Instead of predicting the entire structure at once, PAR builds proteins progressively: it first generates a coarse overall topology and then refines structural details step by step, similar to how a sculptor gradually shapes a statue. Our experiments show that PAR generates high-quality protein structures efficiently, scales effectively with increased computational resources, and generalizes well to previously unseen tasks.