Foundation VAE for CT Reconstruction, Augmentation, and Generation

Variational autoencoders (VAEs) compress high resolution CT volumes into compact latents while preserving clinically relevant structure. However, training CT-specific VAEs from scratch or heavily fine-tuning them incurs substantial computational and engineering cost, and often degrades under heterogeneous scanners, protocols, and diseases. This paper makes a progressive stride toward training-free medical VAEs by leveraging a critical observation: a single Foundation VAE, pretrained at scale on natural images and videos, can serve as a unified interface for CT Reconstruction, Augmentation, and Generation. With both encoder and decoder frozen, the Foundation VAE reconstructs CT volumes with preserved anatomy while suppressing acquisition noise; training segmentation models on these reconstructions improves surface accuracy by 3.9\% NSD on average for pancreatic tumor and lung tumor. Within the same Foundation VAE latent space, a conditional latent diffusion model achieves 3.9\% lower average FVD with 36.2\% higher CT CLIP score, and improves multi-disease generation faithfulness across 18 types by 2.76\% AUC. These results demonstrate Foundation VAEs as a practical interface for scalable CT representation reuse and faithful CT generation. Our code and demo are available at https://github.com/qic999/Foundation-VAE.

Most CT reconstruction and generation systems rely on a Variational Autoencoder (VAE) to map high-resolution 3D scans into a compact latent space that downstream models work with. The usual recipe is to train a CT-specific VAE from scratch, or heavily fine-tune one on large medical datasets. This is expensive, takes a lot of engineering effort, and the resulting VAE often breaks down on scans from a different scanner, protocol, or disease. A natural question follows — do we actually need a medical-specific VAE at all? We show that a Foundation VAE pretrained on natural images and videos, with no medical fine-tuning, works directly as a shared interface for CT Reconstruction, Augmentation, and Generation. With this frozen Foundation VAE, we reconstruct CT volumes faithfully, use the reconstructions as extra training data to improve segmentation, and train a conditional diffusion model that generates realistic CT scans across multiple chest diseases. A general-purpose VAE can replace the costly CT-specific representation stage that most current pipelines depend on, lowering the barrier for both clinical AI development and large-scale medical data synthesis.