Patch size and its effect on representations in MAEs

Medical images contain a wealth of information that typically spans multiple spatial scales. While Masked Autoencoders (MAEs) are extensively used for self-supervised representation learning in these domains, they traditionally rely on a fixed patch size. In this work, we demonstrate that patch size inherently acts as a spatial filter, dictating the granularity of encoded features and causing models to struggle when representing objects smaller than the patch itself. To address this limitation and avoid the computational burden of training separate models for different resolutions, we introduce Mosaic MAE, an architecture trained dynamically across varying patch sizes and explicitly conditioned on patch scale via adaptive LayerNorm. Through experiments on a multi-scale MNIST-on-ImageNet dataset and lateral DXA bone scans, we demonstrate that Mosaic MAE offers improved representation learning relative to FlexiMAE—a self-supervised variant of FlexiViT. By decoupling feature learning from a single fixed resolution, Mosaic MAE enables the extraction of both fine-grained details and coarse semantic structures from a single pretrained model