Few-Shot Biomedical Image Classification by Alignment of Independently Pretrained Encoders

Few-shot biomedical classification is dominated by methods built on jointly trained vision-language models such as BiomedCLIP, which rely on paired image-report corpora that are costly to assemble. In parallel, strong unimodal biomedical encoders, such as the self-supervised vision model RAD-DINO, are trained independently of text. We introduce a framework that aligns independently pretrained vision and text encoders using only the few-shot support set, combining a closed-form orthogonal Procrustes map with a lightweight flow-matching refinement. On the 11-dataset BiomedCoOp benchmark, replacing the jointly trained biomedical vision encoder with a general-purpose DINOv3 encoder matches or surpasses BiomedCoOp at higher shot counts. On VinDr-CXR multi-label chest X-ray classification, aligning RAD-DINO with BiomedCLIP text consistently outperforms linear probing of RAD-DINO across all shot counts. These results show that post-hoc alignment can reduce dependence on paired multimodal pretraining while enabling flexible combinations of independently trained unimodal models.