Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance

Real-world datasets across image and text domains are often characterized by skewed class distributions and noisy annotations, which jointly degrade model performance, particularly on minority classes. Among existing solutions, active learning offers an effective and efficient paradigm by selectively querying the most informative and balanced samples for annotation. We propose an innovative active learning framework that mitigates class imbalance and selects the most informative samples to annotate. Leveraging foundation model priors, our algorithm enables imbalance-aware co-decisions between foundation model and small model to tackle noisy and imbalanced labels across various domains. We introduce the first study to systematically explore active learning under the dual challenges of label noise and class imbalance across image and text domains. Extensive experiments on imbalanced datasets demonstrate that our method achieves substantial annotation savings—over 50\% compared to the best active learning baseline—while preserving performance and robustness to label noise.

Real-world AI systems learn from large collections of image and text data. However, these datasets are often messy: some categories appear far more often than others, and some labels may be incorrect. As a result, AI models may make poor decisions, especially on rare or underrepresented cases. Active learning helps address this problem by selecting only the most useful examples for humans to label, instead of labeling everything. Our method uses a large pre-trained AI model together with a smaller task-specific model to generate reliable labels and decide which examples should be labeled by humans. This helps the model learn better from imbalanced and noisy data across both image and text tasks. In our experiments, our method reduces human labeling by more than 50% compared with strong existing active learning methods, while keeping the model accurate and robust.