Using the Mixture-of-Experts model and DeepSeek to Learn with Noisy Labels

Noisy labels in training data occur inevitably. Some emerging studies leverage the memorization performance, which can be represented by the confidence scores and becomes larger during training, to prevent deep neural networks from fitting label noise on visual data. However, existing text-based noise handling approaches have insufficiently explored the exploitation of memory capacity for noisy label learning. To fill this gap, we propose a mixture-of-experts-based sample selection and DeepSeek-assisted method (MoE-SD) for learning from noisy labels. Specifically, we first use a two-expert-layer-based framework for text classification, obtaining confidence scores for each text from two experts. Then we propose a sample-based dynamic and static threshold strategy for each sample, based on the momentum of each sample's memorization performance in previous epochs and loss distributions to select noisy labeled data. After that, we explore the utilization of DeepSeek to correct the noisy labeled data. Benefiting from the dynamic and static threshold strategy and two-expert co-learning, we can group each sample into one of the three subsets (\emph{i.e.}, reliable, hard, and purified). Finally, we employ different training objectives to learn these subsets. Experimental results on three text classification benchmarks under various noise settings show that our proposed method outperforms these strong baselines under different noise ratios.