ACO-MoE-LoRA: Evolving-while-Training for Adapting Segment Anything Model 2 to Specialized Domains

Static fine-tuning paradigms impose rigid structural constraints on foundation models like the Segment Anything Model 2 (SAM2), limiting their adaptability to the varying complexity of specialized downstream tasks. To overcome this limitation, we propose ACO-MoE-LoRA, a dynamic framework that introduces an "Evolving-while-Training" strategy by synergizing Ant Colony Optimization (ACO) with a Latent Space Mixture-of-Experts (MoE) architecture. Central to our method is the ACO-ConvLoRA module, which employs a pheromone-guided routing mechanism to actively govern expert selection and topological evolution. By formulating expert assignment as an evolutionary pathfinding problem, this module effectively mitigates the standard routing collapse issue and enables elastic adjustment of LoRA ranks via weight slicing, bridging discrete structural search with continuous parameter training. Extensive experiments across 16 challenging datasets demonstrate that our framework consistently outperforms leading static adapters, while effectively addressing the local optimality limitations of recent dynamic heuristics. This work presents a self-organizing solution that harmonizes swarm intelligence with gradient optimization for efficiently adapting foundation models to specialized domains.

A powerful computer vision system can cut out any object from any image, but it was mostly trained on everyday photos. When doctors try to spot polyps in colonoscopy images or marine biologists track fish in murky water, the system often fails because the images look too different. We created a solution that teaches this model to adapt to new, challenging domains without rebuilding it. Instead of manually adjusting dozens of settings, our method uses a nature‑inspired “ant colony” algorithm: each possible setting is like a path, and artificial ants leave digital pheromone trails to guide the search. The system also learns to mix lightweight “expert” sub‑networks on the fly, focusing only on the most relevant ones—like a project manager assigning the right people to a task. Our approach works across 16 datasets—from medical scans to underwater videos—consistently outperforming existing methods while using fewer parameters. This helps researchers and practitioners repurpose powerful foundation models for specialized applications, from cancer detection to wildlife conservation.