Beyond Model Base Retrieval: Weaving Knowledge to Master Fine-grained Neural Network Design

Designing high-performance neural networks for new tasks requires balancing optimization quality with search efficiency. Current methods fail to achieve this balance: neural architectural search is computationally expensive, while model retrieval often yields suboptimal static checkpoints. To resolve this dilemma, we model the performance gains induced by fine-grained architectural modifications as edit-effect evidence and build evidence graphs from prior tasks. By constructing a retrieval-augmented model refinement framework, our proposed M-DESIGN dynamically weaves historical evidence to discover near-optimal modification paths. M-DESIGN features an adaptive retrieval mechanism that quickly calibrates the evolving transferability of edit-effect evidence from different sources. To handle out-of-distribution shifts, we introduce predictive task planners that extrapolate gains from multi-hop evidence, thereby reducing reliance on an exhaustive repository. Based on our model knowledge base of 67,760 graph neural networks across 22 datasets, extensive experiments demonstrate that M-DESIGN consistently outperforms baselines, achieving the search-space best performance in 26 out of 33 cases under a strict budget.

Building a strong AI model for a new problem often takes many rounds of trial and error. Researchers try one design, evaluate it, adjust it, and repeat. This process can be expensive, and simply reusing a model design that worked before may fail because each new dataset has its own quirks. We propose M-DESIGN, a system that learns how small design changes affected model performance on past tasks and uses this experience to improve performance on new ones. When facing a new task, M-DESIGN uses this prior knowledge to choose a promising starting model and then makes a short sequence of targeted improvements. It also updates its choices as it gathers feedback, so it can avoid copying lessons from past tasks that turn out to be misleading. In tests across 33 tasks involving relational data, using our knowledge base of 67,760 models built for such data, M-DESIGN found the best model design in 26 cases with only a limited number of trials. This result suggests that future AI systems can be designed more efficiently by reusing detailed design knowledge, reducing wasted computation while improving model quality.