Hierarchical Representations for Cross-task Automated Heuristic Design using LLMs

Designing heuristic algorithms for complex optimization problems is a time-consuming and expert-driven process. Recently, Automated Heuristic Design (AHD) using Large Language Models (LLMs) has shown significant promise for automating algorithm development. However, existing works mainly rely on programs to represent heuristics, which are inherently taskspecific and fail to generalize as effectively as established metaheuristics like tabu search or guided local search. To bridge this gap, we introduce Multi-Task Hierarchical Search (MTHS), an LLM-guided evolutionary method that co-designs general-purpose metaheuristics and task-specific programs. MTHS employs a hierarchical representation and adopts a two-level evolution framework to evolve task-agnostic metaheuristics and task-specific program implementations simultaneously across multiple heuristic design tasks. During this evolution, a knowledge transfer mechanism allows learning from elite programs designed for other tasks. We evaluated MTHS on distinct combinatorial optimization problems, where it outperforms both commonly-used heuristics and existing LLM-driven AHD approaches. Our results demonstrate that the hierarchical representations facilitate effective multi-task AHD, and the evolved metaheuristics exhibit strong generalization to related tasks. Code: https://github.com/FeiLiu36/MTHS.

Designing good algorithms for difficult optimization problems, such as routing delivery vehicles or scheduling factory jobs, usually requires years of human expertise and extensive trial and error. Recent work has shown that large language models (LLMs) can help automate this process, but existing methods typically create solutions that only work well for a single problem at a time. We introduce a new framework called Multi-Task Hierarchical Search (MTHS), which allows LLMs to design more general problem-solving strategies that can be reused across different tasks. Instead of directly generating one fixed algorithm, our method separates high-level problem-solving ideas from task-specific implementation details. It also allows successful strategies discovered for one task to help improve solutions for other tasks. We tested our approach on several optimization problems, including route planning, scheduling, and packing tasks. The algorithms created by MTHS consistently outperformed strong existing methods and other LLM-based design systems. We also found that the learned strategies could generalize to new problems and even help different AI models create stronger solutions. These results suggest that AI systems may eventually be able to develop reusable algorithmic knowledge rather than solving every new problem from scratch.