Abstract: Heuristics are indispensable for tackling complex search and optimization problems. However, manual heuristic design is tedious and demands significant human intuition and experience. This paper introduces Evolution of Heuristic (EoH), a novel paradigm that leverages the synergy between Large Language Models (LLMs) and Evolutionary Computation (EC) for Automatic Heuristic Design (AHD). EoH represents heuristic ideas through linguistic descriptions, termed \emph{thoughts}, generated by LLMs, which are then translated into executable \emph{code} representations. The coevolution of thoughts and codes within an evolutionary framework offers superior AHD performance while mitigating computational expenses. Comprehensive evaluations on three types of combinatorial optimization benchmarks demonstrate EoH's outperformance against existing AHD methods. Notably, EoH surpasses FunSearch, identifying superior heuristics with $1000\times$ fewer computational budgets (i.e., queries to LLMs) on online bin packing problem.