LLM Planning Success

Recent literature has shown that LMs cannot reliably plan, especially in comparison to classical planners and planning languages such as PDDL. At the same time, other work has shown that LMs are Turing complete, functionally equivalent to the computers on which they run. In this talk we present our study of this gap. We show how we are able to train LMs and achieve 99% or higher success rates across a range of planning problems such as Towers of Hanoi and Blocksworld variants on which other approaches have failed to scale. Our analysis shows where LMs are able to generalize and some cases where they are not.