A Reasoning-Based Approach to Cryptic Crossword Clue Solving

Cryptic crossword clues are challenging language tasks for which new test sets are released daily by major newspapers on a global basis. Each cryptic clue contains both the definition of the answer to be placed in the crossword grid (in common with regular crosswords), and ‘wordplay’ that proves that the answer is correct (i.e. a human solver can be confident that an answer is correct without needing crossing words as confirmation). This work describes an LLM-based reasoning system built from open-licensed components that solves cryptic clues by (i) hypothesising answers; (ii) proposing wordplay explanations; and (iii) using a verifier system that operates on codified reasoning steps. Overall, this system establishes a new state-of-the-art performance on the challenging Cryptonite dataset of clues from The Times and The Telegraph newspapers in the UK. Because each proved solution is expressed in Python, interpretable wordplay reasoning for proven answers is available for inspection.

We're interested in a type of puzzle common in major newspapers (in the UK, and elsewhere) : Cryptic Crosswords. Each cryptic clue hints to its unique answer in two ways : a 'regular crossword definition' and 'wordplay'. Because the wordplay and definition must have the same answer, solvers know whether they've got the answer correct (even without having other answers in the crossword grid).Our work uses a large language model (LLM) to guess possible answers, and then justify the reasoning, finally delivering its solution in the Python programming language. By getting the reasoning as a small computer program, we can easily tell whether the LLM has got the reasoning correct, and this enables our method to beat ChatGPT and other models.Although we focused on Cryptic Crosswords, our ideas could also be applied to other linguistic tasks, opening them up to methods that are more commonly used for mathematics and programming problems.