PRM-PBE: Process Reward Model for Reinforcement Learning in Programming-by-Example

Programming-by-Example (PBE), as a typical few-shot inductive reasoning paradigm, aims to synthesize corresponding algorithms from a set of input-output examples. Although Large Language Models (LLMs) have demonstrated strong program synthesis potential, they still remain ineffective when handling complex PBE tasks. Specifically, LLMs often struggle to accurately grasp the underlying intent of examples, resulting in synthesized programs that either partially satisfy the examples or completely deviate from the target. To address these limitations, we introduce a process-supervised reinforcement learning method that provides fine-grained feedback during the synthesis process, improving the ability of LLMs to capture the intended behavior of provided examples. Firstly, we develop a reasoning tree construction method that is used to build a PBE process supervision dataset. Subsequently, we train a process reward model through preference learning to evaluate the effectiveness of reasoning steps. Finally, we introduce a curriculum learning strategy based on the difficulty of PBE tasks, using Proximal Policy Optimization (PPO) to optimize the model. Experimental results on representative PBE benchmarks show that our approach achieves an average pass rate of 56.61\%, significantly outperforming the state-of-the-art baseline by 8.73\%.

Imagine giving a computer just a handful of input-output examples, like "1 becomes 2" and "5 becomes 10", and asking it to figure out the underlying rule and write a small program that reproduces it. This task, known as Programming-by-Example, is a basic form of learning from examples, but today's AI coding assistants still often get it wrong on harder cases, producing programs that only match some of the examples or miss the intended rule entirely. In this work, we train the AI to think one step at a time and give it feedback after every step, instead of only judging the final program. We first map out the different reasoning paths the AI might take, then train a separate "coach" model that learns to score how promising each intermediate step is. Using these step-by-step scores, the AI is gradually improved through practice, starting from easier problems and moving on to harder ones. On standard Programming-by-Example benchmarks, our method writes correct programs noticeably more often than the strongest existing approaches. More broadly, the results suggest that giving AI fine-grained feedback during its reasoning, rather than only grading the final answer, is a promising way to make it more reliable on tasks that require multiple steps of thinking.