Timezone: »

 
Poster
Coder Reviewer Reranking for Code Generation
Tianyi Zhang · Tao Yu · Tatsunori Hashimoto · Mike Lewis · Scott Yih · Daniel Fried · Sida Wang

Thu Jul 27 04:30 PM -- 06:00 PM (PDT) @ Exhibit Hall 1 #627
in Poster Session 6 »

Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions. Inspired by collaborative programming, we propose Coder-Reviewer reranking. We augment Coder language models from past work, which generate programs given language instructions, with Reviewer models, which evaluate the likelihood of the instruction given the generated programs. We perform an extensive study across six datasets with eight models from three model families. Experimental results show that Coder-Reviewer reranking leads to consistent and significant improvement (up to 17% absolute accuracy gain) over reranking with the Coder model only. When combined with executability filtering, Coder-Reviewer reranking can often outperform the minimum Bayes risk method. Coder-Reviewer reranking is easy to implement by prompting, can generalize to different programming languages, and works well with off-the-shelf hyperparameters.

[ PDF] 

Author Information

Tianyi Zhang (Stanford University)
Tao Yu (The University of Hong Kong)
Tatsunori Hashimoto (Stanford)
Mike Lewis (Facebook)
Scott Yih (Meta AI - FAIR)
Daniel Fried (Carnegie Mellon University)
Sida Wang (Meta AI)

More from the Same Authors