Learning from others' mistakes: Finetuning machine translation models with span-level error annotations

Despite growing interest in incorporating feedback to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of utilizing fine-grained span-level annotations from offline datasets to improve model quality. We develop a simple finetuning algorithm, called Training with Annotations (TWA), to directly train machine translation models on such annotated data. TWA utilizes targeted span-level error information while also flexibly learning what to penalize within a span. Moreover, TWA considers the overall trajectory of a sequence when deciding which non-error spans to utilize as positive signals. Experiments on English-German and Chinese-English machine translation show that TWA outperforms baselines such as supervised finetuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data.

Language models have gotten so good that it can the difficult to improve them using the standard practice of collecting high-quality examples to train models to emulate. Instead, researchers are increasingly turning towards improving models using various forms of feedback, such as whether text is good or bad, or which option is better within a pair. However, most current methods only use broad feedback—like rating an entire sentence. But what if we could use more detailed feedback that highlights exactly which parts of a sentence are wrong? In our work, we introduce a simple training method called Training with Annotations (TWA) that helps machine translation models learn directly from this kind of precise, span-level feedback. TWA takes advantage of these detailed notes by learning from both the mistakes and correct parts of a sentence. When we tested TWA on English-German and Chinese-English translations, it outperformed existing techniques that rely on broader feedback. This shows that training models with targeted feedback can make them more accurate and effective.