Paper #24: Feedback-to-Rubrics: Can We Extract Expert Criteria from Inline Comments?

One factor that limits the usefulness of Large language models (LLMs) for human-AI co-creation in collaborative writing and review support is that they often lack context-dependent criteria, such as expert preferences or organization-specific conventions. Although providing LLMs with an explicit set of criteria, that is, a rubric, can align their outputs with expert judgments, these criteria are often tacit, undocumented, and hard to obtain directly from humans. We propose Feedback-to-Rubrics, a problem setting for learning criteria from inline comments on artifacts. Such comments are often accumulated within organizations or communities and can be readily leveraged. Our method infers rubrics from these comments and iteratively refines them by observing errors in comment prediction based on the inferred rubrics. In experiments on research proposal review, essay review, and medical chat annotation, our method more closely matches reference comments than LLMs without rubrics and retrieval-based baselines. In a setting with reference rubrics, the refined rubrics move closer to the reference rubrics than the initial ones.