Designing interactions with AI to support the scientific peer review process

Peer review processes include a series of activities from review writing to mata-review authoring. Recent advances in AI exhibit the potential to augment complex human writing activities. However, it is still not clear how to design interactive systems that leverage AI to support the scientific peer review process and what are the potential trade-offs. In this paper, we prototype a system – MetaWriter, which uses three forms of AI to support meta-review authoring and offers useful functionalities including review aspect highlights, viewpoint extraction, and hybrid draft generations. In a within-subjects experiment, 32 participants wrote meta-reviews using MetaWriter and a baseline environment with no machine support. We show that MetaWriter can expedite and improve the meta-review authoring process. But participants raised concerns about trust, over-reliance, and agency. We further discuss insights on designing interactions with AI to support the scientific peer review process.