Peer review processes include a series of activities from review writing to mata-review authoring. Recent advances in AI exhibit the potential to augment complex human writing activities. However, it is still not clear how to design interactive systems that leverage AI to support the scientific peer review process and what are the potential trade-offs. In this paper, we prototype a system – MetaWriter, which uses three forms of AI to support meta-review authoring and offers useful functionalities including review aspect highlights, viewpoint extraction, and hybrid draft generations. In a within-subjects experiment, 32 participants wrote meta-reviews using MetaWriter and a baseline environment with no machine support. We show that MetaWriter can expedite and improve the meta-review authoring process. But participants raised concerns about trust, over-reliance, and agency. We further discuss insights on designing interactions with AI to support the scientific peer review process.
Lu Sun (UC San Diego)
Stone Tao (University of California, San Diego)
Junjie Hu (University of Wisconsin, Madison)
Steven Dow (Carnegie-Mellon University)
More from the Same Authors
2023 Poster: Abstract-to-Executable Trajectory Translation for One-Shot Task Generalization »
Stone Tao · Xiaochen Li · Tongzhou Mu · Zhiao Huang · Yuzhe Qin · Hao Su