Poster
in
Workshop: Next Generation of AI Safety
Is ChatGPT Transforming Academics' Writing Style?
Mingmeng Geng · Roberto Trotta
Keywords: [ statistical analysis ] [ ChatGPT usage ] [ word frequency ] [ academic writing ] [ Large Language Models (LLMs) ]
Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT's writing style in their abstracts by means of a statistical analysis of word frequency changes. Our model is calibrated and validated on a mixture of real abstracts and ChatGPT-modified abstracts (simulated data) after a careful noise analysis. We find that ChatGPT is having an increasing impact on arXiv abstracts, especially in the field of computer science, where the fraction of ChatGPT-revised abstracts is estimated to be approximately 35%, if we take the output of one of the simplest prompts, "revise the following sentences", as a baseline. We conclude with an analysis of both positive and negative aspects of the penetration of ChatGPT into academics' writing style.