ChatGPT has had an “unprecedented” impact on scientific writing, leading to a marked increase in “flowery” language, a new paper has found.
To determine the extent of usage of large language models (LLMs), researchers from the University of Tübingen and Northwestern University analysed more than 15 million biomedical abstracts from the PubMed library.
Their paper compared results from before and after the launch of ChatGPT in November 2022 and discovered that LLMs have resulted in certain words featuring much more regularly.
These were predominantly verbs, such as “delves”, “underscores” and “showcasing”, which all had much higher increases in usage compared with previous years.
Previously this so-called excess vocabulary had mainly been seen in content words. For example, during the Covid-19 pandemic, nouns such as “respiratory” or “remdesivir” appeared in studies much more regularly.
One example highlighted of “flowery language” from a 2023 study said: “By meticulously delving into the intricate web connecting […] and […], this comprehensive chapter takes a deep dive into their involvement as significant risk factors for […].”
The results, published in the journal Science Advances, also show that changes as a result of LLMs resulted in abrupt changes in both the quality and quantity of research papers. And the analysis suggested that at least 13.5 per cent of abstracts published last year were processed with LLMs – about 200,000 papers in total.
“We show that LLMs have had an unprecedented impact on scientific writing in biomedical research, surpassing the effect of major world events such as the Covid pandemic,” said Ágnes Horvát, co-author of the study and a professor at Northwestern’s School of Communication.
The paper, which did not use LLMs for any writing or editing, said that the impact of ChatGPT, which is used to improve grammar, rhetoric and overall readability, could have broader implications for scientific writing as well.
Campus resource: Can we detect AI-written content?
“LLMs are infamous for making up references, providing inaccurate summaries, and making false claims that sound authoritative and convincing. While researchers may notice and correct factual mistakes in LLM-assisted summaries of their own work, it may be harder to spot errors in LLM-generated literature reviews or discussion sections.”
With a risk that LLMs make academic papers less diverse and less novel than human-written text, the researchers warned that such homogenisation may degrade the quality of scientific writing.
And they called for a reassessment of current policies and regulations around the use of LLMs for science in light of the findings.
patrick.jack
