researchers University College Cork (UCC) has been completed study it shows that AI-generated text He continues to display a unique stylistic pattern that sets him apart from the rest. human prose.
This study is the first in the world to use literary styrometry, a computational method traditionally used to identify authorship, to compare human writing styles and large-scale language models like ChatGPT across creative texts.
The researchers found that while AI can produce sophisticated and fluent prose, the writing continues to follow narrow, uniform patterns.
Human authors exhibit a much wider range of writing styles, shaped by their personal voices, creative intentions, and personal experiences.
Dr James O'Sullivan from the UCC School of English and Digital Humanities analyzed hundreds of short stories written by humans and works generated by AI systems.
The study revealed clear and consistent stylistic differences by examining subtle linguistic markers, such as the frequency of common words.
Dr O'Sullivan said the analysis showed clear stylistic differences between human writing and text produced by large-scale language models.
AI systems such as GPT-3.5, GPT-4, and Llama 70B produce tightly grouped clusters, each reflecting a uniform pattern unique to a particular model. In contrast, human writing exhibits much greater variation and individuality.
The researchers found that GPT-4 was able to write more consistently than GPT-3.5, but both are still different from human work. GPT-3.5 sometimes approaches human style, but such moments are rare.
Dr O'Sullivan said that while AI models produce a compact and predictable style, human writing remains more diverse and idiosyncratic, with characteristics that reflect individuality and creative intent.
“While AI writing is often sophisticated and consistent, it tends to be more uniform in word choice and rhythm. Human writing, by contrast, remains diverse and idiosyncratic, reflecting individual habits, tastes, and creative choices.”
“Even though ChatGPT tries to be human-like, its writing leaves detectable fingerprints, suggesting that computers and humans still don't write in exactly the same style.”
Researchers warn against using styrometry as an AI detection tool in education. However, they highlight its value in understanding how human expression differs from algorithmic generation and provide new insights into what constitutes human-like writing.
Dr O'Sullivan said styrometry could reveal broad patterns across large amounts of text. However, he argues that this is not relevant in determining the presence or absence of authorship in education.
“Student writing varies from assignment to assignment and is shaped by context, support, and lived experience. This makes stylometer detection unreliable and ethically questionable when it comes to academic integrity.”
Dr O'Sullivan said the findings demonstrate the need for testing with broader datasets, new prompts and new models, as well as deeper attention to the ethical and creative issues raised by the increased use of generative AI.
He added that it is important for large-scale language models (LLMs) to reliably generate emails and summary reports.
However, the ability to automate literary production raises “serious ethical and philosophical concerns about authenticity, originality, and authorship itself.”
The research was led by Dr. O'Sullivan and published in Nature — Humanities and Social Sciences Communications.
