Since OpenAI launched the ChatGPT chatbot in November 2022, ChatGPT may pretend to be human, but its inaccuracies make it deadly when used for serious tasks like academic papers. error may occur.
Researchers at the University of Kansas have developed a tool that can strip AI-generated academic documents from human-written ones with over 99% accuracy.This work was published in a magazine on June 7. cell report physical science.
Heather Dezere, a professor of chemistry at the University of Kansas and lead author of the new paper, said she was “genuinely impressed” with many of ChatGPT’s results, but its accuracy limitations prompted her to develop new identification tools. It has become. . “Her AI text generators like ChatGPT aren’t always accurate.
“In science, we are built on a common knowledge about our planet, and I wonder what the implications would be if AI text generation was heavily leveraged in this area,” says Desaire. say. “When AI training sets contain inaccurate information, it becomes even more difficult to separate fact from fiction.”
“after a while, [the ChatGPT-generated papers] I got the impression that it was really monotonous. — Heather Desire, University of Kansas
In order to convincingly mimic human-made sentences, chatbots like ChatGPT are trained on large numbers of real-world text samples. Although the results are often convincing at first glance, existing machine learning tools can reliably identify telltale signs of AI intervention, such as non-emotional language use.
However, existing tools, such as the widely used deep learning detector RoBERTa, have limited application in academic papers, as they are likely to omit emotional language already. researchers write. In a previous study of AI-generated academic abstracts, RoBERTa’s accuracy was around 80%.
To fill this gap, Desaire and colleagues developed a machine learning tool that requires limited training data. To generate the training data, the team collected 64 Perspectives articles from journals where scientists described new research. chemistry, We then used those articles to generate 128 ChatGPT samples. These ChatGPT samples contain 1,276 paragraphs of text for researcher tools to explore.
After optimizing the model, the researchers tested it on two datasets, each containing 30 original human-written articles and 60 ChatGPT-generated articles. In these tests, the new model showed 100 percent accuracy when evaluating the entire article, and 97 percent and 99 percent accuracy on the test set when evaluating only the first paragraph of each article. By comparison, RoBERTa’s test set accuracy was only 85 and 88 percent.
From this analysis, the team identified sentence length and complexity as telltale signs of AI writing compared to humans. We also found that human writers were more likely to name their colleagues in their sentences, while ChatGPT was more likely to use general terms like “researcher” and “other.”
Overall, Desaire said this made for a more boring sentence. “In general, I think papers written by humans are more appealing,” she says. “The papers written by the AI seemed to remove complexity, for better or worse.
The researchers hope that the study will serve as proof of practice that off-the-shelf tools can be leveraged to identify AI-generated samples without extensive knowledge of machine learning.
However, these results may only be promising in the short term. Desaire et al. point out that this scenario only scratches the surface of what his ChatGPT can do for academic papers. For example, if ChatGPT were asked to write perspective articles in the style of a particular human sample, it could be even more difficult to spot the differences.
Desaire says he sees a future where AI like ChatGPT is used ethically, but says identification tools must continue to grow along with the technology to make this possible.
“I think it can be used safely and effectively in the same way that spell checking is used today. there is potential,” she says. “If you do this, you must be absolutely sure that this step does not introduce factual error. I have.”
from an article on your site
Related articles on the web
