Preprint website arXiv has announced that researchers who put their names on papers containing errors clearly generated by artificial intelligence (AI) will face a one-year ban and ongoing restrictions.
The move is a response to the increasing influx of AI-generated papers faced by academic journals and sites such as arXiv, which serve as unofficial platforms for publishing research before peer review.
However, not everyone agrees that arXiv’s response to this problem is appropriate. And the solution to the deluge of AI slop research may involve more AI, not less AI.
The rise of bot-assisted writing
AI-generated text is increasing everywhere. Half of all new articles published online are now “primarily generated by AI,” according to a study released last week.
Science is not immune to this trend. Last month, the journal Organization Science published a study on how the rise of AI has affected submissions and peer review since the release of ChatGPT in 2022. Reporting a dramatic increase in submitted papers and a decline in quality, the authors conclude that “the current state of AI tools, amplified by existing publish-or-perish incentives, appears to be driving the system toward an equilibrium of more research, rather than better research.”
A common problem with AI-generated research papers is phantom citations, or references to other research that doesn’t exist.
The traditional safeguard against poor quality in academic publishing is peer review. That is, another expert in the subject reads the research paper and asks questions about the research behind it before it is published.
However, peer review systems were already struggling before AI. Researchers under pressure often have little time or motivation to undertake the unpaid work of peer review.
And arXiv, which publishes preprints (mostly non-peer-reviewed papers), doesn’t even have this system. Last year, the site stopped accepting certain types of articles due to an influx of AI-generated posts.
A study published in January (itself a preprint) estimates that about one in eight biomedical science papers now contains AI-generated text.
Most researchers would agree that AI-generated text itself is not the problem. The problem is the low-quality work that AI can easily create.
Does the punishment fit the crime?
ArXiv’s announcement does not argue against the use of AI, but rather states:
If the submission contains indisputable evidence that the author did not check the results of the LLM generation, this means that nothing in the paper can be trusted.
This may be true to a certain extent. However, the penalty of a one-year suspension for all authors listed on a paper may not be consistent with current research practices.
In the past, research was often done alone or in groups of two or three people. In such situations, it seems reasonable to expect each author to take responsibility for the whole.
But research is now more collaborative than ever. Many papers have four or five authors, and in extreme cases, papers are increasingly being seen as collaborative research by hundreds of scientists, each working in their own area of expertise and trusting that their colleagues are doing the same work.
If one author out of dozens or hundreds includes a reference to AI hallucinations in part of their paper, it seems harsh to ban that lot.
Additionally, there are no equivalent sanctions for publishing other problematic content. For example, it is not prohibited to impose fringe or unreliable theories, or to use poor quality evidence or illogical arguments.
Will AI help fight slop?
The rise of AI poses challenges for publishers and quality assurance. And the idea of some kind of sanction for reckless use of AI (such as including hallucinatory references) is a good one.
But ArXiv’s particular choice seems drastic. If the goal is to improve peer review and quality assurance, AI systems themselves can play a role.
Modern AI systems are fully capable of taking a list of references and verifying that everything on that list is a genuine article available on the internet. All references flagged as questionable may be checked by humans.
AI can also help perform quick sense checks, such as statistical analysis of papers.
Perhaps this, rather than imposing harsh sanctions for relatively minor AI-related violations, is the way forward.
Vitomir Kobanovic does not work for, consult, own shares in, or receive funding from any company or organization that might benefit from this article, and has disclosed no relevant affiliations other than his academic appointment.
/Courtesy of The Conversation. This material from the original organization/author may be of a contemporary nature and has been edited for clarity, style, and length. Mirage.News does not take any institutional position or position, and all views, positions, and conclusions expressed herein are solely those of the authors.
