Scientific research can be a long and tedious process. And it can be frustrating. Just like time, funds are also limited.
Are there solutions to make science more efficient? AI colleagues — at least, potentially.
The first automated AI researcher
In 2024, Tokyo-based startup Sakana.ai introduced an “AI scientist.” — an AI system that allows you to create new machine learning research from scratch, fully autonomously and for just $15 (€13) per paper.
This model can perform the entire research process without human support, from creating new hypotheses to running code to producing results.
And it goes even further, with a unique peer review system that automatically assesses the quality of papers, ensuring they meet scientific standards.
When an independent team of researchers tested the 2024 version of the system, the quality of the results appeared to be quite low. Sure, they could run the entire research lifecycle alone, but the result was, in the authors’ words, akin to “an unmotivated undergraduate student rushing to meet a deadline.”
The most concerning issues were incomplete sections, outdated or limited references, and incorrect or fabricated numerical results, often referred to in AI parlance as “hallucinations.”
Still, the researchers thought the system had promise, especially in terms of its efficiency. They estimated that an AI could do in three and a half hours what would take a “depressed undergraduate” at least 20 hours. Average cost is $6 to $15.
AI paper accepted for academic workshop
Now, a year and a half later, Sakana.ai has tested the latest version of the system. Three AI-generated papers and 40 human-authored papers were submitted for peer review.At a workshop at a leading machine learning conference.
The reviewers knew that some of the posts were created by AI, but they didn’t know which ones.
Approximately 70% of submitted papers passed the first round. Two of the AI-generated papers failed, while one passed. That is, it met the scientific standards of the conference workshop.
These standards are much lower than those of major conferences.
“Research accepted there is not considered a real peer-reviewed paper for all scientists,” machine learning science professor Jakob Macke told Science Media Center Germany.
And the latest version of “AI Scientist” still has flaws. undeveloped ideas, structural problems, and hallucinations of various kinds.
However, as our unique rating system shows, papers appear to be steadily improving in quality over time, meaning that our future with AI scientists may not be too far away.
AI could solve the problem of scientific inefficiency
AI systems are tireless. They can read through research papers in seconds, never complain about overtime, and don’t have to be paid. Or at least cost significantly less than human researchers.
This can potentially yield more results in less time. This has made the process of scientific discovery more efficient.
But the question is, in what direction will these discoveries lead us?
When humans conduct research, the final product is the result of dozens, if not hundreds, of small decisions. No two researchers approach their research topic in exactly the same way.
Once these decisions are made by AI, they disappear into the shadows of what we consider “superhuman” systems: systems that are seen as smarter, faster, and more objective than we are.
AI researchers come with risks
So what happens if we rely on AI to replace our own diverse thinking?
Anthropologist Lisa Meseri and neuroscientist Molly Crockett are hoping for what they call a “scientific monoculture.”
In agriculture, monoculture is the practice of growing only one type of crop rather than multiple crops over a period of time. This usually means higher profits. But it also increases the risk that crops will fall victim to pests and diseases.
Something similar might happen if we let AI systems do science. The types of research that AI initiates may just be the type that AI is best suited for, at the expense of projects that require more context and nuance, or the “human touch.”
Not only does this narrow the scope of science, but it can also introduce the risk of systematic errors in which human awareness disappears when we remove ourselves from view.
“The biggest risk is placing too much trust in the results produced by AI. The key countermeasure will be the ability of humans to think critically,” Irina Gurevich, professor of ubiquitous knowledge processing, said at the Science Media Center Germany.
Without critical thinking, you may objectively accomplish more but understand less.
With AI, we have much to gain as well as much to lose.
The question remains: does human inefficiency add no value in science?
