Google announced it is testing Scholar Labs, a new AI-powered search tool designed to answer detailed research questions. But that demonstration highlighted a larger problem about finding “good” scientific research. To what extent will scientists trust tools that read relationships between words and help surface good work, ignoring the typical methods of measuring work’s popularity in the scientific community?
The new search tool uses AI to identify key topics and relationships within users’ queries and is currently only available to a limited number of logged-in users. Scholar Labs’ demo video featured questions about brain computer interfaces (BCI). I have a PhD in BCI, so I was curious to see what Scholar Labs had come up with.
The first results were published in 2024 in a review paper of BCI research. applied science. Scholar Labs included an explanation of why the results matched the query, and noted that the paper discusses the study of non-invasive signals called brain waves and explores some of the leading algorithms in the field.
However, we found that Scholar Labs lacked filters for common metrics used to distinguish between “good” and “not-so-good” research. One measure is the number of times a study has been cited by other studies since it was published, which roughly describes the popularity of the paper. This also has to do with time. A recently published study may have zero citations, or it may have hundreds of citations within a few months. Studies from the 1990s may tout thousands. Another metric is the “impact factor” of scientific journals. Journals that publish widely cited research have higher impact factors and therefore have a reputation for being more rigorous and meaningful to the scientific community. applied science The impact factor is self-reported to be 2.5. natureFor comparison, it states that its impact factor is 48.5.
The original Google Scholar has an option to rank studies by “relevance” and lists the number of citations for each result. The goal of the new Scholar Labs is to discover “papers that are most useful for users’ research pursuits,” said Google spokeswoman Lisa Oguike. The Verge Google says it ranks papers in the same way as the researchers themselves, by considering “the full text of each document, where it was published, who wrote it, and how often and recently it has been cited in other academic literature.”
However, Oguike said the new Scholar Labs will not sort or limit results based on paper citation counts or journal impact factors. The Verge.
Image: Google Scholar
“Impact factors and citation counts vary depending on a paper’s research field, and for most users it can be difficult to guess the appropriate value in the context of a particular research question,” Oguike wrote. “Restricting by impact factor or citation count often misses important papers, especially those in interdisciplinary/adjacent fields/journals, or recently published papers,” Oguike added.
Matthew Schrag, an associate professor of neurology at Vanderbilt University Medical Center, said in an interview that metrics such as citation counts and impact factors are “a pretty rough assessment of the quality of a paper.” The VergeI agree with Google’s statement. He says they “say more about the social context of the paper” than the quality of the paper, but “we hope these two things are correlated.”
Schrag, who studies Alzheimer’s disease, is one of many scientists and detectives who have flagged questionable data in published scientific studies. As a result of the efforts of data sleuths like Schrag and the attention of the entire scientific community, research results have been removed from reputable journals due to doctored images, corrections by Nobel Prize winners, and federal investigations into falsified data.
Still, the difficult thing is do not have Especially when entering a new field, feel free to use citation counts and journal reputation to scrutinize research. James Smoliga, a professor of rehabilitation science at Tufts University, frequently uses the original Google Scholar and has found that he believes papers with more citations are more reliable. “I’m guilty, just like everyone else,” he said. The Verge. He does so even though thousands of citations debunk the methods used in the study. “And I know I’m not like that either, but I still fall into that trap because what else am I going to do?”
I repeated the Scholar Labs demo query for BCI research in stroke patients on PubMed, a major repository for biomedical and health research run by the National Institutes of Health’s National Library of Medicine. Unlike Scholar Labs, PubMed relies heavily on relevant filters and terms. orand ands. I narrowed down my results to only review clinical research papers from the past five years. That is, we only reviewed those that were only performed on humans. Preprints were excluded. This is research submitted directly to article repositories such as arXiv or bioRxiv without going through a review process by other scientists. Two of the six results focused solely on EEG as the main type of non-invasive BCI used to help stroke patients.
Users can request “recent” papers in their queries, specify a time period in their requests, and Scholar Labs can use “full text of research papers” to find results that match users’ queries, Oguike added.
Google calls Scholar Labs “a new direction for us” and says it plans to incorporate user feedback in the future. There is a waiting list for access.
Schrag believes AI-powered searches like the new Scholar Labs have a place in the scientific ecosystem. In theory, he added, this could cast a wider net on the surface of papers that would otherwise slip through the cracks, or add additional context about a paper’s popularity across social media platforms. Research requires comprehensive evaluation, which AI may be able to address, he said. “We need to understand what the standards are in this field in terms of rigor and whether the research meets them,” he added.
Ultimately, Schrag said, scientists are responsible for determining what kind of science their science influences. It requires reading and engaging with the scientific literature that “instead of letting algorithms be the final arbiters of what we consider high quality, we don’t let algorithms be the final arbiters.”
