The moon’s cratered surface might seem like the perfect place to let machines do repetitive tasks. There are millions of impact tracks to track, all of which can help tell us about global time. But new analysis shows that many of the moon’s AI-generated crater catalogs are still not as reliable as the headline numbers suggest.
In planetary science, a crater catalog is more than just listing circles on a map. They record the location, size, and other characteristics of impact structures, allowing researchers to estimate surface ages, reconstruct geological history, and study how the landscape has changed over time. If these measurements are wrong, the science built on them can fail as well.
“AI has great potential to help with repetitive and time-consuming scientific tasks, especially some data collection,” said Dr. Stuart J. Robbins of Southwest Research Institute’s Solar System Science and Exploration Division in Boulder, Colorado, who led the study. “However, our analysis shows that researchers should not assume that AI-generated crater catalogs are ready for scientific use based solely on published metrics.”
Robbins and co-author Dr. Rachel H. Huber compared eight lunar crater databases created with automated methods, including machine learning systems and early non-ML approaches. They tested each against an extensive catalog of moons that Robbins had built by hand over several years, and evaluated them using the same crater matching rules across the board.
Where the published numbers start to collapse
This detail is important because performance can vary significantly depending on what counts as a successful crater “match.” For many applications in planetary science, craters are only useful if they are in the right place and the right size. Even if a system appears powerful under broad computer vision standards, it can deviate enough to cause problems in real-world scientific research.
“The Crater Catalog is more than just a random list of circles,” Robbins says. “If craters are misaligned, duplicated, or the wrong size, it can affect the science that relies on those metrics. For example, if you want x number of craters on a surface with a model age of 1 million years, and the AI accidentally duplicates those craters, the model will suddenly double the estimated age of the surface.”
The team found that testing the database against more stringent uniform standards led to a decline in many of the published metrics. Using criteria based on reproducibility of manual crater analysis, nearly all automated catalogs performed worse than published recommendations. In some cases, the reduction was more than a factor of 10.
That doesn’t mean all catalogs fail the same way. Some worked fairly well for certain crater sizes, while others fared much worse for others. A single overall score can make a database appear acceptable even if the usefulness of the database changes rapidly over the diameter range that is most important to researchers.
“Diameter dependence is important,” Robbins says. “While the catalog looks acceptable on its own as an overall number, breaking it down by crater size can be useful for some questions but may be unreliable for many others.”
Why you can’t shake off the size and location of a crater by hand
Crater counts are one of the fundamental tools scientists use to date the solid world. Small asteroids bombard the Moon and other celestial bodies at a near constant rate, so cratered surfaces are generally older than less cratered surfaces. Researchers compare crater size and density and use impact rate models to estimate the age of the terrain.
That’s why these databases are so important. Impact craters are the most common surface process throughout the solar system’s rocky world, and crater catalogs support studies on age modeling, geological reconstructions, and crustal properties. These have traditionally been built using manual or semi-automated methods, a process that can take years and still involves subjectivity.
AI and machine learning promise a way to solve that bottleneck. Improved automation will allow scientists to process vast data sets more quickly and study crater populations at a scale that would be difficult for humans alone. The new study doesn’t deny that goal. Instead, the field argues that we need clearer rules for determining whether automated catalogs are actually trustworthy.
To test it, the authors compared eight large lunar crater databases. 2014, Wang et al. 2015, Silburt et al. 2019, Yang et al. 2020, Wang et al. 2021, La Grassa et al. 2025a, La Grassa et al. 2025b, and Xiong et al. They measured them against Robbins’ manually compiled 2019 reference database, which the study says is nearly complete, at least for craters 1 to 2 kilometers in diameter.
This paper also argues against using intersection over union (IoU) as the default score. Although this metric is common in computer vision, the authors argue that it is not well suited for impact craters because it can accept craters that look well overlapped even if their diameters and locations are imprecise enough to distort scientific analysis.
A rapidly changing field with no shared benchmarks
The broader problem, the authors argue, is one of contradiction. Different teams have different definitions of the game and different levels of size and location tolerance, but they often do not clearly explain their choices. At the same time, many users may treat published precision and recall values, without independent verification, as evidence that the catalog is ready for scientific use.
A new comparison suggests that confidence is often misplaced. The study reported that most of the AI-based databases exhibited size and location biases, and that the variability in measurements was generally worse than that seen among human expert crater analysts. In practice, this means that the ages estimated from these craters may contain greater uncertainties and biases than the standard error models indicate.
The authors also found no clear trends indicating steady improvement in the new AI crater catalog under a common unified test. One of the best performing datasets in our comparison was created in 2014 and relied on deterministic AI techniques rather than modern machine learning, but also included manual checks of all features.
Huber said the key is not to drive AI out of planetary science, but to make its use more rigorous. “Our research highlights the next steps needed to standardize benchmarks, including transparent reporting of match criteria and independent validation, so that AI-generated catalogs can be appropriately used for scientific analysis,” she said.
Robbins put it more clearly. “AI could ultimately transform crater cataloging, revolutionizing the way we collect scientific data and potentially saving years of time,” he said. “For now, researchers don’t need to pursue it as a be-all and end-all solution. We need to understand how these tools work, where they fall short, and whether their performance is sufficient to support the science that is currently being done.”
Practical implications of the research
For planetary scientists, the message is caution, not retreat. Automated crater catalogs may still be useful as a starting point for manual screening or for less accurate but acceptable research questions. However, the study argues that just because they report strong headline metrics, they should not be treated as compatible with expert-authored catalogs.
The authors recommend clearer reporting of fit tolerances, independent checks against reference databases, and precision and recall values broken down by crater size.
They also suggest that future AI efforts may be most useful by focusing on small craters, where manual cataloging is most difficult and where current systems still have difficulties.
