The algorithm submitted for the North American Radiology Society (RSNA) hosted AI challenge demonstrates excellent performance for detecting breast cancer in mammographic images, increasing screening sensitivity while maintaining low recall rates, according to a study published today. RadiologyRSNA's premier journal.
The RSNA Screening Mammography Breast Cancer Detection AI Challenge was a crowdsourcing competition held in 2023, with teams of over 1,500 people participating. Radiology For more information on the article, see the analysis of the performance of an algorithm led by Dr. Yangchen, a professor of cancer screening at the University of Nottingham, UK.
We were overwhelmed by the amount of competitors and the number of AI algorithms submitted as part of the challenge. This is one of the most participating RSNA AI challenges. We were also impressed with the performance of the algorithm, taking into account the requirements for sourcing training data from open source locations, considering the relatively short windows where the algorithm was allowed to develop. ”
Dr. Yang Chen, Professor of Cancer Screening at University of Nottingham
The goal of the task was to procure AI models to improve the automation of cancer detection in mammogram screening, and to help radiologists work more efficiently, improve patient care quality and safety, and reduce costs and unnecessary medical procedures.
RSNA invited teams from around the world to participate. Emory University in Atlanta, Georgia and Breast Screen Victoria in Australia provide a training data set of approximately 11,000 breast screening images, allowing challenge participants to source published training data for the algorithm.
Professor Chen's research team evaluated 1,537 working algorithms submitted to the assignment, tested them on a set of 10,830 single-breasted tests, and separated them separately from the training data set.
Overall, the algorithm resulted in a median specificity of 98.7% to confirm that no cancer is confirmed in mammographic images, a sensitivity of 27.6% to positively identify cancer, and a recall rate (recall rate) for the proportion of cases where AI was 1.7% positive. When researchers combined top 3 and top 10 execution algorithms, they increased sensitivity to 60.7% and 67.8%, respectively.
“When I surrounded the top performance entries, I was surprised to see that the various AI algorithms were very complementary and identifying different cancers,” Professor Chen said. “Because the algorithm had a threshold optimized for positive predictors and high specificity, different cancer features in different images triggered high scores differently with different algorithms.”
According to the researchers, creating an ensemble of the 10 most performant algorithms produced performances close to the performance of the average screening radiologist in Europe or Australia.
Individual algorithms showed significant differences in performance depending on factors such as cancer type, the manufacturer of the imaging instrument, and the clinical site where the image was acquired. Overall, this algorithm was more sensitive to detecting invasive cancer than non-invasive cancers.
Because many of the participants' AI models are open source, the results of the task could contribute to further improvements in both experimental and commercial AI tools for mammography, with the aim of improving breast cancer outcomes around the world, Professor Chen explained.
“By making the algorithms and comprehensive imaging datasets available to the public, participants will provide valuable resources to facilitate further research and enable the benchmarks needed for effective and secure integration of AI into clinical practice,” she said.
The research team will conduct follow-up studies to benchmark the performance of top challenge algorithms for commercially available products using larger and more diverse data sets.
“We will further explore the effectiveness of smaller, more challenging test sets, including robust human reader benchmarks, and compare their usefulness with those of larger data sets, including those developed by Performs Scheme, a UK-based program to assess and assure the quality of radiologist performance as an approach to AI assessment.
RSNA hosts an annual AI challenge, and this year's competition calls for the submission of models that will help detect and localize intracranial aneurysms.
sauce:
North American Society of Radiology
Journal Reference:
Chen, Y. et al. (2025) 2023 Performance of algorithms submitted in the RSNA screening mammography breast cancer AI challenge. Radiology. doi.org/10.1148/radiol.241447.
