Open the FDA’s device certification database and search for AI-powered clinical decision support tools. Next, let’s look for published clinical validation data for each. A multicenter study by UNC, Duke University, Oxford University, Columbia University, and the University of Miami was found to do just that, halting the callous behavior of clinical operational leaders. Of the 521 FDA-cleared AI-powered clinical devices, 43% had no publicly available clinical validation data. These tools are already in the hospital. They are already influencing care decisions. And nearly half of them have never been independently tested in published studies.
That’s the signal. The research network being built by Emily Tutt of Columbia University Irving Medical Center and NewYork-Presbyterian Hospital and Peter Brodeur of Beth Israel Deaconess Medical Center, described in the JAMA+ AI Conversations article, represents one of the structurally most serious attempts to address that gap from the inside out. Their work focuses on clinical AI evaluation. Rather than building AI, we are stress testing it in ways that the FDA currently cannot require and the market does not currently reward.
The verification gap left by the FDA
Previous assumptions were that FDA approval served as a signal of quality. A tool is verified if a government agency has approved it. Sponsors, health systems, and trial operators accepted this logic. This is because the alternative of running an independent evaluation study on every AI system deployed did not seem operationally possible at scale. That assumption is now untenable.
The FDA’s AI/ML-Based Software as Medical Devices Action Plan, released on January 12, 2021, provided a framework for monitoring adaptive AI systems. It was a serious document and moved the conversation forward. However, no post-market clinical validation studies were mandated, and no independent evaluation infrastructure was established. Five years later, the FDA’s 2024 White Paper on AI across CBER, CDER, CDRH, and OCP strengthens the alignment posture, but still falls short of the need for prospective, site-specific performance testing that tells clinical sites whether the tools actually work in patient populations.
The market hasn’t closed the gap either. The AI-powered clinical decision support market was valued at $730 million in 2024 and is projected to reach $1.79 billion by 2030, growing at a CAGR of 15.6%. At this rate of growth, the number of unvalidated tools in clinical settings will grow faster than regulatory agencies can retrospectively audit them.
This is a structural problem that the Columbia and Beth Israel networks are trying to solve before markets completely overtake governance.
Who will be exposed first?
Clinical trial operations are at the highest risk of this risk. When clinical AI tools impact facility-level decisions, such as flagging protocol eligibility, predicting dropout, or supporting endpoint adjudication, they become part of the evidence chain that FDA scrutinizes during bioresearch monitoring (BIMO) inspections. Unverified tools embedded in that chain are GCP compliance exposures with dollar signs attached.
Therapeutic areas with complex protocols face the highest risks in the short term. Oncology trials using AI-assisted image reading, CNS trials relying on digital biomarkers for endpoint capture, and cardiovascular studies using algorithmic safety signal detection are all being conducted using tools that have passed the FDA, often without publishing a single independent validation study. FDA Warning Letter 320-26-58, dated April 2, 2026, marks the first time FDA has explicitly cited inappropriate AI use in drug manufacturing and signals a shift in enforcement. What started in manufacturing also extends to clinical operations.
Distributed trial designs face a complex version of this problem. DCT platforms are increasingly built on AI layers for remote monitoring, ePRO anomaly detection, and siteless eligibility screening. When these AI components lack external validation data, the entire DCT architecture has built-in reliability assumptions that have not been tested against the actual patient population it serves. CMS recognized a version of this risk when it clarified that as of February 6, 2024, Medicare Advantage plans that use AI to determine coverage must continue to meet individualized care standards. This shows that reviewers are not satisfied with the algorithm’s output alone.
Sponsors conducting adaptive trials are particularly at risk. If an AI tool influences adaptive decision rules during ongoing trials, and the tool’s performance in a particular enrollment population has never been independently benchmarked, then the adaptive framework is based on a foundation that cannot be audited. The FDA’s statistical reviewers will notice.
operational command
If you are currently deploying clinical AI tools for endpoint determination, eligibility screening, safety monitoring, or data quality flagging, obtain vendor validation documentation today and ask one question: Has the tool been validated in a patient population comparable to the population enrolled in the trial, and is that validation publicly available and independently reproducible? If the answer to either is no, protocol risk is not built into the vendor agreement, but is included in the risk management plan. The NSF-NIH Smart Health and Biomedical Research Program is actively funding evaluation infrastructure, but its grant cycle is faster than the next BIMO test.
The work that Tat and Brodeur are conducting through their research network represents the type of independent evaluation architecture that sponsors should be actively involved in, rather than waiting for regulatory requests. Submitting your implemented tools to an external evaluation consortium before you are asked to defend them by the FDA is a proactive compliance stance that protects both the trial and the patients participating in the trial.
Stay tuned for FDA’s next AI/ML SaMD guidance. This is expected to address post-market performance monitoring requirements more specifically than the 2021 Action Plan. As that declines, sponsors who have not yet established an independent validation baseline for their clinical AI stacks will be scrambling to build one in time. The question for clinical operations leaders now is: Do they want that baseline to exist because they built it, or does it exist because the FDA tells them it has to?
References
- JAMA+ AI — “Designing Trustworthy Clinical AI” (JAMA+ AI Conversations, 2026)
- Health Economics — “Nearly half of FDA-approved AI tools may be ineffective” (citing multi-institutional study in Nature Medicine)
- FDA — “Artificial Intelligence/Machine Learning (AI/ML)-Based Software as Medical Devices (SaMD) Action Plan,” January 12, 2021
- FDA/King & Spalding — “Artificial Intelligence and Medical Products: How CBER, CDER, CDRH, and OCP Work Together,” 2024
- Mordor Intelligence — “AI-powered Clinical Decision Support Market Forecast to 2030”, June 13, 2025
- Manatt Health — “CMS Considers Use of Algorithms and Artificial Intelligence,” February 2024
- TeleDirect MD — “FDA First AI Warning Letter 2026,” citing Warning Letter 320-26-58, April 2, 2026
- NSF — “Smart Health and Biomedical Research in the Era of Artificial Intelligence and Advanced Data Science (SCH)”, solicitation NSF 25-542
Moe Alumidaie is editor-in-chief of The Clinical Trial Vanguard. Moe has decades of experience in the clinical trials industry. Moe also serves as Director of Research at CliniBiz and Chief Data Scientist at Annex Clinical Corporation.
