Artificial intelligence (AI) has the potential to transform healthcare services by improving diagnostic accuracy, streamlining administrative tasks, and enhancing patient engagement. Between 2017 and 2021, the healthcare sector received more private AI investment globally than any other sector. Attracting $28.9 billion.
This enthusiasm for new medical technologies has long been accompanied by concerns about patient safety, harmful bias, and data security. Regulators face the challenge of facilitating these innovative tools while keeping machine learning algorithms safe, fair, and secure within the constraints of regulatory frameworks created in an era of physical devices, paper records, and analog data. The rapid introduction of AI into the healthcare process has created an urgent need to rethink existing regulatory frameworks.
Recognizing this gap, Stanford University's Human-Centered AI Institute (HAI) convened a closed-door workshop in May 2024, handpicking 55 leading policymakers, scientists, health care providers, ethicists, AI developers, and patient advocates. The conference was organized by HAI's Healthcare AI Policy Steering Committee, an interdisciplinary committee of Stanford faculty who drive policy and research in these areas, to identify key AI policy gaps and rally support for regulatory change.
Read a related conversation with HAI Deputy Director Curt Langlotz: How can health AI be better regulated?
Under the Chatham House Rule, participants discussed shortcomings in federal health AI policy in three areas: AI software for clinical decision support, AI tools for healthcare enterprises, and patient-facing AI applications. Below we summarize the key themes, policy considerations, and participant comments for each regulatory area.
It's like driving a 1976 Chevy Impala on the roads of 2024.
Healthcare is one of the most highly regulated industries in the United States, and the industry's extensive regulatory framework already applies to AI.
The Food and Drug Administration (FDA) is responsible for regulating many software systems, primarily through the 510(k) device approval process. Software as a Medical Device (SaMD)AI applications used in administrative and clinical enterprise environments must follow regulations from the Office of the National Coordinator for Health Information Technology, for example. Algorithmic transparencyGovernance of direct-to-consumer health AI tools falls under various consumer product frameworks, but there is still little enforcement in this new field.
These regulatory frameworks are outdated. Established in 1976, FDA regulators were designed to regulate hardware devices, not software, which relied on training data and required close, ongoing performance monitoring. Similarly, the Health Insurance Portability and Accountability Act (HIPAA), a 1996 law that established national standards for health data privacy and security, predates the explosion of digital health information. Its provisions did not anticipate the vast amounts of patient records that would be needed to train machine learning algorithms.
Regulators are effectively driving a Chevrolet Impala on the roads of 2024 and struggling to adapt to today's road conditions, one participant noted. Traditional regulatory paradigms in healthcare must quickly adapt to a world of rapid AI development. The majority of workshop participants believe that effective healthcare AI governance requires new or significantly modified regulatory frameworks.
Use Case 1: AI in Software as a Medical Device
Developers of new AI-powered medical devices with diagnostic capabilities currently face a major challenge: the FDA's device approval process requires submission of evidence for each individual diagnostic function. For an AI product with hundreds of diagnostic functions, such as an algorithm that can detect nearly every abnormality that can appear on a chest x-ray, it is not commercially feasible to submit each one for regulatory approval. As a result, software companies around the world could bring lower-grade, less innovative products to market, stifling AI medical device innovation in the United States.
Workshop participants proposed new policy approaches to streamline market approval of these multi-function software systems while ensuring clinical safety. First, public-private partnerships are critical to manage the burden of evidence for such approvals, potentially focusing on promoting post-market surveillance. Second, participants supported better information sharing during the device approval process. Sharing test data and details about device performance during the approval process would allow healthcare providers to better assess whether software tools will operate safely in their practice. Approximately 900 medical devices Medical devices incorporating AI and machine learning software have been approved by the FDA, but clinical adoption has been slow because healthcare providers have limited information on which to base purchasing decisions.
Finally, some participants called for more granular risk categories for AI-powered medical devices, most of which are currently classified as Class II devices with moderate risk. Clinical risks vary significantly across types of AI/machine learning software devices, requiring a more tailored approach. For example, an algorithm that measures blood vessel dimensions for later human review is less risky than an algorithm that triages mammograms to bypass human review.
Use case 2: AI in enterprise clinical operations and management
When autonomous AI tools are integrated into clinical practice, do humans always need to be involved? Fully autonomous AI technologies, such as those that diagnose eye diseases or automatically report normal chest x-rays, hold the promise of solving the severe physician resource shortage. Other forms of automation, such as ambient intelligence technologies that compose replies to patient emails or record progress notes during physician-patient interactions, can also greatly improve the efficiency of clinical practice.
Some participants argued that human oversight was necessary to ensure safety and reliability, while others warned that the requirement for human involvement could increase administrative burden on physicians and reduce their sense of ownership of clinical decisions. Others pointed to clinical testing as a successful hybrid model, where a physician monitors the device and performs regular quality checks. Any out-of-range values are checked by a human.
The integration of AI into clinical practice also raises the question of what level of transparency is needed to ensure that healthcare professionals and patients can use AI tools safely. Developers have a responsibility to communicate information about the design, capabilities, and risks of their models, for example by: Model CardThis is similar to a “nutrition label” that healthcare providers can use to make an informed decision about whether to use AI tools.
Additionally, should patients be told at what stage of their treatment AI is being used, and if so, how and when it is being used? Patients often defer to their caregivers and their healthcare provider for decisions about which technology to use, from scalpels to decision support pop-up windows. And less sophisticated forms of AI, such as rules-based systems that warn of drug-drug interactions, are already in use throughout the healthcare system. But many participants felt that patients should be informed that AI plays a role in some situations, such as email messages purporting to be sent by their healthcare provider.
Use case 3: Patient-facing AI applications
The rise in patient-facing applications, e.g. Mental Health Chatbot Based on LLM,promises to democratize access to healthcare or provide new services to,patients through mobile devices.,However, even when chatbots claim not to be providing,medical advice, and especially when they share information in a,way that closely resembles medical advice, there are no targeted,guardrails in place to prevent these patient-facing LLM-powered,applications from providing harmful or misleading medical information.
Clarification of the regulatory status of patient-facing products is urgently needed, but workshop participants were divided on whether generative AI applications should be regulated like medical devices or medical professionals, for example.
The patient perspective is crucial to ensuring the trustworthiness of medical AI applications and the healthcare system more broadly. Many participants noted that patients rarely have participation in the development, deployment, and regulation of patient-facing AI applications. To ensure that regulatory frameworks can address health disparities caused or exacerbated by AI, the needs and perspectives of the entire patient population must be considered.
What's next?
These are just a few of the many questions and concerns surrounding the future of healthcare AI regulation. Much more multidisciplinary research and multi-stakeholder discussions are needed to answer these questions and develop actionable policy solutions that support an agile approach to bringing innovative, life-saving AI applications to market while ensuring safety. HAI and its Healthcare AI Policy Steering Committee will continue to research these areas to support policy and regulatory frameworks that lead to the safe, equitable, and effective use of healthcare AI.
Stanford HAI's mission is to advance AI research, education, policy, and practice to improve the human condition. learn more.
