AI is rapidly penetrating the medical field, and while it brings potential benefits, it can also bring pitfalls, such as bias that can lead to unequal treatment and burnout for doctors and other health care workers. How it should be regulated in the U.S. has not yet been decided.
In September, the Joint Commission on Accrediting Hospitals and the Health AI Coalition issued recommendations for the implementation of artificial intelligence in healthcare, but the burden of compliance falls primarily on individual facilities.
I. Glenn Cohen, a professor at Harvard Law School's Petrie Flom Center for Health Law, Biotechnology, and Bioethics, and colleagues suggested in the Journal of the American Medical Association that the guidelines are a good start, but changes are needed to reduce potential regulatory and financial burdens, especially on smaller hospital systems.
In this edited conversation, Mr. Cohen, Professor James A. Atwood and Leslie Williams Professor of Law, discussed the difficulty of balancing thoughtful regulation with avoiding unnecessary obstacles to breakthrough innovation during rapid adoption.
Is it clear that AI in healthcare requires regulation?
Whenever medical AI deals with medium to high risk, there will be a need for regulation, either internal self-regulation or external government regulation. Until now, this has been largely done internally, with differences in how each hospital system validates, reviews, and monitors healthcare AI.
If done on a hospital-by-hospital basis, the cost of doing this type of evaluation and monitoring can be significant, meaning some hospitals can do it while others cannot. In contrast, top-down regulation is slow and may be too slow for some advances in this area.
Hospitals have a complex mix of AI products. Some help with things like internal purchasing and reviews, but more are clinical or clinically adjacent.
Some medical AI products interface directly with consumers, such as chatbots that people may be using for mental health. For this reason, there has not even been an examination within the hospital, making the need for regulation even clearer.
With technology advancing so rapidly, does speed even matter in regulation?
This is an innovation ecosystem rich in startup energy, which is great. But you're talking about something that can scale very quickly without a lot of internal review.
When we enter into what I call “racial dynamics,” there is a danger that ethics can quickly be left behind. Whether it's a race to be the first to develop something, a race between start-ups battling a lack of funding, or a national race between nations to develop artificial intelligence, the pressures of time and urgency can make it easy for ethical issues to be overlooked.
The vast majority of medical AI is not reviewed by federal regulators, and probably never by state regulators. We want to establish standards for healthcare AI and incentives to adopt them.
But for those obsessed with Silicon Valley's speed of development, putting everything through the rigorous FDA process, even in the case of drugs and medical devices, will often be prohibitively expensive and prohibitively time-consuming.
Conversely, many of these technologies pose a much greater risk to the general public if they perform poorly than the average device on the market.
When taking aspirin or statins, their effects differ from person to person, and these differences can be characterized to some extent in advance. When medical AI reads X-rays or does anything in the mental health field, how it is implemented is key to its performance.
Implementation must be considered very carefully, as different hospital systems may have very different outcomes based on resources, staffing, training, and user experience and age. This poses an unusual challenge for agencies like the FDA (which often says it doesn't regulate medical practice) because of the complexity of where approval for AI systems ends and medical care begins.
Your research examines the regulatory systems proposed by the Joint Commission, the Hospital Accreditation Agency, and the Coalition for Health AI. Are accrediting bodies something that hospitals should or should naturally focus on?
that's right. In nearly every state, you must be certified by the Joint Commission to bill Medicare and Medicaid. This is a big part of almost every hospital's business.
There is a robust process for becoming certified and is subject to re-evaluation. It's serious business.
The Joint Commission has not yet stated that these AI rules will be part of the next certification, but these guidelines indicate a possible move in that direction.
“I talk about legal and ethical issues in this field, but I'm an optimist about this. I think the world will be significantly better off in 10 years because of medical artificial intelligence.”
Think you need some recommendations?
I think it's actually pretty good, although some things are tougher than I expected.
Where AI has a direct impact on patient care, mandating that patients be informed where appropriate and, where relevant, consent must be obtained for the use of AI agents, is a strong position to take.
Many academics and other organizations do not take the position that medical AI should be disclosed whenever it directly impacts care. Furthermore, we do not take the position that informed consent should always be required.
The guidelines also call for continuous quality monitoring and continuous testing, validation, and monitoring of AI performance.
The frequency of monitoring is adjusted to the level of risk of patient care. These are good things, but they are difficult and expensive. A multidisciplinary AI committee should be formed to constantly measure accuracy, errors, adverse events, fairness, and bias between populations.
If you think about it seriously, it probably won't be possible for many hospital systems in the United States. You will need to determine the breaking point for implementing AI.
In your JAMA article, you point out that most hospitals in the United States are small community hospitals, and resources are a big issue.
I've heard from people at major hospital systems that are already doing this that properly vetting a complex new algorithm and its implementation can cost between $300,000 and $500,000. That's completely out of reach for many hospital systems.
In reality, some aspects of implementation are unique to each hospital, while others are common to many hospital systems and may be worth knowing. The idea of repeating assessments in multiple locations and not sharing what you learn seems like a huge waste.
If the answer is, “If you can't play in the big leagues, you shouldn't be at bat,” then that creates a distribution of haves and have-nots in terms of medical access. We already have that in general health care in this country, but this will further enhance that dynamic at the hospital level.
Access to AI to aid healthcare depends on being within the network of large academic medical centers that are proliferating in places like Boston and San Francisco, as opposed to other parts of the country that don't have such medical infrastructure.
Ideally, the goal is to better centralize and share information, but these recommendations place a heavy burden on individual hospitals.
Wouldn't a system that doesn't allow some hospitals to participate negate the potential benefits of this latest generation of AI? AI can assist low-resource locations by providing expertise that is lacking or difficult to find.
It would be a shame if we had great AI that could help people and be most effective in low-resource settings, but that setting could not meet the regulatory requirements for implementation.
It also becomes a sad ethical reality if we train these models on data from patients across the country and it turns out that many of those patients will never benefit from these models.
If the answer is that the review and oversight of medical AI should be done by a larger organization, then is it the government?
The Biden administration's idea was to create an “assurance lab,” a private organization that could work with the government to vet algorithms based on agreed-upon standards so health care providers could trust them.
The Trump administration agrees with this issue, but has indicated that it does not like this approach. They haven't fully shown what their vision is yet.
It's a complex landscape, but it also sounds like it's changing rapidly.
It's complex, but also challenging and interesting.
I talk about legal and ethical issues in this area, and I'm an optimist about this. I think the world will be a lot better off in 10 years thanks to medical artificial intelligence.
The proliferation of these technologies into low-resource settings is very exciting, but only if incentives are properly aligned. It does not happen by chance, and it is important to consider these distributional concerns as part of any attempt to legislate in this region.
Source link
