Using AI in healthcare can save time, money and life. However, when technology known to lie at times is introduced into patient care, serious risks also increase.
One London-based patient recently experienced how serious these risks are after receiving a letter inviting him to a gaze screening for diabetes. Problem: He was not diagnosed with diabetes or showed no signs of his condition.
After opening an appointment late one night, the patient, a healthy man in his mid-20s, said luck He was briefly concerned that he had been unconsciously diagnosed with this condition before he concluded that the letter had to be merely a management error. The next day, with a pre-scheduled regular blood test, the nurse questioned the diagnosis and when the patient confirmed that he was not diabetic, the pair reviewed his medical history.
“He showed me notes about the system, and those were AI-generated summaries. At that point I realized something strange was happening,” said a patient who called for anonymity to discuss private health information. luck.
After fully requesting and reviewing the medical records, the patient noticed that an entry introducing a diagnosis of diabetes was listed as a summary “generated by Annie AI.” The records appeared at about the same time he attended the hospital for a severe case of tonsillitis. However, the record in question did not mention tonsillitis. Instead, he said he showed chest pain and shortness of breath due to “a more likely angina pectoris due to coronary artery disease.” In fact, he had none of these symptoms.
Reviewed records luck, It should also be noted that the patient was diagnosed with type 2 diabetes late last year and is currently undergoing a series of medications. It also included details of the dosage and administration of the drug. However, these details were not accurate, according to patients and several other medical records reviewed. luck.
Health Hospital in Health City
Even strangers thought the record was attributed to the address of a medical document that appeared to be processing a fictitious “health hospital” at 456 Care Road in “Health City.” The address also included the invented postal code.
Dr. Matthew Noble, head of the NHS, I said luck The GP practice responsible for monitoring adopted “limited use of monitored AI” and the error was “one-time cases of human error.” He said that the medical summary initially found an error in the patient's record but was distracted and “said they accidentally saved the original version, not an updated version. [they] I was working on it. ”
However, the fictitious AI-generated records appear to have produced downstream results with invitations of patients attending an appointment for screening diabetic eyes based on false summary.
Most AI tools used in healthcare are monitored by strict human surveillance, but another NHS worker said luck A leap from the original symptoms, namely reducing inflammation – to what returned – angina due to coronary artery disease – alarm bells have been generated.
“These human error mistakes are fairly inevitable when there is an AI system that generates completely inaccurate summaries,” said an NHS employee. “Many elderly or fewer literacy patients may not even know there is a problem.”
AnimaHealth, the company behind the technology, did not respond Fortune Questions about the problem. However, Dr. Noble said, “Anima is an NHS approved document management system that helps practice staff help process incoming documents and take action on necessary tasks.”
“There are no documents processed by AI. Anima only proposes a summary to code and human reviewers to improve safety and efficiency. All documents require human review before action and submission,” he added.
Anxious developments in AI's health sector
The incident represents some of the growth pain associated with AI deployments in healthcare. The challenge is also tackling the challenge of integrating rich technology into a high-stakes environment as hospitals and GP practices compete to adopt automated tools that promise to mitigate workloads and reduce costs.
The pressure to innovate technology and potentially save lives is high, but the need for strict surveillance is also high, especially as tools that were considered “support” are beginning to affect actual patient care.
The company behind Anima Health, behind technology, promises that healthcare professionals can “save one day a day through automation.” The company offers services such as automatically generating “patient communications, clinical notes, administrator requests, and documents that doctors deal with every day.”
Annie, an AI tool from Anima, is registered as a Class I medical device with the UK's Medicine and Healthcare Products Regulatory Agency (MHRA). This means it is considered low risk and is designed to assist clinicians with test lighting and bandages rather than automate medical decisions.
AI tools in this category require clinicians to review the output or items to be entered into the patient record before an action is taken. However, in this case of misdiagnosed patients, this practice seemed unable to properly address the de facto error before it was added to the patient's records.
The incident comes amid an increase in scrutiny within the UK health services regarding the use and classification of AI technology. Last month, the health services boss warned GPS and hospitals that current use of AI software could violate data protection regulations and put patients at risk.
In an email first reported by Sky News and confirmed by luckNHS England warned that unauthorized AI software that violates minimum standards is at risk of putting patients at risk. The letter specifically addresses the use of ambient voice technology, or “AVT” by some physicians.
The main issue with AI transcription or summary is manipulation of the original text, said Brendan Delaney, professor of medical informatics and decision making and PT general practitioner at Imperial College London. luck.
“It gives medical device purpose, not just passive recording,” Delaney said. However, recent guidance issued by the NHS means that some companies and practices are making regulatory catch-ups.
“Most of the devices that were commonly used now have class 1 [categorization]Delaney said. “I know at least one, but perhaps a lot of others are now in a hurry to start class 2A.
Whether a device should be defined as a Class 2A medical device depends essentially on its intended purpose and level of clinical risk. The UK Medical Device Regulations may require reclassification as a Class 2A medical device, a category subject to stricter regulatory control, if the output of a tool is dependent on informing a care decision.
Anima Health is currently pursuing Class 2A registration along with other UK-based health technology companies.
UK AI for Health Push
The UK government is embracing the possibilities of AI in healthcare and hopes to be able to boost the country's tense national health system.
In a recent “10 Years of Health Plan,” the UK government states that it aims to make the NHS the most AI-enabled care system in the world, aiming to use technology to reduce the burden on managers, support preventive care, and strengthen patients through technology.
However, deploying this technology in a way that meets current rules within an organization is complicated. Even the UK Health Minister appeared to suggest earlier this year that some physicians may be pushing restrictions on integrating AI technology into patient care.
“I've heard anecdotes about a pub truly going down the pub. Some clinicians are ahead of the game and already use ambient AI to use something like notes and things, even if practice and trust haven't caught up yet.”
“Right now, rather than encouraging it, there's a lot of issues out there, but they say, 'Oh, people don't want to change, the staff are very happy and they're really resistant to change.' That's the opposite.
AI Tech certainly has great potential to dramatically improve speed, accuracy, and access to care, especially in areas such as diagnosis, medical record management, reaching patients in a lack of resources or remote settings. However, walking the line between high-tech possibilities and risk is challenging in sectors like healthcare, which can deal with sensitive data and cause serious harm.
Looking back at his experience, the patient said luck: “In general, I think we need to use AI tools to support the NHS. There is a great possibility of saving money and time. However, LLM is still experimental and should be used with strict surveillance. This should be used as an excuse to pursue innovation, but when caution and surveillance is required.”
