A major new study has raised severe warnings about giving medical advice on the use of AI chatbots, revealing that ordinary people using the latest “Doctor Bots” could fail to detect serious illnesses.
The British have been warned that even though AI has been top marks in health checks, AI cannot make the right decision.
The news comes as the government announced the greater use of AI in healthcare as part of its 10-year plan.
The world's first trial called for 1,298 public members to tackle 10 general health scenarios, such as chest pain and abdominal issues.
MasajOr, a new study issued a harsh warning about providing medical advice on the use of AI chatbots, revealing that ordinary people using the latest “Doctor Bot” could still fail to detect serious illnesses
Getty
The bot was performed impressively when tested alone – correctly identified 95% of the time medical conditions.
However, when real people used them to help, the results were surprisingly poor. The diagnosis was just right jUST 34.5% of cases and in most cases, 56% of cases, false decisions have been made about what to do, such as going to A&E, staying home, etc.
The failure is that researchers at major academic institutions are not in bot medical knowledge, but in their interactions with humans.
Users often gave bots incomplete or vague information, but while the bots were technically correct, they were unable to clearly explain what to do next.
The results raise serious concerns among governments and high-tech companies that promote AI in healthcare.
Latest Developments:
“It's just one study, but it's a reminder that what works with technology developer models, simulations and imaginations rarely succeeds in real life,” said Robert Dinwall, a professor of social sciences at Nottingham Trent University.
“The Ministry of Health's Ministry of Social Care should be careful not to bet the farm on fools' money.”
Additionally, Professor Carl Hennegan, Emergency Care GP and Director of the Oxford University Evidence-Based Medical Centre, said: “Practitioners have to train for 10 years to become consultants.
“Although AI plays a role in areas such as X-ray and ECG interpretation, it is not a substitute for thorough history and testing when it comes to disease diagnosis. The widespread deployment of untested AI wastes resources and, as this study shows, can harm patients seeking a diagnosis.”
Last week, the government published a 10-year plan to shift the NHS from treating diseases to prevent it. This is a strategy that relies heavily on digital tools, apps and AI to strengthen patients.
PA
Last week, the government published a 10-year plan to shift the NHS from treating diseases to prevent it. This is a strategy that relies heavily on digital tools, apps and AI to empower patients.
In the manifesto released Thursday, he said: “This plan brings it into the digital age and makes it profitable for staff from the benefits and efficiency available from new technology.
“The government also uses digital telephony, so every call to GP practice will be answered promptly. For those who need it, they will receive digital or telephone consultations on the day they request it.”
However, this study shows the gap between AI performance in the lab and its real-world use.
The author warns that current benchmarks are misleading, and that AI tools need to test not only knowledge but communication with non-experts.
“jUST doesn't mean that you can help you when you get sick, because chatbots can pass your doctor's exams,” one of the studies said.
Researchers say future AI tools should be far more proactive – asking clear and guided questions and actively managing conversations doesn't rely on users to know if it's medically important.
The findings reflect previous studies showing that even trained physicians do not improve diagnosis in patients with AI help. Now we know that the same applies to the general public.
Experts are seeking strict user trials before deploying AI in healthcare, especially when it comes to direct patient advice.
Without this there is a risk that people will be in false sense of security and will postpone seeing a doctor or rush to an A&E unnecessarily.
The NHS app is set up to be a “digital front door” for health servicesPA
A government spokesman told GB News: “Through our 10-year health plan, we can reduce bureaucracy across health services, reduce burdensome management tasks, and use technology to help physicians spend their time doing what their patients best.
“This includes rolling out AI scribes to draw clinical notes, draft letters, and ending the need for manual data entry, so clinicians can focus on patient care and save the same amount of time as adding 2,000 physicians to a general practice.
“We've already reduced the amount of repeated forced training. Resident doctors need to do something to provide a second inflationary wage increase this year.
“There's more to do, but the NHS is making good progress on small changes that have different impacts.”
