The AI tools make potentially harmful mistakes on social work records, ranging from false warnings about suicidal ideation to simple “gibberish,” frontline workers said.
Keir Starmer last year defended social work transcription technology as an “incredible” time-saver. But research across 17 councils in England and Scotland, shared with the Guardian, found that AI-generated hallucinations were mixed in.
As many local authorities begin to use AI notetakers to speed up the recording and summarization of meetings with adult and child service users, an eight-month investigation by the Ada Lovelace Institute found that “some potentially harmful misrepresentations about people’s experiences are occurring in official care records”.
The independent think tank found that one social worker who used an AI transcription tool to create a summary said the technology erroneously “indicated there was suicidal ideation”, but that “the client…didn’t actually talk about any suicidal thoughts or plans”.
Another said the AI’s notes might be referring to “fish fingers, flies, or trees” when in reality the child was talking about his parents’ fight. Social work experts said such glitches are particularly concerning because they can potentially miss dangerous patterns of behavior.
Other social workers expressed concerns about inaccurate transcriptions of conversations with people with regional accents. One person explained that AI-generated transcriptions often contain “garbage.” Another said: “It’s become a bit of a joke in the office.”
Dozens of councils from Croydon to Redcar to Cleveland are giving social workers access to AI transcription tools that record and summarize case conversations. The potential time savings are attractive to city halls, which suffer from chronic staff shortages.
One popular system, called Magic Notes, is sold to councils at a cost of between £1.50 and £5 per hour of transcription. Most of the social workers interviewed used the specialized Magic Notes AI or the general-purpose Microsoft Copilot AI.
The study found that AI transcription provides measurable time savings, freeing up social workers to focus more on relationships with service users.
“Our evidence shows that these tools can also improve the interpersonal aspects of care work and the quality of information recorded by social workers,” the report said after interviews with 39 anonymous social workers.
But when a social worker used an AI tool to rewrite care documents with a more “person-centered” tone, the system inserted “all the words that weren’t said.” Another social worker reported that the technology “crossed the line between your assessment and the AI’s assessment.”
“Inaccuracies entered into these records by AI can have far-reaching effects, including social workers making incorrect decisions regarding child care, which can lead to harm to children and professional consequences for social workers,” the report concludes.
According to the British Association of Social Workers (BASW), the impact of AI mistakes is already being felt in the profession, with reports of disciplinary action for AI note takers failing to properly check their output or overlooking obvious mistakes. We are calling on social work regulators to issue clear guidance on when and how AI tools should be used.
While there is “real excitement” among some social workers about their potential, “these tools also pose new risks for social work and society, from potential bias in report summaries to inaccurate ‘hallucinations’ in records,” said Imogen Parker, associate director at Ada Lovelace. “These risks have not been adequately assessed or mitigated, leaving frontline workers to address these challenges themselves.”
Social workers often receive little AI training, in one case as little as an hour. Some social workers said it took up to an hour to check the AI’s records, while others said it took as little as two minutes. One said: “It literally took 5 minutes to do a simple test.” […] One person said cut-and-paste care plans generated by AI could be “scary.”
Others said some colleagues were too lazy or busy to check their records.
“The risk here is that people don’t check what’s written to them,” said Andrew Rees, head of strategy for England and Wales at BASW. “The time you spend writing helps you understand what you hear. When a computer does that for you, you miss out on an important part of reflective practice.”
Beam, which runs Magic Notes, emphasized that its work is a first draft, not a final record. “AI tools are being embraced by social workers for good reason,” said co-founder Seb Barker. “Services are overflowing and difficult to access. A generation of social workers is at risk of burnout and the need for accurate and compliant documentation is increasing.”
He said a bias assessment found that Magic Note worked “consistently and fairly” and highlighted social work-specific features such as automatic hallucination risk checks. “Not all AI tools are the same, and unspecialized, low-quality, or generic tools will not meet the specific needs of this field,” he said.
The UK government and Microsoft have been contacted for comment.
