Governments around the world want AI to do more of the heavy lifting when it comes to public services. The plan appears to be aimed at making things more efficient, with algorithms quietly handling the day-to-day management of the country.
For example, AI could help tackle tax fraud by devising ways to target those most likely to be in violation. Or it could help public health services test for different cancers, triage cases at scale, and alert patients deemed most at risk.
But what if such a triage system makes a mistake? Or what if a government agency deploys AI to identify fraud and its model simply gets it wrong?
There is already sobering evidence that errors in AI can have devastating consequences. In the Netherlands, for example, a flawed algorithmic assessment of tax evasion was dealt with in a way that tore families apart and separated children from their parents.
In this case, we used a risk scoring system to identify families we thought were likely to be committing benefit fraud. These ratings were then input into automated operations that ordered repayments and drove innocent households into financial ruin.
Therefore, countries should be extremely cautious about replacing human judgment with AI. The assumption that machines will almost always get it right is simply not true. People’s lives cannot be easily reduced to data points from which algorithms draw conclusions.
And who is to blame if things go wrong? What about human responsibility?
These are the kinds of questions that are often overlooked amidst all the noise and huge levels of investment that AI attracts. But even if we set aside the possibility that this is another speculative bubble about to burst, there is growing evidence that AI in its current form is not delivering on its promise. The problem of “hallucinations” – When AI generates content that is plausible but differs from reality. [remains unresolved] [https://dl.acm.org/doi/pdf/10.1145/3703155]and expensive development is often overwhelming.
Even leading figures in the industry, including the co-founders of OpenAI, acknowledge that simply making large language models (LLMs) bigger will not significantly improve the situation.
However, these systems are rapidly being incorporated into key areas of our lives, including law, journalism, and education.
It is not difficult to imagine a university of the future where lectures and assignments are generated by LLMs run by specific departments and absorbed and completed by LLMs run by students. Human learning could then become a byproduct of machine-to-machine communication, with long-term consequences potentially hollowing out the very institutions responsible for fostering critical thinking and expertise.
All in?
But all this integration has big benefits for AI companies. The more AI is integrated into public infrastructure and business operations, the more these companies will become essential and difficult to challenge and regulate.
For example, integration into the defense sector with the development of autonomous weapons could result in companies becoming too big to fail if a country’s military security depends on it.
And when things go wrong, the asymmetry in expertise between governments and citizens on the one hand and AI developers on the other only increases overall dependence on the very companies that own the systems that caused the problem in the first place.
To understand where this trajectory is heading, it’s worth looking back over the decades when social media companies first emerged. Its purpose was clearly a simple one: to unite people from all over the world.
But today, the influence and influence of some of these companies is a cause for significant privacy, surveillance, and manipulation concerns. Scandals have occurred on everything from undermining democracy to spreading misinformation to inciting violence.
But we now find ourselves experimenting with a powerful combination of social media, AI, and machine learning. While social media captures attention, an LLM can generate a tremendous amount of attention-grabbing content. Meanwhile, machine learning systems decide what each of us sees on our various screens, trapping us in ever-tighter information bubbles.
Studio George/Shutterstock
So, even if, for the sake of argument, AI evolves as promised, becoming more accurate, more robust, and more capable, should we really cede control of more areas of our lives to algorithmic adjustments in the pursuit of order and efficiency?
Technology alone cannot solve social, economic, and moral problems. If we could do that, no child would go hungry in a world where enough food is already produced to feed everyone.
AI critics are often dismissed as Luddites. However, this is a misreading of history. The Luddites were 19th century British textile workers who opposed automated machinery in some of the factories where they worked, but not against technology itself.
They simply objected to its misuse and unreflective deployment and called for deeper investigation into how technology reshapes work, communities, and everyday life. Almost 200 years later, that’s certainly still a reasonable request.
