While AI is making digital products more appealing, it is also increasing risks around addiction, mental health, and harmful customer behavior.
adobe stock
There’s nothing new about products designed to be addictive. The food and beverage industry spends millions of dollars developing flavors that keep customers coming back. And smoking and alcohol are huge industries built around addictive products.
But what about technology? Designers of software, tools, and apps clearly want to create “compelling” products that we want to use. But when does developing a product so great that we can’t imagine life without it become dangerous for both our customers and our business?
The most pressing question is: What are the implications for technology companies’ use of AI to learn about us, predict our behavior, and develop ever more addictive products?
In March, a California court shocked the tech industry by finding that both YouTube and Meta services were intentionally designed in a way that could harm mental health.
$6 million in damages may seem like small change for a tech giant. However, the situation could escalate rapidly, with hundreds or even thousands of similar incidents likely to follow. And if regulators fail to better define the boundaries between using AI to drive engagement and using it to cause addiction, small businesses that build tools using algorithms will also be blamed.
I believe these developments have shed light on perhaps the least understood aspect of responsible AI use: safety. In addition to ensuring that AI systems are reliable, secure, transparent, and accountable, businesses and society at large still have much to learn about the risks around mental health, addiction, and the dangerous paths AI can take us down.
From engagement to addiction
Modern digital products are continuously optimized by AI to meet your needs, and can respond in real time to the signals that are most likely to keep you scrolling or clicking.
A very obvious example is a content recommendation engine that chooses what appears next in your social feed. From a platform perspective, the most important question an algorithm must answer before deciding what to show is what is most likely to keep us paying attention?
Today, most people know the term “doomscrolling.” Why do people spend hours aimlessly browsing content when they have better things to do? Because out of the millions of content on the internet, they are being shown what AI finds most appealing.
This is very powerful and becomes a problem when it crosses the line from an attractive process to one that leads to addiction (a term used to describe mental health problems).
There is another side to this danger. Just as we understand that drug users can be driven to harder substances by softer gateway drugs, dynamic algorithms can drive users toward more extreme or addictive content. This may include content from groups with extreme ideologies or content that promotes misinformation or propaganda.
It has also been shown that the use of AI can lead to unhealthy behavioral patterns, even if it does not directly promote bad behavior. More and more people are becoming emotionally dependent on chatbots, blurring the line between beneficial interactions and unhealthy attachments.
For businesses, the risks are becoming clearer. Allowing AI systems to shape customer behavior in negative ways can be a significant liability.
So how should companies respond?
Regulators are already working on this. Legislation such as the EU Digital Services Act requires platforms and service operators to assess and mitigate systemic risks, including, among other things, impacts on mental health.
The UK Online Safety Act also states that when considering AI safety, it is not just harmful content that must be considered, but also harmful AI system design.
The first step for companies is to understand that this is a new category of risk. As well as data security and cybersecurity, you must also consider the potential for your system to have a detrimental effect on your users.
Most companies are not prepared to deal with this. But this does not absolve them of responsibility when AI causes harm in this way.
This could mean asking some tough questions. What are your AI systems actually optimized for? How do you tell when engagement becomes addictive? And what warning signs and indicators should you use to trigger human intervention?
Most importantly, if asked to do so, can you demonstrate that you have considered all of this when assessing risk?
This risk will become more real and real as more companies adopt algorithmic methods to increase user engagement and retention. So now is the time to start thinking about how to respond to it.
