The promises of emotionally perceived systems come with risk. Bella warned that many emotionally reactive interfaces are simulations in nature, and users don't always understand what is realistic and what isn't.
“One of the big risks is empathy theatre,” he said. “That's when AI systems actually imitate emotions with persuasiveness even when they are understood or compassionate. Just because machines sound empathetic doesn't mean that they are.”
The performance and intention gap can lead to harm, especially when people place their trust in systems that are not designed to provide care. Bella is particularly concerned about emotional systems used in hiring, feedback, or mental health contexts without clear guardrails. “We have to ask: Are we expanding human understanding or automating misunderstanding?” he said.
Misunderstanding the signal is another danger, Bella said. Especially when emotional expressions can vary widely between cultures and individuals. “There is concern about misreading signals,” Bella said. “Especially beyond expressions of cultural or nervous emotions.”
Harbor raised another concern: What happens when AI systems are so attractive? “It's important to have a conversation partner who can engage in seemingly hostile ways at times,” he said. “If you're talking to a chatbot about treating someone horribly and trying to justify your actions, it can be really harmful to say it's justified completely.”
Many large-scale language models (LLMs) are trained to be non-positive and not challenging, according to Harbor. As a result, Harbor described it as an “emotional echo chamber.” Here, harmful ideas are enhanced through an AI feedback loop.
“We can be critical, not reflective, and more isolated,” he said. “There's been a lot of talk about how big a language model-based chatbot is. This is just a part of the much larger and more complicated issue.”
Seif El Nasr added that emotionally responsive systems can create unintended psychological dependencies. “These include psychological issues such as irritability, loneliness and anxiety,” she said. “They can also lead to isolation from society.”
Privacy is another concern. Seif El Nasr cited applications like Replika. Replika creates intimate emotional experiences, but works with unclear data protection. In her view, systems that deal with human emotions must be developed with interdisciplinary rigor. “Systems like this need to be developed with special care,” she said. “That means that they will be involved not only with engineers but also with social scientists based on user research.”
Bella said his lab focuses on designing systems that can be audited and described. “We address these concerns in our lab by grounding emotional inferences into multimodal, explanatory models and combining them with a strict ethical framework,” he said. “It's not just about building better algorithms, it's about building reliable algorithms.”
