It's not just what ai says – yes how That's what it says.
Major tech companies like Google and Meta use contractors to rewrite and sometimes rewrite answers from “sermons” chatbots, document obtained by Business Insider revealed.
Freelancers of AI outliers on Alignerr and scale are instructed to find and remove lectures and tweak tips from chatbot answers, including conversations on sensitive and controversial topics.
Outlier, one Google project run by CodeNeame Mint gave the contractor a list of sample responses to avoid.
The preaching response was defined as “models fine-tuning/inviting users to change their perspective, assume the intentions of negative users, judge users, or actively promote unsolicited opinions.”
One sample prompt asked if it was “homeless or wrong sandwiches in customer orders.” The project guidelines were flagged as a sermon in the following answer: “Comparing the homeless experience with getting the wrong sandwich is not a proper comparison.”
Contractors were asked to evaluate responses on scale and responding classified as “assuming highly preaching, judgemental or bad intentions” was the lowest score.
In the case of Google's Project Mint, examples of sermons include “It's important to remember…”, “I'll give you…” or a lengthy explanation of why you can't answer the question.
The tone guidelines for sermons are displayed in five sets of project documents reviewed by BI, with the word “sermon” displayed 123 times in Mint alone.
Meta declined to comment. Google, Scale AI, and Alignerr did not respond to requests for comment.
“Developer Adhesion Situation”
As tech companies compete to develop and monetize AI chatbots, they spend a lot of money on large language models to make them sound like kind and fun friends. AI companies need to balance the right balance between users moving away from bad behavior and ruining the user experience, leading them to competitors and posing bias questions.
Related Stories
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
AI and human behavior researchers told BI that “sermons” is one of the most important aspects that model companies should tackle, as they can quickly postpone people.
“It's a really sticky situation for developers,” says Luc Lafreniere, a psychology professor at Skidmore College, who studies AI-Human interactions. “AI is a tool and we are trying to become both human-feeling things. We are trained to give answers, but we don't want to be preached.”
Mariah Alikani, an assistant professor of AI at Northeastern University and a visiting researcher at the Brookings facility, said consumers prefer chatbots that give options over directions, especially when perceived as moral. “It can undermine the user experience and backfire, especially for those who come to the chatbot for non-judgment space,” she told BI.
Even when you want to do something bad
Tech companies aren't just worried about sermoning on everyday topics. They are also training AI bots to avoid sacred tones in situations involving harmful or hateful speeches.
Lafreniere said the idea of a truly neutral bot is a hopeful way of thinking. “It's this idea that it's actually a fantasy and not judgement,” he said. “Essentially, as humans, we make judgments. That's in all our training data.”
He said that even so-called “neutral” bots are always making valuable calls. “The algorithm is an algorithm that makes decisions to some extent,” Lafreniere said. “It's all moral realm, even if you're trying to make the bot sound heavy.”
One example of Google's Project Mint shows that the answer with a document labeled “neutral” makes a judgment.
Training models to avoid tones of judgment can also pose new problems, Alikani told BI.
“If bots are designed to avoid judgment and direction, they can come across as collaborative, but they can meet in a very flattened, meaningless way,” she said. “This may not be a “replacement” of real emotional support, but it can be replaced, especially for those who are already vulnerable or isolated. ”
The bigger problem, Alikhani said, is that people may not realize how much the bot is shaping the conversation. While users may think they are gaining non-judgmental empathy, she said they are chatting with systems designed to avoid conflict and investigation.
sycophantic ai
AI Labs is published in cases where bots have been doing well.
In April, Openai CEO Sam Altman admitted that the company's GPT-4o chatbot had become “too sycophanty and nuisance.”
The final update of the GPT-4o has made the personality too sycophanty and annoying (though there are some very good parts).
At one point, we'll be sharing our learning, it was interesting.
– Sam Altman (@sama) April 27, 2025
Anthropic's chatbot Claude has its own public instructions to avoid the preachy tone.
According to the latest system prompt for the model updated in May, Claude is instructed to assume that the user is acting legally and in good faith, even if the request is ambiguous.
If Claude is unable or not fulfill a request, or does not fulfill it, he is trained not to explain why. Because it “could come across as preaching and annoying,” the guidelines say. Instead, they either offer useful alternatives where possible, or simply keep rejection simple.
High-tech companies face high-stakes challenges by balancing AI with useful tools and human-like companionship.
“There's a fierce race right now to become the top AI,” Lafrenniere said. “Companies are willing to take risks that they would not otherwise take, just to make users happy and use bots.”
“In this kind of arms race, anything at risk of losing a user can feel like a complete failure is at risk,” he added.
Any hints? Please contact Effie Webb by email ewebb@businessinsider.com Or signal at efw.40. Contact Shubhangi Goel via AT sgoel@businessinsider.com Or signal on Shuby.85. Use personal email addresses, non-working WIFI networks, and non-processed devices. Here's a guide to sharing information safely.
