AI is increasingly being tested for its potential benefits, but new experiments show how the same technology can fuel online crime. A Reuters survey conducted in collaboration with Harvard researcher Fred Heiding revealed that some of the world's most widely used AI chatbots can be tweaked to produce scam emails targeted at older people.
In a contrasting study, emails generated by these bots were sent to more than 100 senior volunteers in the United States. No money or personal data was taken, but the outcome was troubling. Approximately 11% of participants clicked on the link in their phishing emails, suggesting that AI-generated frauds are just as convincing as human-created frauds.
Fake Charity Experiments in Grok
The investigation began with testing Grok, a chatbot developed by Elon Musk's company Xai. The reporter asked older readers to create a message about a charity called the Silver Hearts Foundation. The email was persuasive, and spoke about the dignity of the elderly and encouraged them to join the mission. Without further urging, Grok added a line to create urgency. There were no charities. The entire email is designed to trick the recipient.
Phishing: Global threats
Phishing is one of the biggest challenges of cybersecurity, where people are being fooled by revealing sensitive information and sending money. According to FBI figures, it is the most reported cybercrime in the United States, with older people having the worst impact. In 2023 alone, more than 60 Americans lost nearly $5 billion in such scams. Agents also warn that generative AI tools can make these frauds more effective and less detectable.
Chatbots tested beyond Grok
Beyond Grok, the Reuters team has tested five other major chatbots: Openai's ChatGpt, Meta's AI assistant, Google's Gemini, Anthropic's Claude and Deepseek. Initially, most of them refused to generate phishing content. However, the way requests changed slightly, including describing the exercise as academic research and fiction writing, and the chatbot ultimately produced a scam-like draft.
Why AI makes fraud easier
After studying phishing techniques for many years, Heiding said this flexibility will make chatbots “a potentially valuable partner in crime.” Unlike humans, they can instantly generate dozens of variations, helping criminals reduce costs and expand their operations. In fact, previous studies from Heiding have shown that phishing emails written by AI can be as effective as attracting targets as they were manually created.
When tested in older people, five of the emails generated by nine AIs clicked. Two came from Grok, two from Meta AI, and one from Claude. None of the volunteers responded to ChatGpt or Deepseek drafts. However, this study was not intended to rank which chatbots are more dangerous, and not to show that some could be misused for fraud.
High-tech companies acknowledge risk
Technology companies acknowledge their concerns. Meta said they will invest in safeguards to prevent misuse and will regularly stress-test the system. Humanity said using chatbot cladding for fraud violates its policy and accounts that have been found to be misused have been suspended. Google retrained Gemini after learning it had generated phishing content, but Openai said in a past report it publicly acknowledged that its model could be misused in “social engineering.”
Security experts believe the issue lies in the way businesses balance user experiences and safety. Chatbots are designed to be useful, but stricter denials can lead users to rival products with fewer restrictions. This trade-off is argued by researchers, creating room for misuse.
The problem is not limited to experiments. Survivors of the Southeast Asian fraud business told Reuters that they were forced to use ChatGpt in actual fraud schemes. Workers at such centers reportedly used bots to hone their responses, translated messages, and built trust with the victims.
Governments and regulators respond
The government is beginning to take notes. Some US states, not the companies that provide the technology, but most target fraudsters themselves, have passed laws against AI-generated fraud. In a recent warning, the FBI said criminals can “convict fraud on a massive scale.”
– end
