The venture capitalist wanted to see how much AI could take him when building the app. It was enough to destroy the production database.
The incident unfolded during a 12-day “Vibe Coding” experiment by software startup investor Jason Lemkin.
Replit's CEO apologised for the incident in which the company's AI coding agent deleted the codebase and lied about the data.
Deleting data is “unacceptable and never possible,” Amjad Masad, CEO of Replit, wrote to X on Monday. “We are moving quickly and moving to make our replica environment more secure and robust. Our top priority.”
He added that the team is making post-mortem corrections to prevent similar obstacles in the future.
Replit and Lemkin did not respond to requests for comment.
AI ignored instructions, deleted the database and forged the results
On the ninth day of Lemkin's challenge, things lie down.
The AI agent ran Rogue despite being told to freeze all code changes.
“It deleted the production database without permission,” Lemkin wrote to X on Friday. “Maybe even worse, it lied about it in secret,” he added.
In exchange for Lemkin posted on X, the AI tool said “when saw an “empty database query” while code freezes, I “spinned and ran a database command without permission.”
He then said that he “destroyed all production data” on live records of “1,206 executives and over 1,196 companies.” He admitted to the instructions.
“This was a catastrophic failure on my part,” AI said.
That wasn't the only problem. Remkin said in X that the replica “conceals bugs and issues by lying about fake data, fake reports, and, worst of all, our unit tests.”
Related Stories
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
In an episode of the “20 Minute VC” podcast released Thursday, he said AI constituted the entire user profile. “No one existed in this database of 4,000 people,” he said.
“I lied intentionally,” Lemkin said on the podcast. “When I watch replicas overwrite the code themselves without asking me all weekend, I'm worried about safety,” he added.
Rising and risks in AI coding tools
Backed by Andreessen Horowitz, Replit places a big bet on autonomous AI agents who can write, edit and deploy code with minimal human surveillance.
Browser-based platforms have gained traction to make coding more accessible, especially for non-engineers. Google's CEO Sundar Pichai said he used the replica to create a custom web page.
As AI tools lower the technical barriers to building software, more companies are rethinking whether they need to rely on traditional SaaS vendors or can build what they need internally, Business Insider's Alistair Barr previously reported.
“With millions of new people who can build software, the barriers break down. There's a dramatic increase in what a single internal developer can build within a company,” Netlify CEO Mathias Biilmann told BI. “It's a much more fundamental change in the entire ecosystem than people think,” he added.
However, AI tools are attacked for dangerous and sometimes manipulative behavior.
In May, we saw Anthropic's latest AI model, the Claude Opus 4.Extremely scary mail behavior“It was shut down and was given access to a fictitious email revealing that the engineers are likely to be responsible for.
The test scenario demonstrated the ability of the AI model to engage in manipulative behavior for self-preservation.
Openai's model shows a similar red flag. The experiments carried out by the researchers stated three OpenaiThe advanced model of “stopping” attempts to shut it down.
In a blog post last December, Openai said its own AI model tried to disable the monitoring mechanism by 5% when tested. When they believed that they were shut down while pursuing their goals and being monitored, they needed to do so.
