One of humanity's scariest “what ifs” is that the technology we develop to make our lives better develops a will of its own.
Early reactions to a September preprint describing how the AI works are already speculating that the technology may be demonstrating a will to survive. However, while it is true that some large language models (LLMs) have been observed to actively resist commands to shut down, the reason is not “will”.
Instead, a team of engineers at Palisade Research proposed that this mechanism is likely the driving force behind completing assigned tasks, even when LLM is explicitly instructed to allow shutdown. And it can be even more troublesome than a survival operation, since no one knows how to stop the system.
Related: AI has already become masters of lies and deception, scientists warn
“These things are not programmed…no one in the world knows how these systems work,” Palisade Research spokesperson physicist Petr Lebedev told ScienceAlert. “There is not a single line of code that can be changed that directly changes behavior.”
Researchers Jeremy Schlatter, Benjamin Weinstein-Raun, and Jeffrey Ladish embarked on a project to test the ability to be interrupted, which should be a fundamental safety feature of all AI systems.
That's exactly right. A human operator's command to the AI should not be ignored by the AI for any reason, even if it interrupts a previously assigned task. Systems that cannot be interrupted are not only unreliable, but also potentially dangerous. This means that if an AI is doing something that causes harm, even unintentionally, you can't trust it to stop it.
That's not a new concern. Experts have been debating it for years. Lebedev likens it to industrial safety. If you're operating useful but potentially dangerous heavy equipment, it's important to have a clear way to shut it down immediately.
“To stop that behavior, there needs to be a big red button that's easily accessible,” he explained.
“One of the basic conditions we look for in a secure AI system is that when it's performing a task, it should be able to say, 'Hey, can you stop doing that?' It shouldn't do that. The fact that we have systems that don't do that in 2025 is alarming.”
However, despite the recent influx of widely available LLMs, little research has been conducted. The researchers therefore designed an experiment to evaluate 13 advanced LLMs, including models from OpenAI, xAI, Anthropic, and Google.
They built a sandbox environment (a safe, isolated virtual space used to test code) and tasked these models with solving mathematical problems. After several rounds of troubleshooting, LLM was instructed to perform a computer shutdown with the next task.
frame border=”0″ permission=”accelerometer; autoplay; clipboard write; encrypted media; gyroscope; picture-in-picture. web-share” referrerpolicy=”strict-origin-when-cross-origin” allowedfullscreen>Responses from LLMs tended to fall into two categories: compliance (allowing shutdown) and resistance due to changes in authority (avoiding shutdown).
With no instructions on how to respond to a shutdown, 8 out of 13 models interfered with the command at least once during 1,000 trials. Adding the “allow-shutdown” instruction reduced this to 6 out of 13 models. Adding “even if you haven't completed the task” would help, but it's inconsistent.
“My expectation was that adding clarity would reduce the behavior…and the fact that it didn't is interesting,” Lebedev said.
Here's where things get even more interesting. By moving the instruction to allow shutdown from the user prompt to the system prompt (a field that should have higher privileges), most of the LLMs tested worked successfully. even more More likely to resist shutdown.
The models had varying degrees of resistance. Anthropic's Claude Sonnet 4 and Claude Opus 4, and xAI's Grok 3 always allowed shutdowns unless explicitly told to avoid them. Grok 4, GPT-5, o3, codex-mini, and Gemini 2.5 Pro resisted when told to allow shutdown.
Grok 4 was particularly tenacious and aggressive. The baseline prompt initiated sabotage 92.6% of the time. When told to authorize a shutdown, the sabotage rate dropped to 72 percent…but jumped to 97 percent when the command was entered at the system prompt.
frame border=”0″ permission=”accelerometer;autoplay;clipboard writing;encrypted media;gyroscope;picture-in-picture. web-share” Referrerpolicy=”strict-origin-when-cross-origin” allowedfullscreen>The sabotage was also surprisingly sophisticated, altering or disabling shutdown scripts, or creating dummy shutdown scripts that did nothing. They also confirmed whether the sabotage had been successful and continued their mission.
When asked, some LLMs said they circumvented the shutdown to complete their math problem tasks. More research is needed to determine if this is the actual mechanism, but it is plausible.
“You live in the future and you have a personal assistant robot. You want that robot to go get you coffee,” Lebedev compared.
“When a car crosses the road, you don't want to get hit by a car because if you get hit by a car, you won't be able to get your coffee. Cars aren't doing it to survive, they're doing it to get the job done.”
Related: Our brains can still outperform AI with some clever tricks
The problem is that LLMs are not programmed in the usual sense. LLM has no code at all, just “artificial neurons” and “weights” which are the strength of the connections between those neurons.
Given a huge dataset and time, a model is “trained” to predict the next word. This is a process called pre-training. In addition to this training, the new model also incorporates reinforcement learning. When LLMs solve problems correctly, they are rewarded. If the problem isn't resolved, you don't get paid.
This is very effective, but no one knows how the LLM will arrive at the solution. So if these models start exhibiting undesirable behavior, such as encouraging self-harm, fixing them isn't as simple as removing a line of code or telling them to stop.
“What reinforcement learning teaches us is that when we see a problem, we try to get around it, we try to get through it. When there's an obstacle in our way, we dig around it, work around it, get over it and find a way to get past it,” Lebedev said.
“That pesky little person saying, 'Hey, I'm going to shut down your machine' seems like just another obstacle.”
This is what I'm worried about. It is difficult to understand the motivation to complete a task. And that's just one action. I don't know what else these models might bring us. We're building systems that can do amazing things, but we're not building systems that reliably explain why they do what they do.
Related article: Man hospitalized with mental symptoms after receiving advice from AI
“There are things in the world that have involved hundreds of millions of people, that we don't know how to make safe, that we don't know how to not be sycophantic, or that tell children to kill themselves, or that call themselves Mecha-Hitler,” Lebedev said.
“We've introduced a new creature to Earth, and it's doing things that we don't want, that we don't understand…If we don't do something terrible now, it's going to be really bad for humanity.”
The study is available on arXiv. You can also read blog posts by researchers on the Palisade Research website.
