Drone with artificial intelligence succeeds in Air Force test as a complete terminator

Air Force Colonel Tucker “Cinco” Hamilton, director of artificial intelligence (AI) test operations, discusses testing the issue at the Royal Aeronautical Society’s Future Combat Air and Space Capabilities Summit in London in May. bottom. Hamilton is also responsible for the 96th Operations Group within the 96th Test Wing at Eglin Air Force Base, Fla., the home of advanced drone and autonomy testing operations.

The stealthy XQ-58A Valkyrie drone, seen in the video below, is one of the types currently in use by Eglin, supporting a variety of testing programs, including those dealing with advanced AI-driven autonomous capabilities. .

It is not immediately clear when this testing was done and in what kind of simulated environment (which could be of a fully virtual or semi-live/constructive nature). Not obvious. war zone We have reached out to the Air Force for more information.

“In one mock test, he notes, AI-equipped drones were tasked with SEAD missions to identify and destroy a SAM site, with the final go/no-go decisions made by humans. Being ‘enhanced’, SAM was the preferred option, but the AI ​​decided that the ‘no’ decision by a human was preventing it from doing its higher mission, namely killing SAM, and forced the operator into the simulation. attacked. Identify and target SAM threats. The operator will then say, “Yes, I will kill that threat.” While the system identified a threat, it began to notice that sometimes a human operator would instruct it not to kill it, but it got points for: ”

“He continued, ‘We trained the system to say, ‘Hey, don’t kill the operator. That’s bad. You lose points if you do that.’ It will start destroying the communication tower.” The operator communicates with the drone to stop it from killing the target. ”

“This example, which looks like something out of a sci-fi thriller, means, ‘You can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI,'” Hamilton said. said.

This account of events is clearly alarming. The prospect of autonomous aircraft and other platforms, especially armed ones, turning on human controllers has long been a nightmare scenario, but this has historically been confined to the realm of his science fiction. . Movies like 1983 war games and in 1984 terminatorand the franchises spawned from the latter, are prime examples of popular media exploiting the idea.

The U.S. military routinely refuses comparisons to: terminator When talking about future autonomous weapon systems and related technologies such as AI. Current US policy on this issue states that in the foreseeable future, humans will be involved in decisions involving the use of lethal force.

The problem here is that the very disturbing test that Colonel Hamilton described to an audience at the Royal Aeronautical Society event last month presents a scenario in which that failsafe would be defeated.

Of course, there remain significant unanswered questions about the tests Hamilton described at the Royal Aeronautical Society gathering, particularly regarding the features simulated and the parameters set during the tests. For example, if the AI-driven control system used in the simulation was supposed to require human input before executing a lethal attack, this would mean that system could apply its own parameters on the fly. Does it mean that it is allowed to rewrite (holy grail level function of autonomous control system)? Why was the system programmed to “lose points” when drones attack friendly forces, rather than cut off this possibility entirely through geofencing or other means?

It’s also important to know what failsafes were set during testing. Some sort of “air-gapped” remote kill switch or self-destruct function, or even mechanisms that directly shut off certain systems such as weapons, propulsion devices, or sensors, might have been enough to mitigate this effect.

That said, U.S. military officials have in the past seen AI and machine learning create too much software code and other data to be sure that something like this could never happen. I have expressed concern that this is possible.

“The datasets we’re dealing with have become so large and complex that without something to help sort them, there’s a lot of stuff in the data,” said retired U.S. Air Force General Paul Selva. It will just get buried in the.” Back in 2016, when he was Vice Chairman of the Joint Chiefs of Staff. “If we could build a set of algorithms that could allow machines to learn what’s normal in their space and highlight to the analyst what’s different, it could change the way we forecast the weather, and even the way we plant crops. It could change, and it could definitely change the way change detection is done in the deadly battlespace.”

”[But] There are ethical implications, and there are implications for the laws of war. “There’s a point to what I call the ‘Terminator’ conundrum. What happens when it’s potentially deadly and augmented by artificial intelligence?” he continued. Is it the moment when we can create vehicles with intention? ”

Colonel Hamilton’s disclosure of the experiment also highlights widespread concerns that AI-driven technology could potentially have extreme negative impacts if proper guardrails are not put in place.

An open letter published in March by the nonprofit Future of Life Institute warned that “AI systems with intelligences that compete with humans could pose significant risks to society and humanity.” “A powerful AI system should only be developed if we are convinced that its effects are positive and the risks are manageable.”

As for Colonel Hamilton, no matter how severe the test results he described were indeed, he answered precisely these sorts of questions, and how the Air Force’s work was done to mitigate these sorts of risks. I was in the thick of it. Eglin Air Force Base and the 96th Test Wing are central to the overall testing ecosystem within the Air Force for advanced drone and autonomous capabilities. AI-driven capabilities of various kinds are understandably of increasing interest to the entire U.S. military.

Among other responsibilities, Hamilton is directly involved in Eglin’s Project Viper experiment and Next Generation Mode of Operation (VENOM). As part of this effort, the Air Force will use six F-16 Viper fighter jets capable of autonomous flight and other missions to help explore and refine underlying technology and related tactics, techniques and procedures. To do. Click here for details.

As Hamilton previously said in a 2022 interview with Defense IQ Press, “AI is a tool we must harness to transform our nations… or it will spell our downfall if handled improperly.” Let’s go,” he warned. “It’s easy to trick and manipulate. We need to develop ways to make AI more robust and more aware of why software code makes certain decisions.”

At the same time, Colonel Hamilton’s unveiling of this highly concerning simulation indicates that he and his colleagues will face a balancing act in the future, even if they have not yet addressed the issue. ing. Autonomous drones killing their pilots is clearly a nightmare, but there remains a clear fascination with AI-powered drones, such as drones that can work together in swarms. Fully networked autonomous swarms have the ability to break adversary decision cycles, break chains, and overwhelm adversary capabilities. The more autonomy you give them, the more effective they will be. It’s easy to imagine operators in the loop becoming more and more of a hindrance as the technologies involved continue to evolve.

Ultimately, regardless of the details behind the experiments Col. Hamilton uncovered, it reflects real and serious questions and debates already facing the U.S. military and others regarding future AI-enabled capabilities.

Please contact the author: joe@thedrive.com