Hours earlier, President Donald Trump had ordered federal agencies to stop using Claude after a dispute with its manufacturer, but the tool was so deeply embedded in Pentagon systems that it would take months to resolve the issue in favor of a more docile rival. It was also used in the January operation that led to the capture of Nicolas Maduro.
Also read | Israeli airstrikes, Iranian leadership vacuum, and President Trump’s latest moves: What you need to know on day 5
But what do “intelligence assessment” and “target identification” actually mean? Was Claude flagging a location to attack or estimating casualties? No one has disclosed such information and, surprisingly, no one is under any such obligation.
Artificial intelligence has long been used in warfare, including analyzing satellite images, detecting cyber threats, and guiding missile defense systems. But the use of chatbots is the same underlying technology that billions of people use for everyday tasks like composing emails, and is now being used on the battlefield.
Last November, Anthropic partnered with Palantir Technologies Inc., a data analytics company that does a lot of work for the Department of Defense, to turn its large-scale language model Claude into an inference engine within military decision support systems.
And in January, Anthropic submitted a $100 million proposal to the Department of Defense to develop voice-controlled autonomous drone swarm technology, Bloomberg News reported. The company’s pitch: use Claude to translate a commander’s intent into digital instructions to coordinate a fleet of drones. That bid was rejected, but the contest asked for more than just a summary of the intelligence report you’d expect from a chatbot. The contract was to develop “target-related awareness and sharing” and “launch-to-termination” for potentially lethal drone swarms.
Amazingly, all of this is happening in a regulatory vacuum and using technology that is known to make mistakes. The hallucinations of large-scale language models are the result of training that rewards knowing the answer in exchange for acknowledging uncertainty. Some scientists say the persistent challenges of AI writing may never be resolved.
This is not the first time unreliable AI systems have been used in war. Lavender is an AI-driven database used to identify military targets associated with Hamas in Gaza. Although it wasn’t a large-scale language model, it analyzed vast amounts of monitoring data, such as social connections and location history, to assign each individual a score from 1 to 100. If someone’s score exceeded a certain threshold, Lavender flagged that person as a military target.
The problem, according to an investigation report published by Israeli-Palestinian media outlet +972, is that Lavender was wrong 10% of the time. “About 3,600 people were mistakenly targeted,” Maria Rosalia Tadeo, professor of digital ethics and defense technology at the Oxford Internet Institute, told me.
“These systems have incredible vulnerabilities and are extremely unreliable for something as dynamic, sensitive and human as war,” says Elke Schwartz, professor of political theory at Queen Mary University of London and author of Death Machines: The Ethics of Violent Technologies.
Schwartz points out that AI is often used in wars to speed things up, with undesirable consequences. Enable faster decision-making at larger scale and with less human scrutiny. Over the past decade and a half, military uses of AI have become more uncertain, she says.
And even before it is applied to war, secrecy is built into the way AI laboratories operate. These companies refuse to disclose what data their models were trained on or how their systems reach their conclusions.
Of course, military operations often have to be conducted in secret to protect the combatants and keep the enemy from getting their scent. However, defense is tightly regulated by international humanitarian law and weapons testing standards, and in theory must also address the use of artificial intelligence. But such standards are missing or woefully inadequate.
Taddeo points out that Article 36 of the Geneva Convention requires new weapons systems to be tested before deployment, but an AI system that learns from its environment becomes a new system each time it is updated. This makes applying this rule nearly impossible.
In an ideal world, governments like the United States would disclose how these systems are used on the battlefield, but there is precedent. Americans began using armed drones after 9/11 and expanded their use under President Barack Obama’s administration, but have refused to acknowledge the existence of such programs.
It took nearly 15 years of leaked documents, pressure from the press, and continued litigation from the American Civil Liberties Union before President Obama finally released the death toll from the drone strikes in 2016. Although these were widely considered to be undercounts, they allowed the public, parliament, and the media to hold the government to account for the first time.
The crackdown on AI will only get tougher, and more pressure from the public and legislators will be needed to force the recalcitrant Trump administration to create a similar reporting framework.
Schwartz said the goal is not to reveal exactly how Claude was used in operations such as Operation Epic Fury, but to outline the general outlines. And especially to disclose if something goes wrong.
The current public debate about the conflict between humanity and the Department of Defense, about what is legal and ethical for AI when it comes to mass surveillance of American citizens and the development of fully autonomous weapons, is missing a larger question about the lack of visibility into how the technology is already being used in war. This is very necessary, as new and untested systems like this are prone to mistakes. “We haven’t decided yet whether we as a society should allow machines to decide whether or not to kill humans,” Taddeo said.
It’s important to promote transparency about AI in war before it becomes so commonplace that no one cares to ask questions. Otherwise, you may end up waiting for a fatal mistake to occur and imposing transparency only after the damage has been done.
