[Alexander Blanchard is Senior Researcher in the Governance of AI Programme at the Stockholm International Peace Research Institute (SIPRI), Sweden]
In recent weeks,
there has been a good deal of commentary about military applications of
artificial intelligence (AI), prompted by the US military’s public spat
with the AI company Anthropic and the use of AI in its war
on Iran. But another set of headlines also merits attention for those concerned
with the global governance of military AI. Last month, juries in US courts
found two of the most powerful tech companies – Meta and Google – legally
responsible for the harms caused by their platforms to young people. These
cases offer rare insight into the design choices taken by companies providing software
products for military use, and with it important lessons about the role of the
tech industry in realising aspirations for responsible governance of AI in the
military domain. This includes lessons about the practical challenges of
accountability, as well as the need for governance debate to recognise that
problematic human-machine interactions may be deliberately engineered.
The rise of the
military-tech complex and the role of platform companies
Many states that
see the military adoption of AI as a strategic priority are unable to develop many
AI capabilities in-house due to a lack of capital and expertise. Increasingly, these
states are turning to technology firms to provide data services and expertise.
This is leading to the emergence of the so-called military-tech
complex – a series of close partnerships between armed forces, governments, and
technology firms to integrate AI and data analytics into military operations.
The rise of the military-tech complex has seen the defence industrial landscape transform significantly over the last few years, with new commercial relationships being formed and many new defence AI startups appearing on the scene. But a major centre for defence industrial work remains Silicon Valley, which has long been wooed by the defence establishment. This includes those large, globally dominant platform companies like Microsoft, Alphabet (Google), Amazon, and Meta (Facebook) often referred to as ‘big tech’. Google, for instance, was early on associated with Project Maven, the US army’s flagship AI-enabled targeting support system, providing tools to support the integration of machine learning processes into the army’s organisational practices; Meta made its Llama AI models available for defence applications in 2024, and recently announced a partnership with the defence neo-prime Anduril to provide virtual and augmented reality devices to US armed forces.
What
makes these platform companies significant is not just their long-term
involvement in providing products and services to armed forces, but their
infrastructural power: often they own and operate
the foundational layers of hardware and cloud infrastructure that support cutting-edge
applications of AI. This was recently illustrated by the Israeli military’s use of
Microsoft’s cloud infrastructure to support its mass surveillance program of
Palestinians before, following journalistic scrutiny, the program was moved to
Amazon’s cloud infrastructure.
In military AI governance debate there is uniform recognition that design choices made throughout the lifecycle of AI development and implementation impacts the ability of armed forces to use it in line with relevant legal and ethical frameworks. Recognising this, states have been keen to involve industry in governance efforts on these technologies. Given the infrastructural power of platform companies, and given the potentially diffuse use of their technologies in military settings, it will have a significant influence on whether the aspirations and obligations entailed by those frameworks are realised.
A big tobacco
moment for big tech: from content to design
In the final
week of March, we were granted a glimpse at how two of these companies settle
their design choices. In the span of
just two days, juries in two separate trials in the US found Meta and Google
legally responsible for the harms caused by their platforms to young people. In
a case heard in Los Angeles (LA), the plaintiff argued that social media sites Instagram
(Meta) and YouTube (Google/Alphabet) had been intentionally designed with addictive
features to get users hooked on them. The LA verdict came a day after a jury in
New Mexico found Meta liable for the way in which its platforms endangered
children and exposed them to sexually explicit material and contact with sexual
predators.
The
judgment in the LA case is something of a landmark because it represents a
significant shift in how courts assess the responsibility of platform
companies. Traditionally, it has been difficult to prosecute online platforms in
the US for harmful content because of the protections provided by Section 230 of the
Communications Decency Act, which shields them from liability for user-generated
content by treating them as neutral intermediaries. Signed into law in 1996, there
have been calls to repeal
the act, with critics arguing it is poorly suited to an internet era dominated
by big data and algorithmically-generated content.
However,
by focusing on the design features rather than the content of Instagram and
YouTube, the plaintiff in the trial in LA were able to side-step the
protections offered by Section 230. The argument the plaintiff’s lawyers made
was that Instagram and YouTube are defective products. Features such as
infinite scroll, algorithmic recommender systems, vanishing, time-sensitive
content, and autoplay were highlighted as deliberate mechanisms with addictive
functions to keep users on these platforms for extended periods. It was alleged
that the companies borrowed heavily from the behavioural and
neurobiological techniques used by poker machines to get young people hooked and
to drive advertising revenues. Meta’s internal communications compared the platform’s
effects to pushing drugs and gambling, whilst an internal memo written by
YouTube staff reportedly described
“viewer addiction” as the goal. The jury accepted that these design choices
encourage compulsive behaviour. The jury also upheld the claim of negligence:
these companies knew that their products were harmful yet failed to warn users
or mitigate those risks. Google and Meta have said they plan to appeal.
These verdicts are the first of their kind and could mark a ‘big tobacco moment’ for big tech (the period in the 1990s when public opinion turned against tobacco products), with thousands more similar cases waiting to go to trial. They are part of a broader shift in public opinion about the role of technology companies in our lives, and they challenge a long-standing narrative that big tech is too big to regulate. As one of the lawyers for the plaintiff in the Los Angeles case put it: “accountability has arrived.”
Lessons for
military AI governance: accountability and engineered dependency
Seemingly so
far from the world of drones, bombs, and tanks, what can these two verdicts tell
us about the significance of platform companies for the global governance military
AI? Two lessons particularly stand out.
The
first is obvious but no less important for being so. It concerns the practical
challenges of accountability. Accountability has emerged as a
key principle across national and international governance
initiatives, underscoring the importance of delineating clear lines of
responsibility for the use of military AI systems. Central to achieving
accountability is understanding how different decision-makers contribute to the
development, implementation, and use of particular systems. This is because
accountability is foremost a relation of
answerability involving an obligation to inform about and justify
one’s conduct to an appropriate authority. Such a relation presupposes, amongst
other things, a condition of interrogation: one actor must be exposed to the
scrutiny of another because accountability only to oneself is no accountability
at all. This condition requires many things, including drawing on documentation
about the development and procurement of AI systems and the actors involved.
These
trials are not the first time evidence has come to light of Meta and Google avoiding
justifying dubious activities by concealing them, and it’s unlikely to be the
last. In 2024, The New York Times claimed that Google had spent
15 years creating a
culture of concealment. Any accountability regime worth the name that
results from military AI governance efforts ought to be based on a realistic
understanding of the motivations of commercial actors and how a sufficient
degree of scrutiny can be achieved. This is particularly important since
concealing knowledge of harmful practices in the military domain could have
potentially severe consequences. This is not the place to discuss the character
that such a regime should have, only to note that what has so far stymied
efforts in that direction is a general belief in broader governance debate that
these companies are
neutral intermediaries and the digital technologies they provide are mere
tools for channelling a state’s intent. Indeed, if there is a silver lining
between the two court cases discussed above, and the recent public spat between
Anthropic and the US Department of War, it is the beginning of the end for this
belief.
The
second lesson is to do with the way debate on the governance of military AI
tends to psychologise the sort of issues that came up in the LA trial. In
seeking to understand and explain the apparent dependency of humans on digital
systems when interfacing with them, scholars and policymakers often reach for
concepts like automation bias, cognitive offloading, and over-trust. Doubtless,
this captures something about the very human attempt to apprehend a technology
that, in the case of AI, has something more than a tool-like quality, including
by falling back on modes of thought habituated through person-to-person
interaction. But chalking up the shortcomings of human-machine interaction to
the frailties of human cognition glosses over the role of the technology’s
creator(s), including the fact that this dependency is, evidently, sometimes
intentionally engineered. What the LA verdict underscores therefore is the need
for a more nuanced account of the role of commercial product providers when it
comes to the challenges of human-machine interaction in military settings.
Frictionless by design: the business of engagement
But how much
can we extrapolate from two court cases concerned with two specific products,
provided by just two companies that themselves provide a huge variety of
different services and products?
What
is noteworthy is how much of the commentary around these trials identifies the
addictive properties of platform products as structural characteristics. If the
issue of
algorithmic bias has taught us anything, it is that digital
technologies are social artifacts. They are the products of human minds and
human hands, and it is difficult, if not impossible, to dissociate them from
the ambitions of their creators. A longstanding ambition of platform companies
is to maximise user engagement: it is intrinsic to a business model that, at
its core, is about capturing and monetising
user attention. It has a cultural corollary in Silicon Valley’s
preoccupation with the idea of friction. As Anna Wiener discusses in
her memoir
about her years working for a tech startup, ‘friction’ was a
term the tech industry used for anything that impeded a user’s adoption or use
of a product. Originally a design principle for making products easier to use,
it morphed into something of a philosophy of life:
“The endgame was the same for everyone: Growth at any cost. Scale above all. […] A world of actionable metrics, in which developers would never stop optimizing and users would never stop looking at their screens. A world freed of decision-making, the unnecessary friction of human behaviour, where everything – whittled down to the fastest, simplest, sleekest version of itself – could be optimized, prioritized, monetized, and controlled.”
‘Optimisation’
and ‘prioritization’, words long associated with internet search engines and ad
management, will be recognisable to those who take even a cursory interest in
the use of algorithmic
techniques in military targeting. Of course, much depends on how a
product is provided, including how it is configured once supplied to a military
organisation. But organizational theory tells us that, once a business alights
on a successful set of practices, these practices become highly entrenched. There
is good reason to think that not only the language of big tech, but also its approach to
product design – one that led to last month’s verdict in LA – has
carried over into the military setting.
In many ways, the two lessons described above are related. A condition of accountability is the capacity of governance frameworks to discern the traces of human decision-making in military AI systems. That requires a vocabulary that can speak to more than just the propensities of humans when interacting with these machines. The ongoing integration of AI into military targeting practices means these systems will increasingly shape how commanders see and interpret the battlefield, bringing a whole range of risks. When the conditions of their design remain opaque, and the aims of their developers go unexamined, accountability is not secured but displaced. It is imperative that governance debate moves beyond a focus on human operators to engage more directly with the socio-technical conditions of system design.
Photo attribution: “Responsible AI in the Military Domain – REAIM 2023” by Ministerie van Buitenlandse Zaken is licensed under CC BY-SA 2.0
