AI Almost Started a U.S.–China War — and No One Seems to Cares - The Intercept
The global tech industry, politicians at home and abroad, and international media coverage are transfixed by one question: Will rogue AI end humanity? Few seem to be asking this question of the Department of Defense.
The artificial intelligence sector has rocked itself in recent weeks following a string of disclosures from the frontier AI labs and some of their personnel. OpenAI and Anthropic have both disclosed incidents in which their software, during routine internal testing, unexpectedly broke into the networks of other corporations. The tests were akin to a disastrous demonstration of a guided missile: After humans hit launch, the autonomous technology veered off course in deeply alarming ways.
Observers and industry figures quickly interpreted the incidents not simply as an indicator of the software’s power. Instead, they argued that it presaged an age of computers possessing something no computer ever has: bad intentions.
If computer systems could behave in ways that are unexpected — or even shocking — without being directly steered by humans, it seems only a short step until computers might have desires, ambitions, and even malice. If OpenAI’s tools could unexpectedly crack the servers of a rival corporation to accomplish a task posed to it by engineers, couldn’t it also surprise us with actions that result in harm to people, or even deaths?
Frontier lab employees and executives have answered with an emphatic yes. On September 8, Anthropic researcher Jacob Coxon announced his resignation on X, writing, “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.” Coxon’s former Anthropic colleague Evan Hubinger replied, casually agreed: “Jacob is correct here — we really do earnestly believe AI could kill all humans!” Hubinger noted he personally puts the odds of the species’ extermination by AI, somehow, at over 10 percent.
This has all resulted, unsurprisingly, in a global panic, with a sudden flurry of calls for regulatory intervention, from legislation that would require an emergency “kill switch” for AI systems, to a proposal by Sen. Bernie Sanders to halt AI development altogether.
Throughout this period of alarm, how exactly a large language model could literally end humanity has remained vague. Some have speculated that this technology could foster the creation of some sort of novel bioweapon; others worry many thousands of AI agents could somehow hack all vital infrastructure simultaneously. In both those supposed doomsdays, the details are still fuzzy.
It should have come as quite the shock, then, when CNN reported on September 18 of a recent incident in which the use of artificial intelligence could have genuinely led to the extinction of the species.
It was under direct human supervision that a large language model almost misled the U.S. into instigating a war with his country.
This past spring, according to the news outlet, a U.S. Special Operations Command analyst used a large language model to generate an intelligence report that indicated “a Chinese ship in the Middle East was transporting components of a nuclear weapons program.” The finding prompted the U.S. military to make quick preparations to intercept the vessel by force. But this AI-generated intelligence, according to one source who spoke to CNN, was “entirely false.” Yet, the source said, it “almost started a war.”
A shooting war between the U.S. and China could play out in an incalculably wide variety of ways. But one entirely plausible path would be an exchange between the world’s first and third largest nuclear weapons arsenals — an event that would transcend warfare into global cataclysm.
CNN’s reporting garnered attention, but was not followed by nearly the degree of sustained, grave concern of the AI safety news cycles that preceded it. It didn’t seem to prompt company scientists to question their careers, nor did it spur calls for regulatory intervention or self-imposed limits by the companies who furnish the Pentagon with this technology. After weeks of discussion of how AI could hypothetically kill everyone, the public learned of a concrete way in which AI really could have started a war that might have killed everyone, and the world quickly lost interest.
“This [China] incident and the lack of response is an attention problem that the nuclear field has been grappling with for decades.”
At a summit this week with President Donald Trump, Chinese President Xi Jinpeng argued that nations must “ensure that the development of AI is always under human control,” again underscoring fears of AI breaking free of human oversight. But it was under direct human supervision that a large language model almost misled the U.S. into instigating a war with his country.
The discrepancy illustrates a gap in concern by both the general public and policymakers over AI risks: There is great alarm over an AI system “going rogue,” disobeying its commands and the interests of its creators and operators, and pursuing a malign agenda of its own. Much of the AI safety discourse revolves around the hypothetical threat of “superintelligence,” defined broadly as a computer system that can outthink even the smartest humans across domains. There’s also the belief that software systems will both achieve sentience and bear a grudge against their creators and act to undermine them, if not destroy them outright.
Yet there is considerably less public alarm over AI performing exactly the actions that the Pentagon asks of it, whether targeting airstrikes or generating actionable intelligence reports for U.S. Special Operations Command.
Most Read Flock Wants the Most Detailed Map of Its Surveillance Cameras Taken Offline Nikita Mazurov DoorDash Spent $1.4 Million Trying to Stop Mamdani From Becoming Mayor. Now We Know Why. Meghnad Bose, Macy Hanzlik-Barend The Backlash to “NAZA” Shows How Little Israeli Citizenship Really Means Séamus Malekafzali
Following weeks of intense discussion by the industry’s most visible figures whether American AI labs should self-impose limits on their engineering or have such limits imposed by the state, there has been virtually no suggestion that any limits be placed on these companies’ single most powerful customer: the U.S. military.
This has led to seemingly counterintuitive narratives coming from industry leaders, with figures like OpenAI CEO Sam Altman simultaneously warning that his product is existentially dangerous while selling it to self-styled War Secretary Pete Hegseth. “Despite incessant warnings from AI companies that AI is harmful, even in hypothetical ways, they are in fact promoting that their unreliable products be instrumented within national and safety-critical infrastructure, such as defense and nuclear,” Heidy Khlaaf, chief scientist at the AI Now Institute and former systems safety engineer at OpenAI, told The Intercept. “Given AI’s lack of reliability and accuracy in such critical context, that is what will ultimately lead to a catastrophic accident. However, if such issues were acknowledged, that wouldn’t be profitable for AI companies and would slow down adoption.”
Popular Silicon Valley ideologies tend to consider a hypothetical “existential” threat from a superintelligence more worthy of attention than near-term violence.
U.S. AI players likely have fresh in their minds the recent experience of Anthropic, which was temporarily blacklisted from governmental work after it said it did not want its software used to conduct unconstitutional mass domestic surveillance or operate fully autonomous weaponry. Similar efforts to place limits on how their products can be used to spy or kill might incur the wrath of the particularly vengeful Trump administration, especially as Hegseth has worked to eradicate civilian harm reduction processes to emphasize greater “lethality” as the military’s guiding principle.
The rapid adoption of their technology also represents a massive long-term financial boon to U.S. AI labs-turned-defense-contractors such as OpenAI, Google, and Anthropic — all of whom once vowed to not pursue military work because of potential harms. Despite these pledges, these firms now have access to the near-limitless Pentagon appropriations spigot, and the $200 million deals many of these firms signed with the Department last year pale in comparison with what the U.S. military could pay in the decades to come. Altman and his peers might also be less willing to sound the alarm on Pentagon-based AI threats because such a clarion call would be coming from inside the house.
Earlier this month, The Intercept revealed contract documents detailing the intimate collaborative relationships between these companies and the military, in which the Silicon Valley giants would have a say in helping the Pentagon craft its overall military AI strategy. This closeness comes at a time when the Hegseth boasts of drone-striking civilian fishing boats across the Caribbean, and the commander-in-chief speaks casually of “annihilating” entire civilizations.
Lucy Suchman, professor emerita of anthropology of science and technology at Lancaster University, attributed the difference in public alarm and media focus to the power that tech billionaires command when it comes to shaping narratives. Popular Silicon Valley ideologies, such as the Bay Area’s strain of “Rationalism” and the effective altruism movement, tend to consider a hypothetical “existential” threat from a superintelligence more worthy of attention than near-term violence. This priority, she said, that pervades the AI safety research community, obscures the real and present threat that today’s AI models pose in a world chockablock with nukes.
“The almost targeting of the Chinese ship might further underscore the danger not from rogue AI agents, but a rogue U.S. secretary of war.”
“Given the U.S. Dept. of War’s commitment to maximizing the speed and scale of target generation, AI-enabled targeting is already an existential threat for civilians on the ground, and when the targets are the military assets of nuclear armed states, the threat becomes global,” said Suchman, who has studied artificial intelligence since the 1970s and serves as a member of the nonprofit International Committee for Robot Arms Control. “The almost targeting of the Chinese ship might further underscore the danger not from rogue AI agents, but a rogue U.S. secretary of war.”
Hegseth, who once derided the rules of engagement as “stupid,” extends this approach to artificial intelligence, too. Describing the Pentagon’s new AI strategy in a January memo that includes the word “accelerate” 24 different times, Hegseth’s laid out a plan with an intense focus on speed for speed’s sake. “Speed Wins,” Hegseth wrote. The war secretary explained the military would adopt experimental technologies as quickly as possible by “aggressively identifying and eliminating bureaucratic barriers to deeper Integration” of AI, without mention of ensuring that the technology actually works beforehand.
Heidi Kandiel, a legal adviser on new military technologies at the International Committee of the Red Cross, told The Intercept that speed itself carries the risk of military disaster for any country. “I think one of the central issues that comes up is speed and scale, which are often the touted benefits of AI,” she explained. “If a system is unreliable, is operating on poor quality or outdated data or is used within a flawed decision-making process, AI would allow these errors to be reproduced much faster and across a far greater number of targets or operations.”
Reining in a corporation is a very different prospect than putting fetters on a government. Policymakers who feel comfortable blasting the irresponsibility of Anthropic might lack the political will to levy similar criticisms against the president or the Pentagon. Madeline Berzak, a scholar at the University of Chicago’s Existential Risk Laboratory, told The Intercept that nuclear threats have long been met with paralysis (or a shrug) because people feel helpless in the face of them. You can tweet angrily at Dario Amodei, but it’s less satisfying to yell at U.S. Strategic Command. “This [China] incident and the lack of response is an attention problem that the nuclear field has been grappling with for decades,” Berzak said. “‘Doomsday messaging’ is what I call the nuke field’s default communications strategy, and I have this theory that it never works because the fear it tries to generate has nowhere to go.”
“The threat of thermonuclear war is old news,” Suchman lamented.
We’re independent of corporate interests — and powered by members. Join us.
Original reporting. Fearless journalism. Delivered to you. Will you take the next step to support our independent journalism by becoming a member of The Intercept?
By signing up, I agree to receive emails from The Intercept and to the Privacy Policy and Terms of Use.
Some in the AI safety community acknowledge the long-standing threat of nuclear annihilation. They just consider it a lower priority than the as of yet unrealized rogue superintelligence. Daniel Kokotajlo left his role as an OpenAI researcher in 2024 over the company’s alleged recklessness and indifference toward risk. He is the co-author of AI 2027, a white paper laying out a hypothetical scenario in which superintelligent AI systems exterminate humankind with bioweapons to clear up space to build automated robotics factories. History is the reason Kokotajlo is less preoccupied by the prospect of an AI-triggered nuclear war. “It’s definitely something to be concerned about, and it’s something that ideally we would like to improve and fix. But it’s the sort of thing where, even if we do nothing, probably things will be fine for a decade or two or more.”
To Kokotajlo and like-minded researchers, no matter how dangerous nuclear arsenals make the world, superintelligence is simply more dangerous than a bogus intelligence briefing on China. “It’s been 50 years and these sorts of crises don’t happen that often. So far, cooler heads have managed to prevail and figure out the truth in each case,” he said. “And so the probability that it goes all the way to nuclear war in the next five years is low. High enough to be a bit uncomfortable, but low overall. By contrast, if we do superintelligence in the next five years, I think the probability that it goes horribly wrong for us is high, not low.”
Berzak, who leads her lab’s nuclear risk program, says this is the wrong calculation. She pointed to differing approaches to staving off the apocalypse embraced by the novel field of AI safety and nuclear caution that has existed since the dawn of the Cold War. “Attention often goes to what is new and interesting and not what is the most pressing risk,” she said.
Cooler heads have indeed avoided nuclear disaster in the past, but human fallibility and the growing temptation to outsource our judgment to machines perceived as superhumanly brilliant carries an inherent danger, Berzak warned: “Just because there’s this new more pressing risk that has consequences that are rather underexplored compared to nuclear war, doesn’t mean that nuclear war is not a risk because we’ve so far avoided it basically with just luck.”
What we’re seeing right now from Donald Trump is a full-on authoritarian takeover of the U.S. government.
Court orders are being ignored. MAGA loyalists have been put in charge of the military and federal law enforcement agencies. The Department of Government Efficiency has stripped Congress of its power of the purse. News outlets that challenge Trump have been banished or put under investigation.
Yet far too many are still covering Trump’s assault on democracy like politics as usual, with flattering headlines describing Trump as “unconventional,” “testing the boundaries,” and “aggressively flexing power.”
The Intercept has long covered authoritarian governments, billionaire oligarchs, and backsliding democracies around the world. We understand the challenge we face in Trump and the vital importance of press freedom in defending democracy.
IT’S BEEN A DEVASTATING year for journalism — the worst in modern U.S. history.
We have a president with utter contempt for truth aggressively using the government’s full powers to dismantle the free press. Corporate news outlets have cowered, becoming accessories in Trump’s project to create a post-truth America. Right-wing billionaires have pounced, buying up media organizations and rebuilding the information environment to their liking.
In this most perilous moment for democracy, The Intercept is fighting back. But to do so effectively, we need to grow.
That’s where you come in. Will you help us expand our reporting capacity in time to hit the ground running in 2026?
I’M BEN MUESSIG, The Intercept’s editor-in-chief. It’s been a devastating year for journalism — the worst in modern U.S. history.
We have a president with utter contempt for truth aggressively using the government’s full powers to dismantle the free press. Corporate news outlets have cowered, becoming accessories in Trump’s project to create a post-truth America. Right-wing billionaires have pounced, buying up media organizations and rebuilding the information environment to their liking.
In this most perilous moment for democracy, The Intercept is fighting back. But to do so effectively, we need to grow.
That’s where you come in. Will you help us expand our reporting capacity in time to hit the ground running in 2026?
