Could AI really wipe out humanity – six experts spell out the risks

Experts say claims AI could kill all humans through biological weapons development had no scientific basis. Illustration: wildpixel/Getty ImagesView image in fullscreenExperts say claims AI could kill all humans through biological weapons development had no scientific basis. Illustration: wildpixel/Getty ImagesAI (artificial intelligence)ExplainerCould AI really wipe out humanity – six experts spell out the risksWe examine claims and counterclaims about the risks and calls to slow down the pace of AI development

There have been some shocking claims in recent days about AI safety: we face a 10% chance of doom; AIs are worse than nukes; a “botnet” threatens the entire internet; it’s all a big tech psyop.

Amodei’s concerns should not be entirely dismissed but require a large shaker of salt, said Gary Marcus, an AI sceptic and emeritus professor at New York University.

It was “nonsensical”, he said, to suggest the entire internet is vulnerable and that it was “pretty unclear how that happens without extreme negligence on the part of the labs themselves”. Marcus highlighted a recent assessment by the UK AI Security Institute that Mythos, one of Anthropic’s current leading-edge AIs, could “autonomously compromise [only] small, weakly defended vulnerable systems”.

With large parts of the internet driven by computer infrastructure owned by Cloudflare, Google and Amazon Web Services (AWS), Marcus – assessing the claim with two colleagues – said he doubted that any of those companies could be taken down or controlled for a prolonged period of time without government force. Less defended individual websites might be attacked but “the internet itself is very unlikely to crumble”, they said.

Meanwhile Niels Rogge, an engineer at Hugging Face which was hacked by OpenAI agents, called the internet takeover claim “bizarre nonsense”. The hack into Hugging Face resulted from improper human control and “an insane amount of compute only they can afford”, he said.

Add to Amodei’s assessment of an internet takeover within a year the view expressed on Monday by his Anthropic co-founder Jack Clark, that the moment when AIs “start doing dangerous stuff” was in about 20 years, and the picture becomes more doubtful still.

Prof Alan Woodward at the University of Surrey’s centre for cyber security said botnets tend to stall as the internet is not homogeneous and it is difficult to design them to persist in all its parts. Internet traffic is also resilient and tends to reroute around damage.

“The AIs would not decide to do it on their own,” Woodward said. “It’s the humans that are responsible and the AI is a tool. That is what we need to control, not the AI.”

This statement is an expression of what is known in the AI world as p(doom) – or probability of doom. It’s the likelihood of AI wiping out humanity by evading our control and triggering catastrophe, perhaps through developing potent bioweapons or crashing the global financial system.

One survey of researchers’ forecasts, published this week, asks them to place a number on the chance of “human extinction or similarly permanent and severe disempowerment from advanced AI”. But Heidy Khlaaf, chief scientist at the AI Now Institute research body, is among experts sceptical about giving percentage chances to existential scenarios.

“These claims are neither falsifiable nor verifiable, making then unscientific by their very nature,” she said. The shortage of concrete examples of what a “superintelligent” AI might do to cause “doom” only adds to the cloudiness. A source familiar with Anthropic’s thinking acknowledges that “the exact chances of any one outcome are probably unknowable”.