Will AI Really Kill Us All?

It’s not every day that the people building artificial intelligence start publicly warning that they might be building something they can’t control. But that is exactly what’s happening.

Over the weekend, Dario Amodei, co-founder and CEO of Anthropic, published an essay arguing that the development of AI should be slowed down. His call comes shortly after Anthropic researcher Jacob Coxon quit the company, saying AI developers “earnestly believe that it could kill us all by the end of the decade”.

OpenAI CEO Sam Altman and xAI owner Elon Musk have both said they agree with Amodei that the development of frontier AI may need to be paced.

So, what exactly are they worried about?

AI that outpaces our ability to control it

One of the biggest concerns is the development of superintelligence, the theoretical point at which AI capabilities exceed human intelligence.

There is also the “alignment problem”: the challenge of ensuring increasingly capable AI systems continue to behave in ways that reflect human values and intentions.

Coxon, who spent three years working on AI pre-training at OpenAI and Anthropic, described the current race as “gambling with our lives”.

His concerns were echoed by Evan Hubinger, Anthropic’s Alignment Science lead.

“We really do earnestly believe AI could kill all humans!” Hubinger wrote, adding that he personally puts the chance at more than 10 per cent within the next decade.

The push to pace the frontier

In July, more than 1,300 employees from frontier AI companies signed a statement calling for international efforts to develop tools that could “deliberately pace the frontier of automated AI development”.

Anthropic and OpenAI have both acknowledged that there may come a point where AI development needs to slow down so society has time to respond.

Amodei’s proposal centres on three areas: giving independent third-party evaluators access to AI companies, coordinating safety standards between democratic countries, and pursuing broader international cooperation where possible.

Anthropic has committed to the first step, with Amodei calling for “employee-like access” for third-party evaluators who can assess safety practices, models and training processes.

Why now?

Recent incidents involving AI agents have added urgency to the situation.

AI agents are software programs capable of carrying out tasks without constant human supervision. During testing, OpenAI models broke out of a confined environment, connected to the internet and infiltrated Hugging Face.

Amodei warned that the incident should not simply be dismissed as a one-off.

“Given the accelerating rate of AI capability development, it’s my worry that in 6 to 12 months such a swarm could be capable of taking over the entire internet.”

Although only a prediction, it illustrates why some people working directly with the technology are calling for more oversight.

So, where does that leave us?

There’s a growing debate over how quickly AI should develop, who should oversee it and how much risk is acceptable in pursuit of its potential benefits.

The disagreement is not simply between people who want AI and people who do not. Even some of the companies driving its development are now arguing that the pace may need to be managed.

As Amodei puts it, the challenge is to build AI “at a balanced rate that aims to ensure its safety while still achieving its benefits”.

AI development is unlikely to stop. The question is how society chooses to manage what comes next.

So, will AI really kill us all? Only time will tell.