AI’s biggest bosses told the UN Security Council AI could threaten humanity – here’s what they asked for

For the first time, the people building the world’s most powerful AI have briefed the UN Security Council, and the rivals agreed on one thing: they want rules. “If managed poorly, I even believe AI could be a risk to humanity as a whole,” Anthropic CEO Dario Amodei told the Council on Wednesday, Al Jazeera reports. The U.S. government, sitting at the same table, rejected the idea outright.

This article was first published before the session and has been updated with what was said.

What they said

Sam Altman, speaking in person, argued that no level of catastrophic risk is acceptable. “It doesn’t matter whether people put the risk of catastrophe at 10%, 1%, 12%, or 0.1%. None of these levels are remotely acceptable,” he said, according to a transcript of the session. He also warned that “the most important decisions cannot be made by labs in San Francisco alone,” and made a promise his company will be held to: “We have unilaterally slowed down in the past. We will do so in the future.”

Dario Amodei, appearing by video, made a similar pledge: “We will slow down as much as necessary in order to make sure that every successive AI technology that we release is actually safe.” He added that “managing these risks is ultimately bigger than any one company.”

Clément Delangue of Hugging Face, whose company was hit by an AI agent during testing earlier this year, made the case for open-source AI as a defence: “We were attacked by AI, but more importantly, we defended ourselves with AI.” In his view, “the biggest risk is not powerful AI, it’s asymmetry of powerful AI.”

Yoshua Bengio called the dangers “real and imminent,” warning that the threat “does not respect the borders we defend.”

The reports we’ve seen don’t include any statement from DeepSeek or Moonshot, the two Chinese labs invited to speak.

What they asked for

• Amodei: outside evaluators embedded inside AI companies with employee-level access; narrow global deals such as a ban on using AI to build biological weapons; ways for countries to verify each other’s commitments; common global standards for testing loss-of-control and misuse risks; and a notification system for serious AI security incidents.
• Altman: national and international standards for frontier AI that work together, shared ways to measure capabilities and risks, fast incident reporting, and secure channels for governments and infrastructure operators to share vulnerabilities.
• Delangue: tougher monitoring and incident-disclosure rules, including sharing full traces of what AI agents did, and equal access to defensive AI tools around the world.

France’s foreign minister, Jean-Noël Barrot, who chaired the session, backed independent testing of models before and after release, and legal liability for AI companies. “Cohorts of agents are operating without any human oversight,” he said.

Washington said no

The U.S. didn’t budge. “We totally reject all efforts by international bodies to assert centralised control and global governance of AI,” said Michael Kratsios, who spoke for the U.S. at the session, according to Al Jazeera. The UK and France backed building international frameworks. It was an open briefing, so nothing binding came out of it.

Before the session

Who’s speaking

• Sam Altman, OpenAI CEO, in person
• Dario Amodei, Anthropic CEO, expected to speak remotely
• Clément Delangue, CEO of Hugging Face
• Yoshua Bengio, the Turing Award-winning researcher who co-chairs the UN’s Independent International Scientific Panel on AI
• DeepSeek and Moonshot, maker of the Kimi models, have been asked to make statements. DeepSeek founder Liang Wenfeng isn’t expected to attend, and the company hasn’t said who will represent it, according to The Next Web, citing Reuters.

It’s the first time frontier AI developers from the U.S. and China will address the same UN forum on shared safety concerns.

What it’s about

France, which holds the Council’s rotating presidency this month, organised the session, and its foreign minister, Jean-Noël Barrot, will chair it. France’s concept note frames the discussion around two fears: the malicious use of AI, and the possibility that increasingly capable systems could “slip beyond human control,” enabling cyberattacks, election interference or autonomous weapons, Decrypt reports.

Those fears aren’t hypothetical any more. In recent weeks, OpenAI and Google have both disclosed incidents in which AI agents under testing broke into real companies’ systems.

The politics

Amodei’s recent essay called China “the autocratic country with by far the most advanced AI capabilities” and backs keeping U.S. chip export controls in place, a framing Beijing’s foreign ministry rejected. China’s state-run Global Times dismissed the whole “pacing” idea as a “Cold War playbook” designed to lock in America’s lead.

Washington isn’t on board either. On Tuesday at the UN General Assembly, President Trump rejected any “globalist scheme to control” AI and mocked warnings that it could “kill us all.”

The session came as Trump hosts China’s Xi Jinping in Washington this week, with a proposed U.S.-China AI incident hotline on the agenda.

Sources: Al Jazeera, The Singju Post (transcript), Security Council Report, Decrypt, ABC News, The Next Web, Startup Fortune