Gavin Newsom speaking at the California Democratic Party State Convention in San Francisco in June 2019 (file photo). Image: Gage Skidmore / Wikimedia Commons, CC BY-SA 3.0, cropped

California should be able to order an immediate shutdown of a dangerous AI model, and to cap AI that builds its own successors, under recommendations released by Governor Gavin Newsom on Wednesday. The state, not the companies, should set the safety standards every big AI lab must meet, says the report from California’s Government Operations Agency.

What the report proposes

The report answers Newsom’s September 18 executive order, which asked for ideas in four areas: standards, independent verification, incident reporting and emergency shutdowns. Dated November 16, it proposes:

  • State-set standards: baseline safety, security and transparency rules that all large frontier developers must follow from training to deployment. Licensed independent verification organisations (IVOs), created under SB 813, would turn them into technical tests.
  • Ongoing outside checks: IVOs would get full access to the people and information they need and share unredacted assessments with the state. If an IVO can’t vouch for a model, the state could order corrective action, which the developer could contest in court first.
  • Faster incident reports: cut the deadline for most critical safety incidents from 15 days to 48 hours, keep 24 hours for imminent catastrophic risks and cover attempted weight theft and any loss of control even when nobody is hurt.
  • A cap on AI building AI: labs would report how much of their compute goes on automating AI research, and the state could cap recursive self-improvement (RSI) if it judged it a catastrophic risk.

Not a single off switch

Each large developer would need a Risk Containment Protocol, tested by an IVO and certified by a senior company official, showing it can step in at several levels: compute, agents, internal access and deployment. Labs would have to test whether a model would try to block its own shutdown, for example by copying its weights.

If a danger were imminent and a developer refused to act, officials could order an emergency shutdown that takes effect at once and stays in force while a court reviews it. Open-weight developers would be covered only for deployments they control, since they can’t switch off copies others host.

Rob Reich, a Stanford professor and one of four outside experts who advised on the report, put the gap this way:

Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design.

Rob Reich, Stanford University

“Allowing the fox to guard the henhouse”

The report says no federal law makes AI companies follow safety standards, submit to independent testing or shut down a dangerous model, and calls voluntary commitments “not enough”. Newsom said Donald Trump “shrugs his shoulders and leaves AI regulation up to the companies themselves”, which he called “allowing the fox to guard the henhouse”. Newsom added:

California isn’t waiting until it is too late. We are updating and strengthening our nation-leading framework to move as quickly as technology evolves.

Gavin Newsom, Governor of California

He wants Congress and Trump to adopt the framework or treat it as “a floor, not a ceiling”. It builds on a run of California AI laws, from SB 53 to the No Robo Bosses Act.

What it means for OpenAI, Anthropic and Google

The rules would bite on “large frontier developers”, the SB 53 term for labs that train the biggest models and earn more than $500m a year, a group that takes in OpenAI, Anthropic, Google and Meta. The report wants to add labs that spend above a set amount on frontier research, and to bring in any model able to cause catastrophic harm, whatever its training compute. It says the current 10^26 operations threshold kept OpenAI’s Hugging Face incident out of scope; the state’s attorney general sued OpenAI on Monday over that and other rogue-agent incidents.

The labs have partly asked for this. OpenAI urged Newsom last month to require shutdown controls and audits, and OpenAI and Anthropic signed the Paris Call’s pledge to build fail-safe mechanisms. None had publicly responded to the report by Wednesday evening.

What happens next

These are recommendations, not law. The report hopes they will shape debate “as the legislature returns next month”, and lawmakers would have to write and pass the bills. Newsom leaves office in January after two terms.

Why it matters

If passed, California would be the first US state with the power to order a frontier AI model switched off and to cap AI that improves itself. With two-thirds of the world’s top 50 AI companies based there, by the report’s count, its rules would set the bar for models used worldwide, the opposite of Washington’s hands-off line.

Sources: Office of the Governor of California; California Government Operations Agency, AI Safety in 2026 and Beyond (PDF, Nov 16, 2026).

Latest Policy & Safety news

More Policy & Safety news