Anthropic’s logo. Image: Anthropic
Anthropic, the AI company that has built its brand on safety, has spent the weekend defending a federal AI safety bill it never meant the public to see. A 37-page draft it has been quietly showing to lawmakers, published by Politico on Friday, would override parts of state AI laws and set a high bar before the government could stop a dangerous model. Safety campaigners have lined up to attack it, and the company now says it will rewrite parts of it.
The draft, marked "Discussion Draft / Privileged & Confidential" and last edited on September 29, carries working titles including the "Artificial Intelligence Evaluation, Governance, and Incident Safety (AEGIS) Act". It turns Anthropic's Advanced AI Framework, ideas the company has promoted for months, into legislative language. It has not been introduced in Congress.
What the draft would do
The bill covers developers of the most capable models. The biggest, those with more than $500 million a year in AI revenue or over $1 billion in development spending, would have to publish a safety framework, hire independent evaluators to test their models, report serious safety incidents to the Department of Commerce and file a signed statement each year certifying they have stuck to their own framework.
The safety rules centre on "catastrophic risk": a single incident that kills or seriously injures more than 50 people, or causes more than $1 billion in damage, through weapons help, an unsupervised cyberattack or crime, or a model evading its developer's control.
Only one route exists to halt such a model. Commerce would have to ask a federal court in Washington, DC for a "blocking order", backed by a written determination signed personally by the head of the department and an independent evaluator's assessment. The court could act only if it found the risk "present or impending" and that no less restrictive step would work, and it would be told to weigh the harm an order would cause to the developer, users and the public. No other federal agency could suspend a model under the act. Fines are left blank in the draft, at "the greater of [$1,000,000] or [__] percent" of global revenue.
On states, the draft would stop them imposing their own rules on AI developers for risk disclosures, third-party evaluations and incident reporting. Child safety laws, whistleblower protections and state courts' ordinary liability law are carved out, and the text says any doubt should be "resolved in favor of preserving the authority of the States". But it also lists state orders suspending a model as a covered area, which critics read as taking that power away from states altogether.
"Viscerally angry"
The reaction from AI safety advocates, usually some of Anthropic's closest allies, was fierce. Nathan Calvin of Encode, a group that campaigns for AI safety laws, wrote on X:
I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Nathan Calvin, Encode
Alex Bores, the New York lawmaker who wrote the state's RAISE Act and is running for Congress, said it "comes nowhere close to the speed of developments nor the speed we need to keep people safe", pointing to this summer's run of rogue AI incidents. Writing in Transformer, Shakeel Hashim compared it unfavourably with the bipartisan FRONTIER Act from Representatives Jay Obernolte and Lori Trahan, which he says sets standards and licensing for evaluators where Anthropic's draft does not.
Anthropic says it will change the preemption language
Anthony Cimino, who leads Anthropic's federal affairs team, responded on X on Sunday. He said the draft had been shared with "a small number of congressional offices" and that the company plans "to update our discussion language on preemption", making clearer that states keep a role in areas such as standards for evaluators and a lab's ability to shut down a model. He said the 50-death and $1 billion threshold was taken from California law but that it is "probably time to re-evaluate thresholds", and that proposals on evaluator standards and stricter incident reporting are coming:
But the bar can’t be so high that the power never gets used when it’s needed.
Anthony Cimino, head of federal affairs, Anthropic
"At its core, our proposal is intended to be the strongest on the table," he wrote.
Why it matters
Congress is still fighting over whether federal AI rules should override the states, with Ted Cruz blocking binding safety rules in the Senate and Big Tech already signed up to a voluntary White House pact. A bill from the lab most associated with safety carries weight with lawmakers. Anthropic has also just let six outside testers inside the company, the kind of independent evaluation its draft would write into law.
Sources: Politico (draft bill text), Anthony Cimino on X, Anthropic Advanced AI Framework, Nathan Calvin on X, Alex Bores on X, Transformer.
Latest Anthropic news
