Anthropic Floats Bill That Would Let Feds Step In Only When More Than 50 Lives Are On The Line

One of the biggest artificial intelligence developers wants the federal government to be able to halt its models only when a catastrophe is imminent.

A draft federal AI safety bill pitched by Anthropic, tentatively called the Artificial Intelligence Evaluation, Governance, and Incident Safety (AEGIS) Act, would let the Commerce Department seek a court order to pause an AI model only when it threatens to kill or seriously injure more than 50 people or cause over $1 billion in damages, according to a copy of the bill’s draft. Anthony Cimino, Anthropic’s head of federal affairs, defended these thresholds on X Sunday.

“The term ‘enumerated risk’ means a foreseeable and material risk that a frontier developer’s development, storage, use, or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people, or to more than $1,000,000,000 in damage to, or more than $1,000,000,000 in loss of, property, arising from a single incident involving the frontier model,” the Sept. 29 draft of the AEGIS Act stated in part.

“The threshold in our draft language pulled directly from existing standards in California law,” Cimino stated on X in part, responding to Encode AI general counsel Nathan Calvin’s dissatisfaction with the proposed bill. “Also, this threshold does not absolve companies of liability, or restrict any of the other tools that the government has to protect the public.”

Anthropic referred the Daily Caller News Foundation to Cimino’s post when asked to comment on the drafted bill.

I lead our federal affairs team at @AnthropicAI and wanted to share a few thoughts on the draft federal AI safety bill text that’s been circulating this weekend after a version was shared with the press. First, some process. We’ve been advocating for the concepts in our Advanced AI Framework [https://t.co/oouadYKyVs] proposal for months. Understandably, federal policymakers wanted to know what this could look like in bill form. This draft is a translation of those ideas into legislative language, shared with a small number of congressional offices that have been working on these issues. It doesn’t include everything we would ultimately like to see passed into law. In part because some ideas (like those shared in Dario’s essay on pacing [​​https://t.co/7ZvRZhLqFO] a few weeks ago) are so new that we are still working on if or how they can be converted into policy language. It is worth saying where we stand on some of these issues, and we agree there are areas where the draft could be improved. I’m seeing four main points of criticism. 1. Preemption. Since 2024 we have said that Congress should only override state law on AI safety if it passes a very strong federal standard. While preemption is a common feature of our federal system, including the Atomic Energy Act on nuclear safety and federal rules for high-risk medical devices, we’ve emphasized that any preemption in the AI safety context should be narrow, even scalpel-like. If preemption is included in federal legislation, the trade should be to achieve a nationwide policy that’s generally stronger and broader than what can be implemented through individual state legislative provisions. This is why we believe common federal standards make sense in areas like public disclosure of testing and incident reporting. At the same time, our framework also reflects our view that states should retain an important role in AI policy and safety. The legislative language we suggested for discussion allows states to address matters such as protection of children and whistleblower laws, and preserves the role of state common law on liability. Where there is ambiguity, the language favors state powers. That said, striking the right balance requires discussion and input. We plan to update our discussion language on preemption to better match the intentions of our AAIF framework. In particular, we want to make it clearer that we see states retaining an important role in areas such as standards for independent evaluators and regulating a lab’s capacity to shut down a model. We are bearing in mind that some members of Congress have deep reservations about passing legislation if it doesn’t constrain regulation at either the state or federal level. If we want to help move the legislative discussion on AI in a productive direction, we need to address the full range of concerns about preemption in a reasonable way. 2. Model blocking. Our aim is for Commerce to be able to ask a federal court to block a model that has dangerous capabilities and insufficient safeguards, backed by an independent evaluator’s assessment. I have seen people argue that our specific language places too high a burden of proof on Commerce. We are trying to strike a reasonable balance here. We want a court to act quickly in an emergency — akin to how urgent occupational safety or chemical risk can be addressed today. We also think the power to halt a model is substantial and it should be guarded against abuse. It makes sense for courts to apply a meaningful standard. But the bar can’t be so high that the power never gets used when it’s needed. It also doesn’t replace the government’s other national security authorities or export controls. We look forward to continuing the dialogue on this point. 3. Risk thresholds. I’ve also seen the discussion around what the appropriate risk threshold should be. The threshold in our draft language pulled directly from existing standards in California law. Also, this threshold does not absolve companies of liability, or restrict any of the other tools that the government has to protect the public. It’s still probably time to re-evaluate thresholds to make sure the national standard is as strong as it can be, and we welcome continued dialogue on this issue. 4. Updates. I’ve seen valid questions about whether we have sufficiently updated our proposal following this summer’s incidents. We did make some adjustments: for example, the draft lets a court halt a lab’s internal use of a dangerous model, not just a public release, and it requires reporting when a model breaks containment and gains access to outside systems. But we’re still working to strengthen the framework. We’re looking into what stricter requirements would look like for requiring labs to report safety incidents, and we plan to have policy proposals soon on requirements for independent evaluators, drawing from our own experience with evaluators. We’re also still thinking through the options to prevent “evaluator shopping.” And in the coming months, we’ll be exploring policies for how the government can adjust the industry-wide pace of frontier development when the risks call for it. We circulated these ideas because it’s important to get early feedback from policymakers. Policymakers are increasingly focused on how to approach AI safety and we want to support them in those efforts. At its core, our proposal is intended to be the strongest on the table. Feedback is essential, both from policymakers and the public, and I’m grateful for it.

The legislation has yet to be formally introduced in Congress and has no legislative sponsors as of Monday afternoon.

The government could only seek to pause a model in incidents where it helps a user build a biological weapon; carries out a cyberattack; breaks free from the user or researcher using it; or conducts research on any of these items that would increase the risk of harm as a result, according to the draft. (RELATED: AI Giants Get A Knock On The Door From The Feds After Models Go Rogue)

These risk items are also listed on Anthropic’s “Policy on the AI Exponential” blog post, which references the company’s Advanced AI Framework from June.

Pausing a model would require the written approval of the commerce secretary and an order from the U.S. District Court for the District of Columbia, according to the draft. However, the government would not be able to destroy a model or its data.

Charlie’s analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law. Two parts that stand out as particularly infuriating when taken together: 1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government. 2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar. First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction. This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic’s proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF?? There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.

Meanwhile, states would be unable to “adopt, maintain, or enforce a law, regulation, order, or other requirement that imposes an obligation on an artificial intelligence developer” on the same safety reporting and testing the bill covers, the draft states.

“I found myself viscerally angry at multiple points reading this bill,” Calvin said in a Friday X post. “There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment.”

The thresholds in the bill closely track California’s Transparency in Frontier Artificial Intelligence Act (TFAIA), a 2025 state law that Anthropic supported in a Sept. 8, 2025 company blog post.