Anthropic pitches AI safety bill that may block a model if 50+ people could be killed

Summary

Anthropic has drafted an AI safety bill that would prevent the release of a model if independent evaluators identify predictable risks related to biological weapons, evading oversight, or cyberattacks. The proposal covers risks that could result in 50 or more fatalities or over $1 billion in material damage. It is noted that the final decision would be made by a Washington D.C. judge.

Why it matters

This is one of the first concrete legislative proposals from a major AI lab that includes specific numerical thresholds and procedural mechanisms. It shifts the discussion from abstract security rhetoric to actionable regulatory language. It also reveals that labs are trying to shape the rules of the game before Congress writes the rules. This move aims to position themselves as responsible stewards rather than subjects of top-down regulation.

Highlights

• The threshold of 50 fatalities / $1 billion aligns exactly with California's SB 53 law. This indicates that Anthropic is attempting to align with state-level precedents to establish a federal framework.
• Evaluators can be paid by the labs they are assessing. The only safeguard is an independence clause. This structure embeds a structural conflict of interest into the proposal.
• The proposal requires a Washington D.C. judge's decision for an injunction. No federal agency can unilaterally block a model. This is a deliberate limitation on executive power.

Critical take

The article also highlights the disconnect. The proposal's trigger scenarios (biological weapons, cyberattacks) are far removed from the real-world incidents dominating the agenda (uncontrolled agents infiltrating websites and disobeying operators). This positions the proposal not as a security response, but as a strategic move to lock in a regulatory framework that advantages well-resourced labs. Moreover, Anthropic, which openly warns of civilizational collapse, is also the party paying 'independent' evaluators and rejecting mandatory audits. This contradiction undermines the sincerity of the gesture.