Anthropic researcher's resignation puts AI risks in the spotlight

Summary

Jacob Coxon, an Anthropic pre-training researcher, resigned, accusing both Anthropic and OpenAI of recklessly racing toward self-improving superintelligence. This warning is further reinforced by alignment researcher Evan Hubinger's personal estimate that there is a greater than 10% probability of human extinction within the next decade.

Why it matters

The significance of the resignation lies in the fact that these warnings come from internal actors who have spent three years building these systems, rather than external critics, lending unusual credibility to the safety concerns. Hubinger's support and his specific (albeit personal) probability estimate shift the discussion from abstract philosophy to concrete risk quantification. The article also highlights a structural tension: if labs are genuinely afraid of the outcome, the race to market creates a prisoner's dilemma dynamic that no single company can unilaterally resolve.

Highlights

• Coxon describes the current race as 'an insane gamble' and calls for inter-lab coordination and speed-limiting agreements before deploying superintelligence systems.
• Hubinger emphasizes that Recursive Self-Improvement is the key accelerator that could push capability growth beyond all current projections.
• The 10% extinction rate is explicitly a personal estimate by a safety researcher; it does not represent a peer-reviewed consensus or an official position from Anthropic or OpenAI.

Critical take

The article acknowledges that superintelligence is not yet a reality and that no existing system has realized the extreme scenarios described; therefore, the text risks framing a speculative tail risk as an immediate crisis. Additionally, while well-intentioned, a single resignation does not provide evidence that the broader research community shares this sense of urgency. Although the 10% figure is striking, it is an unverified personal judgment rather than a reproducible scientific prediction.