Claude found software bugs that escaped trillions of tests using formal verification

Summary

Anthropic's Claude model, in collaboration with verification firm Theorem, generated formal mathematical proofs for 239 entry points across 123 open-source libraries. These proofs uncovered bugs that had remained undetected despite trillions of automated test runs.

Why it matters

This approach offers a practical and cost-effective path to formal verification. While large language models (LLMs) draft the proofs, an independent checker named Rocq verifies them. This makes a technique long considered too expensive for most software accessible. Once widely shared libraries are verified and these guarantees are reused downstream, the security posture of the entire software supply chain could change significantly.

Highlights

• Approximately 180 million lines of proof code were generated; a typical single proof covers about 600,000 lines
• The bugs required extreme inputs (20–64 MB payloads, over 2 GB memory usage) that random fuzzing rarely caught
• Thanks to the independent Rocq checker, developers do not need to manually audit every line of generated proof

Critical take

The scope was intentionally narrow: the model addresses a single-threaded, statically linked RISC-V architecture under the Linux kernel. Therefore, generalization to x86/ARM, multi-threaded, and dynamically linked systems commonly used in cloud environments has not yet been proven. Additionally, the phrase 'trillions of test runs' exaggerates the surprise. The detected bugs were adversarial edge cases not systematically designed to be caught by random fuzzing. Thus, finding these bugs is not surprising but an expected outcome.