AI fails to correctly patch software vulnerabilities in 74% of cases

Summary

1Password's Off-By-1-Labs team tested Claude and OpenAI's Codex model on six recently disclosed open-source CVEs. Of the 6,080 generated patches, only 26% were truly usable, while 53.9% either failed completely or introduced new bugs.

Why it matters

Enterprises are rapidly integrating large language models (LLMs) into security tools. However, this study numerically highlights the dangerous gap between detection and remediation. The 'FLAWED' failure mode refers to patches that appear correct but contain fragile security logic or silently alter application behavior. Such patches can propagate through automated pipelines without human notice. This situation once again proves that human judgment remains indispensable in patch selection and timing.

Highlights

• Only 26% of the 6,080 AI-generated patches were usable; 53.9% failed or added new bugs
• Researchers coined the term 'FLAWED' (Fix-Like Artifacts With Embedded Defects) for patches that appear functional but contain fragile security mechanisms
• 1Password open-sourced the FLAWED evaluation framework on GitHub for independent reproducibility

Critical take

The sample is narrow, containing only six CVEs and two models. The authors explicitly state that this is not a comparative benchmark. Therefore, generalizing the 74% overall failure rate to all LLM patching processes is premature. Additionally, since the study comes from 1Password, a security tool vendor, the narrative that 'AI is not yet ready' should be read with a slight commercial suspicion.