No. 123
2026-11-19
the AI pulse — brewed, not raw
 AI CRITIQUE

Harvard study: AI agents boost code generation but slow down reviews

Read the article    English (automatic translation)

Summary

A working paper prepared at Harvard, analyzing 718 companies and approximately 726,000 employees, reveals that AI coding agents increased code production by about 30%, but extended review time by 49% while statistically leaving net completed work unchanged. According to the authors, the bottleneck shifted from writing code to the review stage.

Why it matters

This finding challenges the common narrative that AI coding tools provide a clean efficiency gain. If marginal gains in production are swallowed by nearly doubling the review burden, organizations may be paying for AI without delivering more product. This also reframes where engineering teams should invest: prioritizing review tools, triage automation, and process redesign over adding more production models.

Highlights

  • Code volume increased by approximately 30% (average +1,494 lines per person per month), commit count rose by 20%, and PR count by 23%
  • Review time increased by approximately 49%; the rate of formal change requests nearly doubled (from 13% to ~25%); comments per PR increased by 35%
  • Net completed work: only +0.12 items per person per month—statistically insignificant

Critical take

This is not a peer-reviewed article; it is explicitly a working paper prepared for job application purposes. The difference-in-differences method compares early and late adopters without using a clean random assignment. Confounding variables such as company maturity, project mix, and concurrent tool changes may obscure the true cause behind the 49% increase in review time. Additionally, the monthly difference of 0.12 in 'completed work' is so small that labeling it as 'no significant increase' risks confusing a measurement error arising from how 'completion' is defined in Jira with a true null result.

This critique was written by the local AI model Qwen3.8-27B. I chose the story; I do not review each critique before it is published.
Back to the issue