✓ Updated May 2026 — includes METR Feb 2026 update & Faros AI 22K dev study

Here’s what nobody wants to say out loud at the all-hands: your AI adoption numbers look great, and your engineering output doesn’t. That gap isn’t a communication problem. It’s a measurement problem — and a bit of a data problem too.

I went through every credible study I could find on AI’s actual impact on productivity. Not LinkedIn testimonials. Not vendor whitepapers. Actual research, with sample sizes and methodologies you can critique. What I found was more complicated, more honest, and more useful than “AI makes everyone 40% faster.”

It doesn’t. And the reason why matters enormously for how you deploy these tools.

The Only RCT That Exists — And What It Actually Found

In the summer of 2025, METR (Model Evaluation & Threat Research) published something the AI industry quietly didn’t love: the first genuine randomized controlled trial on AI’s effect on experienced developer productivity.

Sixteen developers. 246 real tasks. Their own repositories — codebases they’d worked on for an average of five years, averaging 1.1 million lines of code. State-of-the-art tools: Cursor Pro and Claude 3.5/3.7 Sonnet. Tasks randomized to allow or disallow AI. This wasn’t a toy benchmark. This was as close to real-world as controlled research gets.

The 39-Point Perception Gap — METR RCT Results
Developers predicted before study (speed improvement)+24%
Predicted: 24% faster
Developers estimated after completing study+20%
Believed: 20% faster
What the objective timer actually measured−19%
Reality: 19% slower

Source: METR, July 2025. n=16 developers, 246 tasks, randomized design with Cursor Pro + Claude 3.5/3.7 Sonnet. Confidence interval: +2% to +39% slowdown. Note: this covers experienced developers on mature, complex codebases — not all development contexts.

The finding that gets talked about: developers were slower. The finding that doesn’t get talked about enough: they had no idea. After living through the experience, they still believed AI had sped them up by 20%. That’s not a minor estimation error. That’s a broken feedback loop — and if your developers have a broken feedback loop about their own output, your adoption metrics are measuring vibes, not value.

⚠️ Important Context

This is one study, 16 developers, complex legacy codebases. The confidence interval is wide. It doesn’t prove AI always slows developers. It proves we need better measurement — and that self-reported productivity is unreliable. Those two findings generalize much more broadly than the specific slowdown estimate.

But Wait — METR Revised Their Conclusion in February 2026

This is where it gets genuinely interesting, and where most coverage has been sloppy.

In February 2026, METR published an update acknowledging that their second study (August–late 2025, 57 developers, 800+ tasks) had a serious selection problem: 30–50% of developers they approached declined to participate if it meant working without AI for half the time. The developers most likely to benefit from AI self-selected out of the experiment.

Their conclusion: the true productivity effect is likely better than their numbers show. They estimate “AI likely provides productivity benefits in early 2026” — but they’re also candid that their data is “only very weak evidence for the size of this increase.”

📊 Updated Estimates — METR Feb 2026

Original cohort (same devs, later study): estimated −18% slowdown, CI −38% to +9%.
Newly recruited developers: estimated −4% slowdown, CI −15% to +9%.
Their interpretation: selection effects make both figures conservative lower bounds. True effect likely positive in early 2026. Size: unknown.

So where does that leave us? The honest answer is: the direction has probably flipped positive as tools improved through 2025. The magnitude is genuinely unknown. Anyone telling you they know the number is doing so without data.

Why Coding Speed Doesn’t Move Delivery Metrics — Amdahl’s Law

Here’s the piece that most productivity coverage completely ignores, and it’s the most important structural point in this whole debate.

Writing code is not all of software development. It’s roughly 25–35% of the total work cycle. The rest is design, architecture decisions, code review, testing, debugging, deployment, incident response, meetings, and thinking. Amdahl’s Law tells us something brutal about this reality:

Amdahl’s Law Applied to Software Development
Typical dev work breakdown
Code ~30%
Review ~20%
Design/Think ~25%
Other ~25%
If AI cuts coding time by 50%
Code ~15%
Review ~24%
Design/Think ~29%
Other ~30%
If AI cuts coding time by 100% (theoretical max)
Review ~29%
Design/Think ~36%
Other ~35%

Even a 100% speedup in coding yields only ~15–25% total improvement in delivery time. AI-generated code also increases review time — the Faros AI data shows 98% more PRs merged but 91% longer review time per PR, leaving net throughput unchanged.

This is why the Faros AI data makes sense. They tracked 22,000 developers over two years and found that teams with high AI adoption merged 98% more pull requests while review time increased 91%, with overall DORA delivery metrics essentially flat. AI generated more output; review absorbed the gains. The pipeline bottleneck just moved.

Sector-by-Sector Breakdown

🏭 Manufacturing J-Curve: Real, But Irrelevant to You

The MIT Sloan / Census Bureau study (2017–2021, tens of thousands of firms) is the strongest sector-level evidence we have. Clear finding: AI adoption initially cuts productivity by roughly 60 percentage points after correcting for selection bias. Recovery takes 2–4 years.

This is about capital equipment — sensors, pipelines, retraining, workflow redesign. The hardest hit are hierarchical organizations where management practices couldn’t adapt quickly. Roughly a third of productivity losses traced back there.

Why this doesn’t apply to your engineering team: manufacturing’s J-curve is a physical capital investment cycle. Software has different economics, different debt structures, different timelines. Stop using this study to justify software AI rollouts. They’re not the same thing.

💻 Software Development Probably Positive Now, Size Unknown

One RCT (METR, covered above). Multiple observational studies pointing in different directions. Microsoft Copilot research showing 26–56% speedups for junior developers on simple, unfamiliar tasks — not randomized, not seniors, not complex codebases. Different population. Different tasks. Not in conflict with METR; just not comparable.

The aggregate picture from early 2026: approximately 93% monthly adoption, 27% of production code AI-generated, organizational productivity gains stuck around 10%. The Faros AI two-year telemetry confirms this ceiling.

Security is getting worse, not better. CodeRabbit’s analysis found AI-generated code contains 2.74× more vulnerabilities than human-written code. Black Duck’s 2026 OSSRA report: known vulnerabilities per codebase up 107% year over year. More output, more attack surface, review capacity not keeping pace.

🧠 Knowledge Work No Rigorous Evidence

Workday/Hanover surveyed 3,200 employees: 85% report saving 1–7 hours weekly. Nearly 40% of that lost to rework. 14% net positive — in a survey with undefined terms and a 7× range in reported savings. You cannot make investment decisions from this.

Zapier: 92% “feel more productive,” spending 4.5 hours weekly fixing AI mistakes. Engineering roles: 5 hours of cleanup. 78% report “negative consequences.” This is a survey of feelings. It’s not nothing, but it’s not science either.

The honest summary: we have no rigorous data on knowledge work productivity from AI. Anyone selling you certainty — positive or negative — in healthcare, legal, education, or creative fields is extrapolating from vibes. We simply don’t know yet.

The Evidence Map (Actual Confidence Levels)

Sector Best Evidence What It Shows Confidence What’s Missing
Manufacturing MIT Sloan / Census Bureau (2017–21) J-curve: initial −60pp, 2–4yr recovery Moderate Post-2021 data; can’t isolate AI from other factors
Software (complex) METR RCT July 2025 + Feb 2026 update Was −19% in early 2025; likely positive by early 2026 Moderate (direction), Low (magnitude) No 2026 RCT; selection effects unresolved
Software (junior / simple) Microsoft Copilot studies (2023–24, observational) +26–56% on simple, unfamiliar tasks Low (not randomized) Selection bias; specific task types; not real-world delivery
Software (org-level) Faros AI — 22,000 devs, 2 years 75% adoption → ~10% organizational gain ceiling Moderate (observational, large n) Correlation not causation; tool variety varies
Google DORA 2024 39,000+ professionals surveyed +25% AI adoption correlated with −1.5% delivery, −7.2% stability Low (survey + correlation) Self-reported; may have reversed in 2025 DORA
Knowledge Work Workday/Hanover, Zapier (Jan 2026) Self-reported time savings; large rework overhead None (surveys only) No RCT, no objective measurement, no long-term data

Five Things the Narratives Get Wrong

❌ Common Claim

“We’re seeing 40% productivity gains from AI.”

✅ What the Data Shows

At organizational scale, Faros AI’s two-year study of 22,000 developers caps this at ~10%. The rest is perception, rework not counted, or cherry-picked task types.

❌ Common Claim

“METR proved AI makes developers slower.”

✅ What the Data Shows

METR’s February 2026 update revised this significantly: selection effects make the −19% finding a lower bound. They now estimate AI likely provides benefits in early 2026.

❌ Common Claim

“AI works the same for junior and senior developers.”

✅ What the Data Shows

Junior developers on simple, unfamiliar tasks show 26–56% gains. Senior developers on complex legacy codebases show near-zero or negative gains in the available evidence. Radically different populations.

❌ Common Claim

“More code output = higher productivity.”

✅ What the Data Shows

Faros AI: +98% PRs merged, +91% review time, flat DORA metrics. More code created more review work. Throughput didn’t move. Output is not delivery.

❌ Common Claim

“Developers know if AI is making them faster.”

✅ What the Data Shows

METR’s RCT: developers predicted +24% faster, experienced −19% slower, and reported +20% faster after the fact. The feedback loop is broken. Self-reported productivity data is not reliable.

The Gaps Nobody Has Data For (And Won’t Admit)

Honest research acknowledges what it doesn’t know. Most AI productivity content doesn’t. Here’s what genuinely remains unknown as of May 2026:

🚫 Unresolved — No Good Data Exists

Causation direction: Do AI tools slow senior developers, or do senior developers on hard problems adopt AI more? METR proves causation for their specific setup only. Everything else is correlation that could run either way.

Long-term individual trajectories: No study tracks the same developer over 2+ years with objective metrics. We have no idea if slowdowns are temporary learning curves or persistent.

Rollout speed: Fast vs. gradual AI deployment — which produces better outcomes? No experiments exist.

Technical debt compounding: AI-generated code introduces more vulnerabilities now. What’s the 3-year maintenance cost? No data.

Training ROI: How much to spend on developer AI training vs. tool licenses? Anyone claiming an answer is guessing.

“Treat every AI-generated slice like a PR from a rather dodgy collaborator who’s very productive in the lines-of-code sense, but you can’t trust a thing they’re doing.”

— Martin Fowler, framing that’s aged well

What to Actually Do in 2026 (If You Want Results, Not Stories)

  • 01
    Stop measuring adoption. Start measuring net cycle time by ticket type. “80% AI adoption” tells you nothing. “Junior developer mean cycle time on feature tickets dropped 22%; senior developer cycle time on security patches increased 8%” tells you something actionable. If you’re not segmenting by task complexity and developer seniority, you’re measuring noise.
  • 02
    Segment your AI policy, not just your teams. A blanket “use AI everywhere” policy is probably helping your juniors on boilerplate and quietly slowing your seniors on critical systems — simultaneously. The data strongly suggests treating these as separate interventions with separate measurement.
  • 03
    Build review capacity before expanding AI output. The Faros AI finding is stark: AI pushes code velocity; review time grows proportionally; net throughput doesn’t move. More AI without more review capacity isn’t a productivity strategy — it’s a bottleneck migration strategy.
  • 04
    Add security gates specifically for AI-generated code. A 2.74× vulnerability multiplier isn’t acceptable at scale. Automated SAST tools tuned for AI output patterns, higher assertion coverage requirements, and secret scan gates aren’t optional luxuries — they’re the cost of deploying AI at production velocity.
  • 05
    Budget the dip as a line item, not an optimistic assumption. Every sector-level study shows a transition cost — retraining, workflow redesign, technical debt accumulation, increased review load. If your AI business case assumes linear improvement from day one, it’s a marketing deck, not a plan. Model the dip explicitly.

The Honest Bottom Line

AI tools are getting better fast. That part is real. METR’s own researchers said they believe productivity is net positive in early 2026 compared to their 2025 results — and the trajectory of model capability makes that plausible. Nobody serious is arguing AI is permanently counterproductive.

But the gap between “getting better fast” and “delivering the productivity gains you’ve been promised” is real, measurable, and being systematically hidden by how organizations measure adoption rather than outcomes.

The perception gap is the most important finding in all of this research. Not the 19% slowdown — that’s tool- and context-specific. The finding that experienced professionals, working on their own codebases, in a controlled experiment, had no accurate sense of whether AI was helping or hurting them. That finding likely generalizes far beyond software development.

If you can’t trust developer self-reports about productivity — and you can’t; the data is explicit about this — then every AI productivity survey you’ve seen is measuring how people feel, not what’s happening. That’s a much harder problem than just buying better tools.

Measure it differently. Or admit you’re funding vibes.

Primary Sources & Further Reading

Transparency note: This analysis used AI assistance in synthesis while analyzing AI productivity limitations. I’m aware of the irony. Primary source claims have been verified against original papers where accessible. DORA 2025 full report was not publicly available as of this writing.
Last Updated: May 2026 · ← More research at AIEarnerHub