Technology

Where do AI agents still fail on this popular software engineering benchmark?

November 10, 202610 min read

Frontier AI coding agents now resolve around 70% of SWE-bench Verified - the benchmark every major AI lab uses to track real-world bug fixing. Three years ago the same number was under 5%. The trajectory is real progress and the headline impresses on its own.

The remaining 30% is more interesting.

This post is about what’s in that gap: what the benchmark actually measures, how the top labs ended up converging on it, and the five distinct ways AI coding agents still fail when you put them in front of real production bugs.

The benchmark every AI lab now competes on

SWE-bench came out of Princeton in late 2023. The design was deliberately punishing. Instead of toy coding puzzles, it scraped real bug fixes from a dozen large open-source Python projects - Django, astropy, sympy, matplotlib, scikit-learn, sphinx, flask, requests, pytest, and others - and turned each one into a test. You give the model the bug report and the repository at the broken commit. It produces a patch. The patch gets graded by the project’s own test suite, running in Docker. No human review, no LLM-as-judge. The failing tests either pass after the patch or they don’t.

When the first paper ran the numbers, GPT-4 resolved about 2% of instances. Claude was similar. The benchmark embarrassed the entire field on release, which is part of why it got attention.

In August 2024, OpenAI released SWE-bench Verified - a 500-instance subset hand-curated for clarity. (A small fraction of the original instances had ambiguous bug reports or hidden test dependencies.) Verified became the version everyone uses. Anthropic’s Claude, OpenAI’s GPT and o-series, Google’s Gemini, DeepSeek’s reasoning models - every major lab posts to the leaderboard, and most publish their scaffolding alongside the score.

The climb has been steep: under 5% in late 2023, north of 70% today. Read only the headlines and you’d conclude AI has roughly solved bug fixing. It hasn’t.

What an instance actually looks like

Each of the 500 instances ships with four things. A natural-language bug report copied straight from the original GitHub issue - sometimes terse, often a wall of text. The exact commit the bug existed at, so the agent works against the codebase as it was. A set of FAIL_TO_PASS tests - the tests broken by the bug, which should pass after a correct fix. And a set of PASS_TO_PASS tests - the tests that currently work and need to stay working.

Both test sets come from the eventual human fix. They’re the ground truth.

The agent’s job is to produce a unified-diff patch. The harness drops the patch into a fresh Docker container, applies it, and runs both test suites. The verdict is binary: every FAIL_TO_PASS passes and every PASS_TO_PASS still passes - resolved. Anything else - unresolved. No partial credit, no escape valve.

That’s what makes the benchmark hard to game. It’s also what makes the failure modes worth a closer look.

How an agent approaches a bug

Every code-fixing agent - frontier, indie, or otherwise - does three things in sequence.

It localizes: given a bug report, find the right file or files. It can search the codebase and read files, but it can’t run the code yet. It generates: given those files, write a diff. This is where the agent’s grasp of the bug’s actual cause (versus its visible symptoms) gets tested. It verifies: most modern agents include a second LLM that reviews the patch before submission, retrying if the reviewer rejects.

Each step has its own characteristic failures.

The five ways agents still fail

What follows comes from analyzing 250 SWE-bench Verified instances run through a production-grade RCA + AutoFix coding agent. 93 of the patches failed the harness; 83 of those fall cleanly into one of five categories.

Wrong file

The agent edited a file that didn’t contain the bug. The patch applied, the agent’s own verifier approved it, the tests failed anyway because the relevant code lived elsewhere.

A clean example from the run: a Django bug where the agent edited django/db/models/query_utils.py, while the actual fix needed to land in django/db/models/enums.py. Both files referenced the symbol in the bug report. Only one was on the call path the failing test exercised, and the agent had no way to tell them apart without running the code.

LLMs localize on lexical similarity - keyword overlap, import patterns, naming conventions. Call-graph reachability isn’t something you can read off the surface of a file.

Right file, wrong fix

The most common failure mode by a wide margin. The agent found the correct file, then wrote a patch that addressed the bug’s symptom rather than its underlying cause. Syntactically clean, plausible at a glance, entirely ineffective.

LLMs are very good at producing code that looks like a fix. The harder step - given exactly what this test asserts, what specifically about current behaviour needs to change to satisfy it - is where they slip. The gap between plausible and correct is where most generation errors live.

Partial fix

The bug had multiple manifestations and the agent fixed one. Bug affects both None and empty list; agent fixes None, doesn’t notice the empty case. Some FAIL_TO_PASS tests pass after the patch, the rest don’t.

Without an explicit enumeration step in the prompt, models tend to fix the most salient instance of a problem rather than the full class of inputs that exhibit it.

Wrong and regressed

The most damaging category. The agent rewrote a function more aggressively than the bug required, broke unrelated tests, and didn’t actually fix the original bug. Net negative.

A correctness-only training signal doesn’t penalize unrequested cleanup - only test failures do, and models are optimistic about failures they haven’t run yet. Without an explicit scope constraint, “fix this bug” tends to drift into “rewrite this function.”

Over-fix

The bug got fixed; valid neighbouring code broke. The agent’s patch encoded an assumption about how the function should behave that didn’t hold for callers it never inspected.

LLMs reason locally. The agent sees the failing case and 50 lines of surrounding code. It doesn’t see the 30 call sites that depended on the old behaviour.

The thread across all five: when a model can read what it needs to reason about, it does well. When it has to infer - call graphs, sibling cases, downstream callers, the gap between a symptom and a cause - it doesn’t. Every concrete improvement to a code agent is, structurally, a way to surface more of that hidden context.

When AI grades AI

Of the 93 patches the harness rejected, around 95% had been approved by the agent’s own verifier - itself an LLM - before submission. The agent said the patch fixed the bug. The tests said no.

This is a structural property of LLM-as-judge architectures, and it isn’t specific to code.

A verifier that can’t see ground truth grades plausibility, not correctness. The verifier in this case saw the patch and the surrounding code. It didn’t see the FAIL_TO_PASS test bodies - the actual specification of what “fixed” meant. So when it asked “does this fix the bug?”, the question it was actually answering was “does this look like a fix?”

The pattern shows up across the industry. RAG systems with LLM-as-judge evaluators, chatbot quality-scoring pipelines, agent self-reflection loops, automated code reviewers - anywhere an LLM grades another LLM’s output without access to the executable specification (the test, the canary, the oracle), the failure looks the same: high approval rates and a substantial slice of false positives.

The fix isn’t a better prompt. It’s giving the verifier access to the artifacts the system actually has - failing tests, canaries, evals - and lowering its confidence explicitly where it doesn’t.

Reading the trajectory

The headline number on SWE-bench Verified will keep climbing. What’s worth tracking isn’t the percentage - it’s how the failure distribution shifts. The five categories above won’t disappear. Their proportions will move. The first one to shrink, and what the labs do to shrink it, will say more about real progress in code agents than the topline ever does.

The benchmark is one of the few honest yardsticks in AI right now. The tests don’t care how clever the prompt was.

Author

Tanay Agarwal

Tanay Agarwal

Software Engineer