OpenAI withdraws SWE-Bench Pro after finding 34% of coding tasks broken
Five months after promoting SWE-Bench Pro as a fix for the compromised SWE-Bench Verified, OpenAI discovered roughly a third of Pro's tasks contain misleading prompts, overly strict tests, or inadequate coverage—and withdrew the recommendation.
OpenAI retracted its endorsement of SWE-Bench Pro this week after finding that roughly 34 percent of the benchmark's tasks are fundamentally flawed. An automated pipeline flagged 200 tasks as broken; a subsequent human annotation campaign identified 249 defective tasks out of 730 total. The root cause: SWE-Bench Pro repurposes real open-source issues and pull requests written for human collaboration, not isolated model evaluation. Overly strict tests reject functionally correct solutions that don't match unstated implementation details. Low-coverage tests pass incomplete answers. Some prompts directly contradict the test suite, steering models toward behavior the benchmark then penalizes.
In late February 2026, OpenAI had promoted SWE-Bench Pro as a successor to SWE-Bench Verified, which the lab had already declared unreliable due to training-data leakage. The new benchmark was meant to isolate coding ability from memorization, but task-design flaws surfaced within months. OpenAI now calls on the community to build evaluation sets designed from scratch for model testing rather than repurposing human workflows. One in three tasks in the Pro set cannot reliably separate signal from noise.
