Through a comprehensive, multi-step quality assurance audit combining automated filters, human-supervised investigator agents, and independent human reviews by experienced engineers, OpenAI has identified widespread task issues in SWE-Bench Pro, leading them to estimate that ~30% of the benchmark’s tasks are fundamentally broken. While the 731-task public split initially showed frontier models jumping from a 23.3% to an 80.3% pass rate over eight months, this data pipeline analysis revealed that evaluation flaws specifically overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts severely compromise the integrity of the results.
Alsi Read: Persistent and Nagarro Form $2.9 Billion AI-Led Digital Engineering Powerhouse via Strategic Merger
“Accurately measuring our models’ capabilities is important for sound deployment and safety decisions, including decisions under OpenAI’s Preparedness Framework,” the organization emphasized, noting that flawed benchmarks misrepresent safety cases and skew research priorities. Because open-source repository pull requests are natively built for human collaboration rather than clean AI evaluation, hidden tests frequently demand specific implementation details that differ from prompt instructions. This discovery mirrors previous design and contamination flaws uncovered in SWE-bench Verified, illustrating the immense difficulty of curating hard yet fair benchmarks. Consequently, OpenAI has announced, “Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.” Moving forward, OpenAI calls upon the broader AI community to build new benchmarks designed by human software developers explicitly for model capabilities, ensuring future evaluations remain “hard to game, easy to trust, and genuinely reflective of model capability or alignment.”
