A Taxonomy of Failing Bug Reproduction Tests on SWT-Bench
Automatic Program Repair (APR) tools require high-quality test generation models to validate candidate patches. Despite the advancements of automated bug reproduction testing tools, they continue to struggle with issues of higher complexity. These struggles may be caused by a framework's lack of problem-specific context. We address the unclassified nature of these framework failures by providing a taxonomy of bug reproduction tests. This study uses the AssertFlip and OpenHands frameworks, the two highest-scoring open-source tools on SWT-Bench Verified. The significance of model selection on the causes of failure is also explored by utilizing GPT-5-mini and GPT-4o-mini on both tools. The error traces of these frameworks are categorized to determine granular modes of failure. Through this process, mechanical and implementation failures were the most common failure classes. Overall, a tool's generation approach has a significant impact on precision and coverage, while the selection of the model alters the quality and quantity of generated tests. Specifically, more powerful models with higher reasoning capability tended to present fewer failures due to a lack of technical or…
Taxonomy of failing bug reproduction tests for APR tools.
- arXiv
- 2611.13395
- Authors
- Luca Filippelli, Hamed Taherkhani, Hung Viet Pham, Hadi Hemmati
- Categories
- cs.SE