SWT-Bench
What it is
SWT-Bench is a NeurIPS 2024 evaluation harness for benchmarking LLMs on repository-level test generation. It uses Docker to validate whether generated tests fail before a bug fix and pass after, measuring model performance on real-world GitHub issues.
The problem it solves
Lacks a standardized, reproducible benchmark to evaluate LLMs on generating tests that correctly identify and validate real-world bug fixes in existing codebases.
What you could use it for
- Evaluating the effectiveness of LLMs in generating reproducing tests for real-world bugs
- Benchmarking code agents on their ability to write tests that validate issue resolutions
- Comparing different LLMs and agents on the SWT-Bench Lite and Verified leaderboards
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.