SWE-bench Science
What it is
A repository-level benchmark for evaluating coding agents on scientific software engineering tasks across 20 domains. It includes 119 tasks from 98 GitHub repositories, categorized into issue-driven, expert-exploratory, and engineering-integration paradigms.
The problem it solves
Existing coding agent evaluations focus on general software tasks and lack insight into the specific challenges of scientific software engineering.
What you could use it for
- Benchmarking the ability of AI agents to repair scientific software
- Evaluating agent performance in handling scientific domain knowledge
- Studying failure mechanisms in automated scientific software engineering
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.