CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents
What it is
CrossSWE is a proposed benchmark for evaluating coding agents on software engineering tasks that span multiple interdependent repositories. It contains 171 real-world instances covering 165 repositories, 20 technical domains, and 13 programming languages, and it validates each instance by checking that modifying only part of the repositories is insufficient to produce the intended behavior. The authors report evaluating five LLMs, with the best scoring 58.48% Pass@1 and cross-repository inconsistency as the most common failure mode (44.2%).
The problem it solves
Existing software engineering benchmarks concentrate complexity within a single repository, so they do not test the dependency reasoning, change propagation, and interface consistency problems that arise when a change must be made jointly across interdependent repositories.
What you could use it for
- Evaluating coding agents on tasks that require coordinated edits across multiple interdependent repositories
- Measuring failure modes such as dependency reasoning errors, change propagation gaps, and interface inconsistency
- Comparing LLM/agent performance across 13 programming languages and 20 technical domains
- Providing a held-out benchmark for research on multi-repository software engineering agents
Why it matters
If adopted, it would shift agent evaluation beyond single-repository tasks toward the multi-repo coordination that real-world development requires, giving researchers a way to measure and improve dependency reasoning and cross-repo consistency.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.