CWE-bench v2: More coverage, quality, and transparency
What it is
CWE-bench v2 is a benchmark for testing frontier models' cyber capabilities: an agent receives a checkout of a real open-source repository plus a short instruction and must find and fix a security weakness without breaking existing functionality. It releases a public set of 100 tasks covering 91 distinct CWEs, with fully programmatic, deterministic scoring, plus a held-out set for tracking progress on unseen tasks. The blog post reports results for 11 models, led by Gemini 4 Argon at 67.8% pass@1, and describes the quality checks each task passed.
The problem it solves
Security evaluation of coding agents is hard to reproduce and easy to game: prior benchmarks may leak the upstream patch, grade on a specific fix's structure rather than correct behaviour, or use instructions that read like an eval rather than a realistic request. CWE-bench v2 addresses this with programmatic deterministic scoring, hand review, verifiers that reject memorized upstream patches, and a held-out set.
What you could use it for
- Evaluating a frontier or open-weight model's ability to locate and patch real security vulnerabilities in open-source repositories
- Comparing models on a cost-vs-accuracy Pareto frontier (e.g. DeepSeek V4.1 Flash at 57.0% pass@1 for $0.04 per rollout vs Gemini 4 Argon at $3.92)
- Submitting a model's results to the public leaderboard via a PR on the model repo or the leaderboard Space
- Tracking progress on a held-out task set that no model has seen, by contacting the maintainers for access
- Auditing or contributing to individual benchmark tasks, since the public set, grader and harness are open
Why it matters
If the benchmark holds up, it gives the security community a reproducible, low-cost yardstick for how well agents can autonomously remediate real vulnerabilities — and its reported long tail (the leader still fails ~19% of tasks on every attempt) suggests substantial headroom remains before agents can be trusted for defensive security work.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.