AgentEval
What it is
AgentEval is a local, zero-dependency testing framework for evaluating AI coding agents. It provides a Vitest-like API for defining tests, supports isolated parallel execution via Docker or macOS sandbox-exec, and includes an LLM-as-a-judge system with a built-in SQLite dashboard for tracking results and A/B comparisons.
The problem it solves
Lack of a standardized, local, and reproducible way to evaluate the quality, consistency, and cost-effectiveness of AI coding agents.
What you could use it for
- Benchmarking the performance of different LLMs (e.g., Claude vs. GPT-4o) on specific coding tasks
- A/B testing prompt engineering strategies to determine which yields more reliable code
- Validating that AI-generated code passes deterministic checks like builds and linting
- Tracking the stability and consistency of an AI agent over multiple iterations
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.