AgentCompass
What it is
AgentCompass is an open-source evaluation framework for LLM and VLM agents that decouples models, benchmarks, harnesses, and execution environments. It provides a unified interface to run 20+ benchmarks and 10+ agent harnesses with features like concurrent scheduling, fault tolerance, and trajectory analysis.
The problem it solves
The lack of a unified, extensible framework to evaluate LLM agents across diverse benchmarks and execution environments, which often requires custom, non-reproducible scripts for each specific combination of model and task.
What you could use it for
- Systematically benchmarking the capabilities of different LLM agents on standardized tasks like SWE-bench
- Comparing the performance of various agent harnesses (e.g., Claude Code, OpenHands) across multiple benchmarks
- Analyzing agent trajectories and tool calls to identify failure modes and abnormal behavior
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.