AgentEvals
What it is
AgentEvals is a library providing ready-made evaluators for assessing the performance of agentic applications, specifically focusing on agent trajectories. It offers utilities for both rule-based matching (strict, unordered, subset) and LLM-as-judge evaluation to verify if an agent's intermediate steps are logical and accurate.
The problem it solves
The black-box nature of LLMs makes it difficult to understand how changes in one part of an agent affect downstream behavior, and there is a lack of specialized tools for evaluating the intermediate steps (trajectories) rather than just final outputs.
What you could use it for
- Evaluating if an agent's tool-calling sequence matches a specific expected path
- Using an LLM to judge the logical consistency and accuracy of an agent's reasoning steps
- Integrating agent evaluation into pytest or Vitest test suites via LangSmith
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.