Claw-Eval
What it is
Claw-Eval is an evaluation harness for assessing LLMs as autonomous agents using 300 human-verified tasks across 9 categories. It evaluates agents on completion, safety, and robustness by auditing full trajectories and requires consistent success across three independent trials (Pass^3) to count as a pass.
The problem it solves
Existing evaluations often fail to capture the consistency, safety, and real-world robustness of autonomous agents, leading to overestimation of their capabilities due to 'lucky runs' or lack of strict safety auditing.
What you could use it for
- Benchmarking the agentic capabilities of foundation models
- Evaluating agent safety and robustness in real-world scenarios
- Comparing LLM performance on multimodal and multi-turn tasks
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.