AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
What it is
AgentHorizon is a benchmark of 1,373 computer-use tasks designed to evaluate the reliability of automatic judges on long-horizon tasks. It uses a paired design where instructions are swapped between similar trajectories to test if judges can distinguish successful executions from those that violate constraints or introduce side effects.
The problem it solves
Existing automatic judges are unreliable on long, multi-application tasks, often failing to detect hidden constraint violations or unwanted side effects in trajectories that appear complete.
What you could use it for
- Evaluating the reliability of LLM-based judges for computer-use agents
- Benchmarking agentic judges on their ability to detect subtle failures in long interaction histories
- Training or selecting judges for automated evaluation of complex multi-application tasks
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.