Terminal Agent RL Bench
What it is
An end-to-end reinforcement learning benchmarking pipeline for LLM coding agents that generates synthetic tasks, executes them in Docker sandboxes, and evaluates results using LLM-as-a-Judge. It is built on the Terminal-Bench 2.0 task format and supports models like Qwen2.5-Coder and Llama-3.3.
The problem it solves
Lack of a unified, end-to-end pipeline for generating, executing, and evaluating coding agent trajectories in a sandboxed environment with reliable automated scoring.
What you could use it for
- Benchmarking the performance of open-weight coding agents on terminal tasks
- Generating synthetic training data for reinforcement learning of coding agents
- Evaluating the reliability and consistency of LLM-based judges for agent trajectories
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.