LLM Eval Harness
What it is
A Node.js CLI tool for evaluating LLMs via OpenRouter by running custom test cases against multiple models. It combines deterministic checks for objective requirements with an optional LLM-as-judge for subjective quality, generating Markdown and JSON reports with cost and latency metrics.
The problem it solves
Generic LLM benchmarks do not reflect real-world application behavior, making it difficult to select the best model for specific tasks like JSON generation, fact-checking, or action selection.
What you could use it for
- Comparing candidate LLMs for a specific production workflow
- Validating model output against strict JSON schemas or business rules
- Benchmarking model cost and latency for a specific use case
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.