lm-evaluation-harness
What it is
A unified framework for evaluating large language models on a wide variety of academic benchmarks. It supports multiple model backends (HuggingFace, vLLM, APIs) and provides a CLI for running evaluations with configurable few-shot settings and metrics.
The problem it solves
Lack of a unified, reproducible, and efficient tool for running diverse benchmarks across different model types and inference engines.
What you could use it for
- Benchmarking LLMs on standard academic datasets like HellaSwag or ARC
- Evaluating the performance of custom fine-tuned models
- Comparing different model architectures or quantization methods
- Running the Open LLM Leaderboard tasks
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.