Judgemetry: LLM Judge Reliability Lab
What it is
A Python library and CLI tool for measuring and correcting biases in LLM-as-judge evaluations, such as position bias, verbosity bias, and calibration errors. It uses pure standard library code to compute metrics with bootstrap confidence intervals and generate HTML reports, with an optional live mode for real API calls.
The problem it solves
LLM judges are widely used for evaluation but are rarely validated, leading to systematic errors (position/verbosity bias) and miscalibrated confidences that compromise the reliability of downstream decisions.
What you could use it for
- Validating the reliability of LLM judges used in automated evaluation pipelines
- Detecting and quantifying position and verbosity biases in pairwise preference tasks
- Recalibrating LLM confidence scores to better match human agreement
- Generating audit-ready report cards for LLM evaluation systems
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.