judge-reliability
What it is
A research repository that calibrates LLM-as-judge reliability using 1,814 human-labeled MT-Bench comparisons. It measures agreement against human ceilings and baselines, quantifies position and verbosity biases, and evaluates the effectiveness of debiasing techniques like swap averaging.
The problem it solves
The lack of standardized calibration for LLM judges, leading to over-reliance on potentially biased or unstable automated evaluations without understanding their limits relative to human judgment.
What you could use it for
- Calibrating LLM judges before deploying them for automated evaluation
- Benchmarking the reliability of different LLM models as judges
- Analyzing the impact of prompt sensitivity and presentation order on LLM judgments
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.