llm-judge-calibration
What it is
A Python CLI tool that calibrates LLM-as-a-judge models against human labels by computing Cohen's kappa, accuracy confidence intervals, and bias probes. It provides an honest verdict on which categories the judge is reliable for and which are not.
The problem it solves
Raw accuracy metrics hide specific areas where an LLM judge disagrees with humans or exhibits biases, leading to false confidence in automated grading.
What you could use it for
- Validating the reliability of an LLM-based auto-grader before deployment
- Detecting position bias in pairwise LLM evaluations
- Identifying categories where an LLM judge is anti-correlated with human judgment
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.