kappa-gate
What it is
An LLM-as-judge evaluation harness that requires candidate judges to pass a Cohen's kappa calibration gate against hand-labeled reference data before their scores are trusted. It computes agreement statistics and refuses uncalibrated judges, ensuring that only reliable graders can gate downstream releases.
The problem it solves
Most eval pipelines trust LLM judges implicitly without verifying if the grader can actually grade consistently, leading to potential false positives/negatives in automated quality gates.
What you could use it for
- Validating the reliability of LLM judges before using them for automated QA
- Detecting degenerate judges that exploit surface features to achieve high accuracy but low inter-rater agreement
- Calibrating evaluation rubrics by identifying dimensions where judge agreement is weak
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.