Gnomon
What it is
Gnomon is a Python library designed to add statistical rigor to LLM agent evaluations. It provides tools for bootstrapped confidence intervals, power analysis, and judge calibration to help teams make data-driven ship/don't-ship decisions.
The problem it solves
Most LLM evaluation tools report point estimates without uncertainty, leading teams to make shipping decisions based on differences that are within the margin of error.
What you could use it for
- Determining if a new LLM agent version is statistically significantly better than the previous one
- Calculating required sample sizes for A/B tests to detect specific effect sizes
- Calibrating LLM judges by measuring agreement with human raters using Cohen's kappa
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.