sigil: Error bars for LLM-as-judge evals
What it is
A dependency-free TypeScript library and CLI tool that computes statistical reliability metrics (calibration, finite-sample risk certificates, drift monitoring) for LLM-as-judge evaluations. It processes user-provided judge verdicts and human labels to generate a byte-reproducible audit report with error bars, without calling any models or requiring network access.
The problem it solves
LLM-as-judge evaluations often lack statistical rigor, relying on single accuracy figures without error bars or calibration checks, which makes it difficult to trust cost-savings claims or detect model drift.
What you could use it for
- Auditing the reliability of LLM-as-judge evaluation pipelines
- Determining optimal model routing policies based on cost-quality trade-offs with statistical guarantees
- Monitoring drift in automated evaluation systems over time
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.