llm-output-quality-scorer
What it is
A CLI tool that evaluates AI-drafted outputs (specifically customer support replies) using a rubric-based LLM-as-judge approach. It scores drafts on four dimensions (relevance, correctness, completeness, tone) and includes a calibration step to compare AI scores against human labels to verify judge reliability.
The problem it solves
Teams typically rely on inconsistent, non-scalable manual inspection of a handful of examples to judge AI output quality, which fails to provide the consistency or rigor needed for launch reviews or version comparisons.
What you could use it for
- Determining if an AI drafting feature is ready for production release
- Tracking quality regressions between different LLM prompt versions
- Providing a shared, data-backed definition of output quality for engineering and design teams
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.