RAGCheck
What it is
RAGCheck is a Python library and CLI tool for evaluating Retrieval-Augmented Generation (RAG) pipelines. It provides quality metrics, judge validation via Cohen's kappa, refusal calibration, and CI-friendly regression checks, allowing users to verify the reliability of LLM-based evaluations.
The problem it solves
Existing RAG evaluation tools often rely on unvalidated LLM judges and lack mechanisms to verify judge reliability, measure refusal behavior, or integrate easily into CI/CD pipelines for regression detection.
What you could use it for
- Validating LLM judges against human labels before trusting automated metrics
- Detecting hallucinations and refusal calibration issues in RAG systems
- Running CI regression checks to ensure code changes do not degrade RAG quality
- Benchmarking different RAG architectures (e.g., hybrid vs. dense retrieval)
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.