prexus-eval-cli
What it is
A local-first, multi-provider LLM evaluation CLI that allows users to build golden test sets and run LLM-as-a-judge evaluations. It tracks regressions across prompt and model changes, analyzes failure modes, and generates reproducible reports while storing API keys securely in the OS keychain.
The problem it solves
Lack of a local-first, secure, and reproducible CLI tool for multi-provider LLM evaluation that supports regression tracking and secure key management via OS keychains.
What you could use it for
- Tracking performance regressions when updating LLM prompts or switching models
- Generating reproducible evaluation reports for AI agent development
- Analyzing failure modes in LLM outputs using a fixed judge model
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.