llm-eval-simple
What it is
A Python-based framework for benchmarking LLMs by sending prompts to OpenAI-compatible APIs and evaluating responses against expected answers. It supports multiple models, tracks response times, and provides a React-based web dashboard for visualizing results.
The problem it solves
Lack of a lightweight, configurable tool for running batch evaluations and visualizing results for multiple LLMs against custom prompt/answer pairs.
What you could use it for
- Benchmarking accuracy and latency of multiple LLMs on a custom dataset
- Comparing performance between different model versions or providers
- Visualizing evaluation results in an interactive web dashboard
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.