LLM Intent & Entity Evaluation Framework
What it is
A Python framework that uses Large Language Models (via Vertex AI) to evaluate ASR models by measuring semantic intent and entity preservation rather than just word error rate. It processes CSV/JSONL datasets containing ground truth and ASR outputs to generate scores indicating whether the core meaning and key entities were correctly captured.
The problem it solves
Traditional ASR metrics like WER and CER penalize all textual variations equally, failing to distinguish between benign errors that preserve meaning and severe errors that break user intent or alter key entities.
What you could use it for
- Evaluating ASR model performance for Indic languages where minor spelling variations do not change meaning
- Assessing the functional utility of voice command systems by checking if user intent is preserved
- Comparing ASR models based on semantic accuracy rather than strict textual match
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.