BehaviorCI
What it is
BehaviorCI is a pytest plugin that performs behavioral regression testing for LLM applications by comparing generated text outputs using semantic similarity (embedding cosine) rather than exact string matching. It allows developers to record baselines of LLM outputs and fail CI builds when the meaning drifts, while supporting lexical guardrails and local-first operation.
The problem it solves
Traditional unit tests using exact string matching are ineffective for LLM outputs because generative text is never byte-stable, leading to false positives or missed regressions when prompts, models, or temperatures change.
What you could use it for
- Detecting semantic regressions in LLM prompts after model or temperature changes
- Testing agent skills to ensure instructions produce consistent behavior
- Validating that LLM outputs maintain specific tone or structural elements in CI pipelines
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.