Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking
What it is
This paper proposes a method to systematically test the faithfulness of natural language explanations generated by LLMs for sequential decision-making policies. It uses probabilistic model checking as an oracle to verify LLM answers against exact reference results and employs a taxonomy of query categories to generate and prioritize test cases.
The problem it solves
LLMs often generate plausible but incorrect explanations for policy decisions, and there is no existing systematic approach to test whether these explanations are faithful to the underlying environment due to the lack of a correctness oracle and structured test inputs.
What you could use it for
- Evaluating the reliability of LLM-generated explanations for reinforcement learning policies
- Benchmarking open-weight LLMs on their ability to accurately describe MDP environments
- Identifying specific types of policy behavior that LLMs struggle to explain correctly
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.