StateBench: Testing how language models track state changes over long documents
What it is
StateBench is a benchmark from Ai2 for evaluating long-context state tracking — how well language models keep track of entity information as it evolves across long documents. It defines four difficulty levels of increasing state-update complexity (R-trivial immutable assignment, aperiodic reassignment by value, soluble self-referential updates, and general finite assignment by reference), analogous to variable assignment operations in code. The authors report that recurrent and hybrid architectures generalize farther than transformers when trained directly on these tasks, while frontier models still degrade on harder state-tracking variants as inputs grow longer.
The problem it solves
Existing state-tracking tests focus only on the hardest cases, so researchers lack a way to tell which kinds of state tracking (simple vs. chained) a given architecture can actually handle.
What you could use it for
- Benchmarking long-context LLMs on how well they track entity state changes across long documents
- Comparing transformer vs. recurrent/hybrid architectures on state-tracking generalization
- Diagnosing which complexity level of state tracking (immutable, reassignment, self-referential, by-reference) a model fails at
- Motivating and evaluating architectures with running memory such as Gated DeltaNet
Why it matters
Provides a graded diagnostic for state tracking rather than only the hardest cases, which could guide architecture choices for long-context tasks like code understanding and narrative comprehension, and give a clearer picture of where frontier models still break down.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.