Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
What it is
This paper documents how keyword-matching benchmarks can falsely credit small language models with tool-use capabilities they do not actually possess. It proposes a ladder of strict, cheap diagnostics, including verbatim-reproduction checks and first-token probes, to distinguish genuine tool-use behavior from memorized patterns or missing priors.
The problem it solves
Lenient keyword-matching benchmarks can incorrectly credit small models for tool use they never perform, leading to misleading capability assessments and potential safety risks.
What you could use it for
- Validating tool-use claims in small language models before deployment
- Diagnosing why a model fails to emit valid tool calls despite high benchmark scores
- Designing targeted SFT recipes to repair broken tool-use triggers in existing models
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.