Asking Earns Nothing: Scoring the Decision to Act in BFCL Multi-Turn
What it is
This paper identifies a flaw in the BFCL multi-turn benchmark where the official scorer ignores turns where the agent should ask for clarification, effectively rewarding guessing over asking. It introduces a turn-level scorer that evaluates the specific decision to act versus ask by comparing paired items (complete vs. incomplete information) and releases a dataset of 31 manually verified bad items.
The problem it solves
Current agent benchmarks like BFCL multi-turn fail to score the decision to ask for clarification, leading to misleading rankings where models that guess on incomplete information are favored over those that correctly identify missing information.
What you could use it for
- Evaluating LLM agents on their ability to distinguish when to request clarification versus when to execute actions.
- Debugging agent behavior by analyzing decision accuracy on paired benchmark items.
- Improving agent prompts by testing how instructions to 'ask' or 'act' affect performance metrics.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.