KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use
What it is
KaliBench is a benchmark and dataset containing 8,504 natural language to CLI command pairs for 1,642 tools on Kali Linux. It uses a multi-stage verification pipeline involving LLM validation, sandboxed execution, and human refinement to ensure commands are semantically correct and executable.
The problem it solves
Existing evaluations fail to measure LLMs' ability to generate executable commands for real-world cybersecurity tools, where minor syntax errors or argument misordering invalidate execution.
What you could use it for
- Evaluating the ability of LLMs to generate accurate cybersecurity CLI commands
- Training smaller models via supervised fine-tuning and reinforcement learning with verifiable rewards
- Benchmarking open-weight models on specific security phases and capability dimensions
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.