PropertyEval
What it is
A dataset of manually constructed property-based tests (Hypothesis strategies) for the HumanEval benchmark, designed to thoroughly evaluate LLM code generation. It includes a script to run these tests against canonical solutions and code samples, demonstrating that property-based testing can match the thoroughness of mutation-based testing.
The problem it solves
Standard unit tests in code generation benchmarks often fail to cover edge cases, leading to overestimation of LLM coding capabilities.
What you could use it for
- Benchmarking LLM code generation models with more thorough test cases than standard unit tests
- Validating the correctness of generated code against canonical solutions using property-based testing
- Contributing improved test contracts to existing benchmarks like EvalPlus
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.