On the estimation and validity of AI time horizons
What it is
This paper analyzes METR's 50% time horizon metric using splines and item-response theory to relax the linear assumption between AI difficulty and human task time. It demonstrates that the relationship is non-linear, specifically flattening between 2-30 minutes, which implies that time horizon jumps in this range are less significant than those at longer durations.
The problem it solves
The standard METR time horizon metric assumes a linear relationship between AI difficulty and the logarithm of human time, which can misrepresent the significance of capability jumps in specific time ranges.
What you could use it for
- Interpreting METR time horizon benchmarks with statistical rigor
- Designing new AI capability benchmarks that account for non-linear difficulty scaling
- Assessing the construct validity of time-based AI metrics
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.