Gold vs. Translation: Proxy Validity of Translated Benchmarks
What it is
This paper investigates whether translated versions of English benchmarks (like MMLU) measure the same constructs as natively created benchmarks in non-English languages. It evaluates 29 models across five Nordic and Baltic languages to test if translated scores add predictive value for native performance, finding that validity depends heavily on content overlap with the native target.
The problem it solves
It is unclear whether translated benchmarks measure the same constructs as native ones, leading to potential misinterpretation of model capabilities in non-English languages.
What you could use it for
- Evaluating the validity of using translated English benchmarks for non-English LLM assessment
- Selecting appropriate evaluation metrics for multilingual model comparisons
- Understanding the limitations of proxy validity in cross-lingual benchmarking
Why it matters
Provides a rigorous framework for assessing the validity of translated benchmarks, helping researchers and developers avoid misleading conclusions when evaluating multilingual capabilities of LLMs.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.