What Actually Keeps an AI Benchmark Useful? Scale
A systematic study of 60 LLM benchmarks finds 29 have saturated: top models are statistically indistinguishable. Age and test set size predict saturation; private test sets, multilingual scope, and open-ended formats don't protect against it once age is controlled for.
Nearly half of the benchmarks the field relies on to compare language models can no longer tell top models apart. Of 60 widely used text benchmarks analyzed in a new systematic study, 29 show high or very high saturation, meaning leading models land within noise of each other and the benchmark has stopped discriminating.
The paper, âWhen AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationâ from the EvalEval Coalition, is the first attempt to define saturation rigorously and test which design choices actually protect against it. The answer to that second question is uncomfortable: most of the safeguards people assume work (private test sets, harder output formats, multilingual scope) donât, once you control for how old the benchmark is.
- 29 of 60 benchmarks are highly or very highly saturated (Sindex ⼠0.7); 14 are past 0.9
- Saturation climbs with age: mean saturation index 0.51 â 0.52 â 0.60 across young, mid, and old benchmark cohorts
- Private test sets show no advantage over public ones (56 public vs. 4 private, no significant difference)
- Citation count loses its correlation with saturation once age is controlled for (Ď=0.22, p=0.12)
- A joint Bayesian model explains saturation well (R² = 0.884), driven mainly by age and test set size
What Counts as âSaturatedâ
The authors define saturation as the loss of reliable discriminative power among top-performing models: when the gap between the best systems is smaller than the evaluation noise, the benchmark can no longer rank them. They formalize this as a Saturation Index, built from the normalized score range among the top-k models (default k=5) versus the expected statistical noise of the benchmark, given its test set size. The index runs from 0 (wide open) to 1 (leaderboard is noise), bucketed into five bands from âvery lowâ to âvery high.â
They applied this to 60 benchmarks spanning 1 to 114 months old, annotated by 23 researchers across 14 properties: public vs. private, English-only vs. multilingual, expert-curated vs. crowdsourced, closed-ended vs. open-ended, templated vs. non-templated, and more. That annotation set is what lets them test specific hypotheses about what should, in theory, keep a benchmark useful for longer.
Saturation Index by benchmark age cohort
Mean Sindex rises with age; share of highly saturated benchmarks nearly doubles from youngest to oldest cohort.
The Safeguards That Donât Hold Up
The paper tests six hypotheses about what should slow saturation. Most fail once age is accounted for:
- Private test sets (H1): No statistically meaningful gap between public and private benchmarks. Keeping a test set secret doesnât, by itself, preserve discriminative power.
- Output format (H4): Closed-ended (multiple choice) and open-ended benchmarks show no significant difference in an age-balanced comparison (p=0.40). Harder-to-guess formats arenât inherently more durable.
- Templating (H6): Templated vs. non-templated benchmarks show no significant difference (p=0.10).
- Multilinguality (H2): Multilingual benchmarks look more robust, but theyâre also 16 months younger on average (32.9 vs. 48.9 months). The apparent advantage is an age confound, not a property of multilinguality itself.
- Adoption (H5): Citation count correlates with saturation in raw form but loses significance once age is controlled for (Ď=0.22, p=0.12); citation growth rate and technical-report frequency show no link at all.
What does hold up: test set scale is one of the strongest predictors of lower saturation, and expert curation looks like it helps at a given age, though crowdsourced benchmarks in the sample skew older, which complicates a clean read on that one (p=0.0017 age confound). A joint Bayesian regression combining all factors reaches R² = 0.884, with age and test set size doing most of the explanatory work.
Why This Matters
A saturated benchmark doesnât just get less useful, it actively misleads. Once top models are within noise of each other, a 0.3-point leaderboard gap that gets cited as âstate of the artâ is often not statistically distinguishable from a tie. The paperâs own framing is that saturation is neutral in principle (sometimes it means a task really is mastered) but becomes a problem exactly when it reflects measurement noise rather than genuine capability ceilings, and most leaderboards donât report the uncertainty needed to tell the difference.
The recommendations are practical: report confidence intervals alongside leaderboard scores, scale up test set size as a first lever, favor expert-curated data collection, and set explicit retirement or refresh criteria for benchmarks rather than letting them accumulate citations past the point of usefulness. None of that requires new benchmark science, just treating saturation as measurable rather than assumed.
Caveats
The benchmark set (60 total) skews toward the fieldâs most cited, longest-running evaluations, which likely overrepresents exactly the benchmarks under saturation pressure. Property annotations are treated as static even though benchmarks get revised post-release (updated splits, new protocols), and leaderboard data quality varies since many boards arenât updated consistently for every new model. The uncertainty estimates are also built for accuracy-style metrics over fixed test sets; benchmarks scored with Elo ratings, pass@k, or LLM-judge grading would need different variance handling not covered here. And throughout, the paper is careful to note itâs establishing correlations, not causal mechanisms, age is confounded with several other properties in ways the data canât fully untangle.
Liked this? We send one like it every week.
Best papers, one email. No spam.