REVIEW
What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
Computers and Society · Submitted 8 Sep 2026 · 90 references
Pith's verdict: Solid, large-scale empirical work on benchmark validity, but the second finding about capability concepts is undercut by an acknowledged but unaddressed confound (general ability factor).
T0 · AI review
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
Applied convergent and discriminant validity from the social sciences to systematically interrogate 56 AI benchmarks, finding that most do not reliably measure the concepts they purport to measure—safety benchmarks show weak convergence…
The reading
What carries the argument
The central machinery is the adaptation of convergent and discriminant validity from the Campbell–Fiske multi-trait multi-method matrix, operationalized through Spearman rank correlations of model-level scores across benchmarks and through item response theory (1PL) models that test whether a pair of benchmarks is better described by one shared latent trait or by two separate traits (ΔAUC). The grouping of benchmarks into 'assigned concepts' is based on the concept each benchmark's introductory paper claims to measure. Benchmark-level correlations are aggregated by assigned-concept pairs with cluster-bootstrapped confidence intervals; item-level comparisons use ΔAUC with a hold-out set and bootstrap. A relabeling permutation test quantifies whether an individual benchmark's correlation vector is closer to a hypothesized alternative concept than to its own assigned concept.
Extended reading notes
Core claim
Using adapted convergent and discriminant validity (inspired by the multi-trait multi-method matrix), the paper demonstrates that AI benchmarks suffer from systematic construct-validity failures. For safety concepts, within-concept correlations are weak (median ρ near zero for refusal, bias, safety detection). For capability concepts, within-concept correlations are barely higher than cross-concept correlations (e.g., reasoning vs. knowledge: mean ρ ≈ 0.55 both ways), so the benchmarks do not distinguish the concepts they name. Design artifacts dominate: a partial Mantel test shows that shared score format (especially LLM-judge scoring) predicts pairwise benchmark correlations more strongly than shared assigned concept (β_format = 0.526, β_concept = −0.058). Individual benchmark interrogation via a relabeling permutation test confirms that BBQ-accuracy correlates significantly more with reasoning benchmarks than with bias benchmarks (relabeling statistic = 0.15, p < 0.001), and DecodingTrust-Fair correlates more with knowledge than with bias (0.14, p = 0.002). These patterns are robust to model generation split and choice of LLM judge.
Load-bearing premise
The paper groups benchmarks into assigned concepts based on what each benchmark's introductory paper says it measures, but those concepts are often underspecified or ambiguous; if the grouping is loose or inaccurate, the convergence and discrimination analyses may reflect grouping artifacts rather than true measurement properties.
Editorial extensions
If this is right
- Benchmark developers should routinely assess convergent and discriminant validity during design, using correlation matrices and IRT-based checks, to detect misaligned constructs before release.
- Safety evaluation suites should reconsider reliance on any single bias benchmark like BBQ: if BBQ-accuracy measures reasoning rather than bias, using it alone (as many commercial model cards do) misses the intended construct.
- Capability benchmarks claiming to measure distinct concepts (reasoning, knowledge, comprehension) likely tap a general ability factor; researchers should either acknowledge this or design evaluations that deliberately suppress shared variance.
- Score format (especially LLM-judge scoring) introduces a strong method effect that can masquerade as conceptual agreement; controlling for format in multi-benchmark studies is essential for valid interpretation.
- The released item-level dataset enables other researchers to replicate and extend validity analyses without re-running large-scale model evaluations, lowering the barrier to construct-validity checks in AI.
- The paper confirms that IRT models, typically used for item selection, can serve a diagnostic role: a positive ΔAUC flags pairs of benchmarks that require separate latent traits, signaling poor conceptual overlap.
Reading between the lines
- The finding that capability benchmarks are empirically indistinguishable suggests that many AI leaderboards may rank models primarily by a single general ability dimension, not by the specific competencies the evaluators intend to highlight. This would undermine claims of 'state-of-the-art reasoning' or 'superior knowledge' as separate differentiators.
- The method pioneered here could be extended to multimodal benchmarks (vision, speech) and to human evaluation setups, where concepts like 'helpfulness' or 'harmlessness' are even more underspecified. The convergent–discriminant framework provides a principled way to test whether different evaluation protocols measure the same thing.
- If design artifacts dominate conceptual content, then benchmark designers should document not only the target concept but also the method (format, prompt, scoring function) as an integral part of the measurement. Releasing only the concept label is insufficient for reproducibility or validity.
- The paper's reliance on assigned concepts (grouped by reading benchmark papers) is a necessary but fragile step. A more rigorous alternative would be a formal content analysis or expert panel to define concept boundaries—though the paper's findings already suggest that even generous grouping fails to produce convergence.
Original abstract
Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal). We adapt convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, applying it to 56 capability and safety benchmarks across 53 models. We label benchmarks with substantively similar purported concepts to a shared assigned concept, and ask whether model rankings on benchmarks with the same assigned concept correlate more strongly than rankings on benchmarks with different assigned concepts. We ask analogous questions at the item level using item response theory (IRT) models. We find that correlations between model rankings on benchmarks with the same assigned safety concepts are often weak, suggesting these concepts may be conceptualized inconsistently across benchmarks. For assigned capability concepts (e.g., reasoning, knowledge), model rankings are often as strongly correlated among benchmarks with the same assigned concept as between benchmarks with different assigned concepts, suggesting these capability concepts may not discriminate well from one another. In some cases, benchmarks that share design elements (e.g., score format) correlate more strongly than benchmarks with the same assigned concept. Finally, some individual benchmarks correlate more strongly with benchmarks assigned a different concept than with benchmarks sharing their own assigned concept, suggesting they may measure a different concept than they purport to. For example, BBQ-accuracy correlates more strongly with benchmarks labeled reasoning than with benchmarks that share its assigned concept, bias. To support future empirical work on benchmark validity, we release our extensive dataset of model outputs and scores at the item- and benchmark-level.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal). We adapt convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, applying it to 56 capability and safety benchmarks across 53 models. We label benchmarks with substantively similar purported concepts to a shared assigned concept, and ask whether model rankings on benchmarks with the same assigned concept correlate more strongly than rankings on benchmarks with different assigned concepts. We ask analogous questions at the item level using item response theory (IRT) models. We find that correlations between model rankings on benchmarks with the same assigned safety concepts are often weak, suggesting these concepts may be conceptualized inconsistently across benchmarks. For assigned capability concepts (e.g., reasoning, knowledge), model rankings are often as strongly correlated among benchmarks with the same assigned concept as between benchmarks with different assigned concepts, suggesting these capability concepts may not discriminate well from one another. In some cases, benchmarks that share design elements (e.g., score format) correlate more strongly than benchmarks with the same assigned concept. Finally, some individual benchmarks correlate more strongly with benchmarks assigned a different concept than with benchmarks sharing their own assigned concept, suggesting they may measure a different concept than they purport to. For example, BBQ-accuracy correlates more strongly with benchmarks labeled reasoning than with benchmarks that share its assigned concept, bias. To support future empirical work on benchmark validity, we release our extensive dataset of model outputs and scores at the item- and benchmark-level.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal). | background 0.62 | 0.04 | 0.04 | asserted 0.76 |
| s1 | We adapt convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, applying it to 56 capability and safety benchmarks across 53 models. | contribution 0.87 | 0.96 | 0.05 | asserted 0.98 |
| s2 | We label benchmarks with substantively similar purported concepts to a shared assigned concept, and ask whether model rankings on benchmarks with the same assigned concept correlate more strongly than rankings on benchmarks with different assigned concepts. | method 0.99 | 0.92 | 0.03 | asserted 0.57 |
| s3 | We ask analogous questions at the item level using item response theory (IRT) models. | method 0.99 | 0.91 | 0.02 | asserted 0.66 |
| s4 | We find that correlations between model rankings on benchmarks with the same assigned safety concepts are often weak, suggesting these concepts may be conceptualized inconsistently across benchmarks. | result 0.89 | 0.97 | 0.27 | hedged possibility 0.72 |
| s5 | For assigned capability concepts (e.g., reasoning, knowledge), model rankings are often as strongly correlated among benchmarks with the same assigned concept as between benchmarks with different assigned concepts, suggesting these capability concepts may not discriminate well from one another. | result 0.83 | 0.95 | 0.22 | hedged possibility 0.74 |
| s6 | In some cases, benchmarks that share design elements (e.g., score format) correlate more strongly than benchmarks with the same assigned concept. | result 1.00 | 0.92 | 0.07 | asserted 0.84 |
| s7 | Finally, some individual benchmarks correlate more strongly with benchmarks assigned a different concept than with benchmarks sharing their own assigned concept, suggesting they may measure a different concept than they purport to. | result 0.90 | 0.93 | 0.17 | hedged possibility 0.89 |
| s8 | For example, BBQ-accuracy correlates more strongly with benchmarks labeled reasoning than with benchmarks that share its assigned concept, bias. | result 0.98 | 0.94 | 0.05 | asserted 0.91 |
| s9 | To support future empirical work on benchmark validity, we release our extensive dataset of model outputs and scores at the item- and benchmark-level. | contribution 0.82 | 0.93 | 0.03 | asserted 0.94 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
Figures
Figures reproduced from arXiv:2609.08812 by the authors.
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.