{"id":"6c0b1e2e-ba76-4579-958e-8a3ed934c7a4","arxiv_id":"2609.08812","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An adapted convergent and discriminant validity analysis of 56 AI benchmarks reveals weak convergence among safety benchmarks, poor discrimination among capability concepts, and dominance of benchmark design over purported concept.","lead":"Researchers adapted convergent and discriminant validity from social sciences to test whether 56 AI benchmarks actually measure what they claim, using 53 models. They found that many safety benchmarks do not agree, capability categories are not distinct, and benchmark design choices often matter more than the concept being tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Without residualizing the dominant general ability factor, the finding that capability concepts are not discriminable (Fig. 2, §4.2) may reflect a single dimension of model performance rather than genuine non-discrimination.","rationale":"The reader identified the subjectivity of assigned concepts as the weakest assumption. While important, that concern applies broadly across all findings and is partially mitigated by using stated author intentions and providing full transparency (Appendix C). A more precise and technically load-bearing threat is the unaddressed general ability confound for the discrimination finding (§4.2). The paper explicitly flags this limitation in Section 5 but does not correct for it, yet the finding that capability concepts are not discriminable is a headline result. If residualization on the first principal component eliminates the pattern, the central claim about capability concept discrimination would be undermined. This is a concrete, fixable gap that the authors can address. Other findings (weak convergence for safety benchmarks, role of score format, individual benchmark relabeling) are less vulnerable to this confound because they either show weak correlations (safety) or compare across categories that include weak-correlation benchmarks. Therefore, I recommend conditional acceptance pending this robustness check.","tokens_in":33054,"tokens_out":4382,"duration_ms":42174,"concrete_test":"Compute the first principal component (PC1) of the 48-benchmark correlation matrix. For each benchmark, compute its residual after regressing PC1 out of benchmark scores. Recompute the mean Spearman correlation within capability concepts and between different capability concepts using these residualized rankings (the within-capability mean minus between-capability mean). If the corrected gap is positive and statistically significant, the original finding that capability concepts do not discriminate is an artifact of the general ability factor. If the gap remains near zero or negative, the finding is robust. This test is directly implementable from the publicly released dataset and would definitively settle whether the second main finding is a confound artifact.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's second main finding (Fig. 2, §4.2) shows that mean correlations between model rankings on benchmarks with different assigned capability concepts are as high as correlations between benchmarks sharing the same assigned capability concept. This is interpreted as evidence that capability concepts are not discriminable. However, a large body of prior work (e.g., Burnell et al. 2023a, cited in the paper) has shown that a strong general ability factor dominates most capability benchmarks, causing all pairwise correlations to be high regardless of the specific concept being measured. The paper explicitly acknowledges this in Section 5: 'Our discriminant analyses also do not account for the single dominant dimension of general model ability that may drive performance across many benchmarks, regardless of the concepts being measured.' Despite this acknowledgment, the paper presents the non-discrimination of capability concepts as a primary result without any correction for this confound. Without residualizing the first principal component or otherwise removing the general ability factor, the within-capability vs. between-capability correlation gap could be zero or negative even if the concepts are genuinely discriminable, simply because all measurements are dominated by a single latent factor. This threat is specific, structural, and acknowledged by the authors themselves, yet left unaddressed.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper adapts convergent and discriminant validity from the social sciences to interrogate 56 AI benchmarks across 53 models. The authors label benchmarks with shared assigned concepts, then examine whether model rankings on benchmarks sharing an assigned concept correlate more strongly than rankings on benchmarks with different assigned concepts, and whether individual benchmarks correlate more strongly with benchmarks from a different concept than their own. Four main findings emerge: (1) safety benchmarks show weak convergence within concepts; (2) capability benchmarks do not discriminate between concepts; (3) shared design elements (e.g., score format) predict correlations more strongly than shared concepts; (4) some individual benchmarks (e.g., BBQ-accuracy) correlate more with a different concept than their own assigned concept. The paper releases an extensive dataset of model outputs and scores.","tokens_in":33407,"tokens_out":2848,"duration_ms":31655,"significance":"If the results hold, the paper provides a systematic empirical framework—convergent and discriminant validity—that could become a standard tool for diagnosing construct validity in AI benchmarks. The released dataset is a substantial community resource. The finding that many safety benchmarks do not converge and that capability benchmarks are dominated by a general ability factor (or fail to discriminate) has direct implications for model governance, ranking interpretation, and benchmark development. The paper is novel in applying MTMM-inspired methods at this scale and in combining benchmark-level correlations with item-level IRT analyses.","major_comments":[{"comment":"The second main finding (capability concepts are not discriminable) is presented without controlling for the dominant general ability factor known to drive performance across most capability benchmarks. The authors acknowledge this limitation in §5 ('Our discriminant analyses also do not account for the single dominant dimension of general model ability...') but still state the finding as evidence that 'these capability concepts may not discriminate well from one another.' Because a strong latent general factor would produce exactly the observed pattern (high within- and between-concept correlations) even if concepts are genuinely discriminable after residualization, the current analysis cannot distinguish non-discrimination from a common-factor confound. This threatens the load-bearing interpretation of Finding 2. I request either (a) residualizing each benchmark on the first principal component of the benchmark correlation matrix and re-running the within- versus between-concept comparison, or (b) explicitly downgrading the claim from a finding about discriminant validity to a descriptive report of the correlation structure and adding a central discussion of why the general-factor interpretation is equally compatible.","section":"§4.2, Fig. 2, and §5 (acknowledgment)"},{"comment":"The assignment of benchmarks to concepts (‘assigned concepts’) is based on the authors’ interpretation of each benchmark’s introductory paper. This grouping is structurally load-bearing for all four main findings. While acknowledged as subjective (§5), the paper provides no sensitivity analysis to assess how results change under alternative or more conservative groupings. For example, merging closely related concepts (e.g., reasoning and comprehension) or splitting broad concepts (e.g., bias into gender bias vs. racial bias) could alter convergence and discrimination patterns. I request a sensitivity analysis that varies the grouping threshold (e.g., leave-one-benchmark-out per concept, or a coarser concept assignment) and reports the range of the main statistics.","section":"§3.1, Tables C.1"}],"minor_comments":[{"comment":"The phrase 'We label benchmarks with substantively similar purported concepts to a shared assigned concept' is grammatically awkward. Consider rephrasing to 'We assign benchmarks with substantively similar purported concepts to a shared label (assigned concept).'","section":"Abstract"},{"comment":"The y-axis label ('Pairwise Spearman correlation') is partially cut off in the manuscript PDF. Please ensure the full label is visible in the camera-ready version.","section":"Figure 1"},{"comment":"In several rows, the ✓ symbols under the 'Passes saturation analysis' and 'Passes format-compliance analysis' columns are not vertically aligned. Consider using a fixed-width column or centering symbols for readability.","section":"Table A.2"},{"comment":"The discussion of Burnell et al. (2023a) mentions that BBQ loads on a reasoning factor. This directly foreshadows the §4.4 result for BBQ-accuracy. Consider incorporating this connection earlier to strengthen the narrative.","section":"§2"},{"comment":"You state that temperature was set to 1.0 for all models. Some recent models (e.g., o-series) only support a default temperature of 1 and do not allow other values. Please note this explicitly and confirm that temperature was set to the default for those models rather than forcing 1.0.","section":"Appendix D.2"},{"comment":"The sentence 'Given the critical role benchmarks play in benchmarks play in development' contains a duplicated phrase. Please correct.","section":"§5, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is carefully done and the dataset release is commendable. The main issue is the unaddressed general ability confound for the capability discriminant finding. If the authors can provide a robustness check (e.g., residualization by first principal component) or convincingly argue why the finding is still meaningful without it, the paper would be substantially stronger. I lean toward major revision rather than rejection because the problem is fixable and the rest of the paper is solid. The subjective concept assignment also merits a sensitivity analysis, but this is secondary. Overall, the approach is novel and the results are provocative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper does something legitimately new: it systematically adapts convergent and discriminant validity from social science and applies it to 56 AI benchmarks across 53 models. The scale alone is worth noting. The dataset release is a real contribution, and the methodology is generally careful (cluster bootstrap, IRT models, partial Mantel tests, robustness checks for model era and judge choice).\n\nThe paper's strongest result is the format-effect finding: LLM-judge scoring predicts benchmark similarity more than shared assigned concept does. That's practically important and methodologically clean. The safety convergence finding (weak correlations among safety benchmarks) is also well-supported and thoughtfully hedged—the authors don't overinterpret it as poor construction. The individual benchmark relabeling tests (BBQ-accuracy, DecodingTrust-Fair) are nice illustrations of the framework.\n\nThe main soft spot is the stress-test concern about the second finding: capability concepts appear non-discriminable because within-concept and between-concept correlations are equally high. The authors acknowledge in Section 5 that the analysis does not account for the single dominant dimension of general model ability. That's the right thing to say, but it undercuts a central claim of the paper. Without residualizing the first principal component, the finding could simply reflect that all capability benchmarks are driven by general ability, rather than that the concepts are genuinely not discriminable. The paper presents this as a primary result, which it is, but the confound is structural and acknowledged. I'd want to see the analysis done with that correction before taking the capability-discrimination claim at face value.\n\nThe concept assignment is also subjective, though the paper is transparent about it and runs robustness checks that help. I don't think that alone is fatal.\n\nOverall: this is a serious, honest paper with real empirical contributions. The format-effect and safety-convergence findings hold up well. The capability-discrimination finding needs stronger treatment. I'd send it to review with the expectation that the authors address the general ability confound, perhaps by residualizing on the first principal component as they suggest. The core methodology and data are too valuable to desk-reject.\n\nFor a reader: this is useful for anyone working on AI evaluation or governance. The dataset alone makes it worth citing.","headline":"Solid, large-scale empirical work on benchmark validity, but the second finding about capability concepts is undercut by an acknowledged but unaddressed confound (general ability factor).","tokens_in":33929,"tokens_out":1067,"would_cite":true,"duration_ms":13640,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Applied convergent and discriminant validity from the social sciences to systematically interrogate 56 AI benchmarks, finding that most do not reliably measure the concepts they purport to measure—safety benchmarks show weak convergence…","keywords":["AI benchmarks","construct validity","convergent validity","discriminant validity","item response theory","model evaluation","safety benchmarks","capability benchmarks"],"falsifier":"If a new set of safety benchmarks were designed with explicit, detailed concept definitions and showed high within-concept rank correlations (ρ > 0.7) and low cross-concept correlations, the claim that safety benchmarks fail to converge would be refuted for that concept. Similarly, if BBQ could be refactored into items that isolate bias from reasoning (e.g., using matched disambiguated pairs) and then showed higher correlation with bias benchmarks than with reasoning benchmarks, the relabeling claim would be weakened.","tokens_in":34688,"feed_emoji":"🔬","tokens_out":2990,"duration_ms":29545,"temperature":0.7,"pith_summary":"The paper adapts convergent and discriminant validity—two established lenses from social-science measurement theory—into a systematic approach for interrogating whether AI benchmarks actually measure the concepts they claim to. Applying this approach to 56 capability and safety benchmarks evaluated across 53 models yields four principal findings. First, model rankings on safety benchmarks sharing the same assigned concept (e.g., bias, refusal) correlate weakly, suggesting inconsistent conceptualization. Second, for capability concepts (e.g., reasoning, knowledge), correlations among benchmarks with different concepts are as high as correlations among benchmarks with the same concept, meaning these concepts are empirically indistinguishable. Third, benchmarks that share design elements—particularly score format—correlate more strongly than benchmarks that share a concept. Fourth, individual benchmarks such as BBQ-accuracy correlate more strongly with benchmarks from a different concept (reasoning) than with their own concept (bias), indicating they may measure something other than what they purport. The authors release a large dataset of item-level model scores to support further validity research.","feed_headline":"Most AI benchmarks fail to measure what they claim","feed_subtitle":"A systematic study of 56 benchmarks finds safety ones don't converge, capability ones don't discriminate, and design artifacts dominate.","key_machinery":"The central machinery is the adaptation of convergent and discriminant validity from the Campbell–Fiske multi-trait multi-method matrix, operationalized through Spearman rank correlations of model-level scores across benchmarks and through item response theory (1PL) models that test whether a pair of benchmarks is better described by one shared latent trait or by two separate traits (ΔAUC). The grouping of benchmarks into 'assigned concepts' is based on the concept each benchmark's introductory paper claims to measure. Benchmark-level correlations are aggregated by assigned-concept pairs with cluster-bootstrapped confidence intervals; item-level comparisons use ΔAUC with a hold-out set and bootstrap. A relabeling permutation test quantifies whether an individual benchmark's correlation vector is closer to a hypothesized alternative concept than to its own assigned concept.","core_discovery":"Using adapted convergent and discriminant validity (inspired by the multi-trait multi-method matrix), the paper demonstrates that AI benchmarks suffer from systematic construct-validity failures. For safety concepts, within-concept correlations are weak (median ρ near zero for refusal, bias, safety detection). For capability concepts, within-concept correlations are barely higher than cross-concept correlations (e.g., reasoning vs. knowledge: mean ρ ≈ 0.55 both ways), so the benchmarks do not distinguish the concepts they name. Design artifacts dominate: a partial Mantel test shows that shared score format (especially LLM-judge scoring) predicts pairwise benchmark correlations more strongly than shared assigned concept (β_format = 0.526, β_concept = −0.058). Individual benchmark interrogation via a relabeling permutation test confirms that BBQ-accuracy correlates significantly more with reasoning benchmarks than with bias benchmarks (relabeling statistic = 0.15, p < 0.001), and DecodingTrust-Fair correlates more with knowledge than with bias (0.14, p = 0.002). These patterns are robust to model generation split and choice of LLM judge.","pith_inferences":["The finding that capability benchmarks are empirically indistinguishable suggests that many AI leaderboards may rank models primarily by a single general ability dimension, not by the specific competencies the evaluators intend to highlight. This would undermine claims of 'state-of-the-art reasoning' or 'superior knowledge' as separate differentiators.","The method pioneered here could be extended to multimodal benchmarks (vision, speech) and to human evaluation setups, where concepts like 'helpfulness' or 'harmlessness' are even more underspecified. The convergent–discriminant framework provides a principled way to test whether different evaluation protocols measure the same thing.","If design artifacts dominate conceptual content, then benchmark designers should document not only the target concept but also the method (format, prompt, scoring function) as an integral part of the measurement. Releasing only the concept label is insufficient for reproducibility or validity.","The paper's reliance on assigned concepts (grouped by reading benchmark papers) is a necessary but fragile step. A more rigorous alternative would be a formal content analysis or expert panel to define concept boundaries—though the paper's findings already suggest that even generous grouping fails to produce convergence."],"forward_implications":["Benchmark developers should routinely assess convergent and discriminant validity during design, using correlation matrices and IRT-based checks, to detect misaligned constructs before release.","Safety evaluation suites should reconsider reliance on any single bias benchmark like BBQ: if BBQ-accuracy measures reasoning rather than bias, using it alone (as many commercial model cards do) misses the intended construct.","Capability benchmarks claiming to measure distinct concepts (reasoning, knowledge, comprehension) likely tap a general ability factor; researchers should either acknowledge this or design evaluations that deliberately suppress shared variance.","Score format (especially LLM-judge scoring) introduces a strong method effect that can masquerade as conceptual agreement; controlling for format in multi-benchmark studies is essential for valid interpretation.","The released item-level dataset enables other researchers to replicate and extend validity analyses without re-running large-scale model evaluations, lowering the barrier to construct-validity checks in AI.","The paper confirms that IRT models, typically used for item selection, can serve a diagnostic role: a positive ΔAUC flags pairs of benchmarks that require separate latent traits, signaling poor conceptual overlap."],"supporting_citations":[{"why":"Supplies the multi-trait multi-method matrix framework for convergent and discriminant validity that the paper adapts.","marker":"Campbell & Fiske, 1959"},{"why":"Provides the social-science standard on measurement validity that grounds the adaptation.","marker":"Adcock & Collier, 2001"},{"why":"Argues that evaluating AI systems is a social science measurement problem, motivating the use of construct validity.","marker":"Wallach et al., 2025"},{"why":"Factor analysis showing BBQ loads on a reasoning factor rather than a bias factor, directly supporting the relabeling hypothesis.","marker":"Burnell et al., 2023a"},{"why":"Finds that safety-benchmark scores correlate strongly with general capability scores, consistent with the paper's safety-to-capability pull result.","marker":"Ren et al., 2024"},{"why":"HELM framework provides the initial benchmark list and comparison data used for pipeline validation.","marker":"Liang et al., 2022"}],"fun_headline_variants":["AI benchmarks fail construct validity: safety weak, capability blurry","Benchmarks measure design artifacts, not the concepts they name","Fifty-six benchmarks: safety doesn't converge, capability doesn't discriminate","Construct validity absent: AI benchmarks correlate by format, not concept","BBQ-accuracy measures reasoning, not bias — benchmark validity fails"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper groups benchmarks into assigned concepts based on what each benchmark's introductory paper says it measures, but those concepts are often underspecified or ambiguous; if the grouping is loose or inaccurate, the convergence and discrimination analyses may reflect grouping artifacts rather than true measurement properties.","fun_headline_variants_meta":{"raw":{"variants":["AI benchmarks fail construct validity: safety weak, capability blurry","Benchmarks measure design artifacts, not the concepts they name","Fifty-six benchmarks: safety doesn't converge, capability doesn't discriminate","Construct validity absent: AI benchmarks correlate by format, not concept","BBQ-accuracy measures reasoning, not bias — benchmark validity fails"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"cost":0.000263228,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000263228,"upstream_inference_prompt_cost":0.000121828,"upstream_inference_completions_cost":0.0001414},"total_tokens":1580,"prompt_tokens":1075,"completion_tokens":505,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":416}},"tokens_in":1075,"tokens_out":505,"cost_source":"openrouter_usage","duration_ms":5520,"temperature":1.0,"reasoning_tokens":416,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T08:28:39.988435+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"falsifier":"If a new set of safety benchmarks were designed with explicit, detailed concept definitions and showed high within-concept rank correlations (ρ > 0.7) and low cross-concept correlations, the claim that safety benchmarks fail to converge would be refuted for that concept. Similarly, if BBQ could be refactored into items that isolate bias from reasoning (e.g., using matched disambiguated pairs) and then showed higher correlation with bias benchmarks than with reasoning benchmarks, the relabeling claim would be weakened.","supporting_citations":[{"cited_title":"The Effect of Sampling Temperature on Problem Solving in Large Language Models","cited_arxiv_id":null,"evidence_quote":"Finds that safety-benchmark scores correlate strongly with general capability scores, consistent with the paper's safety-to-capability pull result."}],"review_version":1}