Pith. sign in

REVIEW 3 major objections 2 minor 26 references

Manifest scaling law slopes in AI leaderboards show low reliability of 0.53 while latent general-factor slopes reach 0.97 stability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 00:13 UTC pith:HBKPP6JM

load-bearing objection Manifest scaling-law slopes show low reliability while latent general-factor slopes look stable, but reported local dependence among benchmarks likely violates CFA assumptions and weakens the main contrast. the 3 major comments →

arxiv 2605.25272 v1 pith:HBKPP6JM submitted 2026-05-24 cs.AI cs.CYstat.AP

AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems

Michael Hardy , Anka Reuel , Lijin Zhang , Jodi M. Casabianca , Sang Truong , Yash Dave , Hansol Lee , Benjamin Domingue
show 1 more author
Sanmi Koyejo
This is my paper
classification cs.AI cs.CYstat.AP
keywords AI benchmarksleaderboard analysisconfirmatory factor analysisgeneralizability theoryscaling lawslatent factorsmeasurement noisemodel evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper applies confirmatory factor analysis and generalizability theory to decompose variance in over 4000 model scores from the Open LLM Leaderboard. It establishes that assumed reporting structures underestimate benchmark relationships and that local dependence among items undermines standard scoring. Contributor metadata accounts for more rank variance than architecture or deployment details. The core contrast is that raw manifest-score slopes for scaling laws prove unreliable while the latent general-factor size slope remains stable under ecosystem controls.

Core claim

Structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; evidence of local dependence among leaderboard items undermines uses of benchmarks as measurement instruments under current scoring systems; contributor metadata explains more rank-relevant variance than architecture or deployment categories; a manifest-score scaling law slope has low reliability while the latent general-factor size slope is highly stable across ecosystem controls.

What carries the argument

Confirmatory Factor Analysis combined with Generalizability Theory to extract a latent general factor and decompose ranking variance sources from benchmark scores.

Load-bearing premise

The confirmatory factor analysis model correctly specifies the factor structure of the benchmark scores and local dependence does not invalidate the generalizability-theory variance decomposition.

What would settle it

Re-running the analysis on the same leaderboard data with an alternative factor model or after explicitly modeling local dependence produces reliability coefficients for the latent slope that differ substantially from 0.97.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Benchmark relationships are stronger than current aggregate reporting assumes.
  • Local dependence among items means benchmarks cannot be treated as independent instruments under present scoring.
  • Contributor metadata explains approximately 9 percent of rank-relevant variance, exceeding other categories.
  • The latent general-factor size slope supplies a stable basis for scaling laws where manifest scores do not.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Benchmark creators could redesign items to reduce local dependence and increase independence of measurements.
  • The same variance-decomposition approach could be applied to other public leaderboards to test whether the stability contrast holds elsewhere.
  • Model developers might prioritize post-training techniques differently once benchmarks are separated into size-driven versus post-training-driven groups.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper applies Confirmatory Factor Analysis (CFA) and Generalizability Theory to scores from over 4,000 models on the Open LLM Leaderboard. It decomposes sources of ranking variance, reports evidence of local dependence among benchmarks, finds that contributor metadata explains ~9% of rank-relevant variance, and contrasts the low reliability of manifest-score scaling-law slopes (R_β=0.53) with the high stability of latent general-factor size slopes (R_g=0.97). It offers diagnostics for trusting benchmark rankings and improving benchmark design.

Significance. If the measurement model is valid, the work supplies a psychometrically rigorous framework for quantifying noise and structure in AI leaderboards, with direct implications for interpreting scaling behavior and for benchmark construction. The reported contrast between manifest and latent reliability, if robust, would be a substantive contribution to how the field evaluates capability claims.

major comments (3)
  1. [Abstract] Abstract: the reported evidence of local dependence among leaderboard items directly contradicts the conditional local-independence assumption required by standard CFA. Unmodeled local dependence biases loadings, inflates general-factor variance, and can artifactually stabilize derived slopes such as R_g=0.97; the manuscript must show how this dependence was accommodated (e.g., via correlated residuals, bifactor structure, or adjusted G-theory partitions) or demonstrate that its magnitude is negligible.
  2. [CFA and G-theory sections] CFA model specification (methods/results sections): the central claim that the latent general-factor size slope remains stable (R_g=0.97) across ecosystem controls rests on the CFA extracting a well-specified general factor. No information is provided on model fit after accounting for the reported local dependence, nor on whether modification indices or residual covariances were examined; without this, the contrast with R_β=0.53 cannot be treated as load-bearing.
  3. [Results on variance decomposition] G-theory variance decomposition (results): the ~9% variance attributed to contributor metadata and the reliability coefficients both rely on the independence structure used to partition sources. If local dependence is substantial and unmodeled, the decomposition is misspecified; the paper should report sensitivity checks that relax this assumption.
minor comments (2)
  1. [Abstract] The abstract states numerical claims (R_β=0.53, R_g=0.97, 9% variance) without accompanying standard errors or sample-size details; these should be supplied with the relevant tables or equations.
  2. [Methods] Notation for the reliability coefficients (R_β, R_g) is introduced without an explicit equation defining how they are computed from the CFA or G-theory components; add a methods equation for reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback, which highlights important methodological considerations for our CFA and G-theory analyses. We agree that local dependence requires explicit accommodation to support the reported contrasts, and we will revise the manuscript to include additional model specifications, fit diagnostics, and sensitivity checks. Our point-by-point responses follow.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the reported evidence of local dependence among leaderboard items directly contradicts the conditional local-independence assumption required by standard CFA. Unmodeled local dependence biases loadings, inflates general-factor variance, and can artifactually stabilize derived slopes such as R_g=0.97; the manuscript must show how this dependence was accommodated (e.g., via correlated residuals, bifactor structure, or adjusted G-theory partitions) or demonstrate that its magnitude is negligible.

    Authors: We acknowledge the violation of local independence in standard CFA. Our original analysis detected local dependence as a substantive finding but applied a standard single-factor CFA without explicit accommodation. In revision we will re-specify the model using correlated residuals for dependent item pairs (identified via modification indices) and a bifactor structure, then recompute R_g under these specifications to confirm stability. We will also report the magnitude of residual covariances to assess negligibility. revision: yes

  2. Referee: [CFA and G-theory sections] CFA model specification (methods/results sections): the central claim that the latent general-factor size slope remains stable (R_g=0.97) across ecosystem controls rests on the CFA extracting a well-specified general factor. No information is provided on model fit after accounting for the reported local dependence, nor on whether modification indices or residual covariances were examined; without this, the contrast with R_β=0.53 cannot be treated as load-bearing.

    Authors: The referee correctly notes the absence of post-dependence model fit information. We will add CFA fit indices (CFI, RMSEA, SRMR), modification indices, and residual covariance matrices in the revised methods and results. The general-factor slopes will be re-estimated after modeling the dependencies, allowing direct evaluation of whether the R_g = 0.97 contrast with R_β remains robust. revision: yes

  3. Referee: [Results on variance decomposition] G-theory variance decomposition (results): the ~9% variance attributed to contributor metadata and the reliability coefficients both rely on the independence structure used to partition sources. If local dependence is substantial and unmodeled, the decomposition is misspecified; the paper should report sensitivity checks that relax this assumption.

    Authors: We agree that unmodeled local dependence can bias G-theory variance components. In revision we will report sensitivity analyses that relax the independence assumption, for example by incorporating residual covariances into the G-study design or using a multivariate G-theory extension. These checks will verify the stability of the contributor-metadata variance share and the reliability coefficients. revision: yes

Circularity Check

0 steps flagged

No circularity: standard CFA and G-theory applied to external leaderboard data with no reductions by construction

full rationale

The paper applies Confirmatory Factor Analysis (CFA) and Generalizability Theory (G-theory) to 4,000+ models from the public Open LLM Leaderboard. The key reported quantities (R_β=0.53 for manifest scaling-law slopes and R_g=0.97 for latent general-factor slopes) are obtained via standard variance decomposition on observed scores and CFA-extracted factors. No equations, self-citations, or ansatzes are shown that would make these reliabilities equivalent to their inputs by construction. The derivation chain is self-contained against external benchmarks and does not invoke author-specific uniqueness theorems or rename fitted parameters as predictions.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Analysis rests on standard assumptions of confirmatory factor analysis and generalizability theory applied to leaderboard scores; no free parameters or invented entities are described in the abstract.

axioms (2)
  • domain assumption Confirmatory factor analysis model accurately captures relationships among benchmark scores
    Central to decomposing ranking variance and identifying local dependence.
  • domain assumption Generalizability theory variance components can be estimated from the Open LLM Leaderboard dataset
    Required to quantify sources of rank-relevant variance including contributor metadata.

pith-pipeline@v0.9.1-grok · 5784 in / 1298 out tokens · 38254 ms · 2026-06-30T00:13:46.732297+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems." pith.science (2026). https://pith.science/paper/HBKPP6JM

@misc{pith2026260525272,
  author       = {Pith},
  title        = {Pith review of: AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HBKPP6JM}},
  note         = {Machine review of arXiv:2605.25272}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance ($\approx9\%$) than architecture or deployment categories in this context; (4) a manifest-score "scaling law" slope has low reliability ($R_{\beta}=0.53$); by contrast, the latent general-factor size slope is highly stable across ecosystem controls ($R_g=0.97$). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.

Figures

Figures reproduced from arXiv: 2605.25272 by Anka Reuel, Benjamin Domingue, Hansol Lee, Jodi M. Casabianca, Lijin Zhang, Michael Hardy, Sang Truong, Sanmi Koyejo, Yash Dave.

Figure 1
Figure 1. Figure 1: AI Latent Landscape Cartography: Study Overview Estimating instead of assuming. A leaderboard ranking is implicitly a measurement-theoretic claim: that a scalar composite of item-level responses reflects one or more latent capabilities of interest, and that variation in the composite score corresponds to variation in the latent capability under measurement. Making this claim explicit requires: 1. Exploring… view at source ↗
Figure 2
Figure 2. Figure 2: Possible Manifest and Latent Scoring Structures used or assumed in AI Benchmark Ecosystem. Left (i, ii, iii): common manifest scores structures. Middle (a, b, c): analogous latent structures, where pairs (a,i), (b,ii), and (c,iii) differ in the causal direction implied by their estimations. Right (d, e, f): other potential latent structures which may be explanatory of observed scores. *Structure for a LMAr… view at source ↗
Figure 3
Figure 3. Figure 3: Noise-controlled Scaling Laws: Log-log of Performance by Number of Parameters. Shaded regions indicate the 95% confidence bands for the linear relationship estimated by robust regression using an M estimator with Tukey’s biweight (Venables & Ripley, 2002). Prior to log transformation of the y-axis, each estimated latent score Θk is converted from its assumed distribution to positive values via the cumulati… view at source ↗
Figure 4
Figure 4. Figure 4: Noise-controlled Scaling Laws by Deployment Type: Log-log of Performance by Number of Parameters. Shaded regions indicate the 95% confidence bands for the linear relationship between x and y estimated by robust regression using an M estimator with Tukey’s biweight (Venables & Ripley, 2002). In this plot, the HF metadata categorizations of “chat model” and “chat template” are combined (cm∨ct) to better high… view at source ↗
Figure 5
Figure 5. Figure 5: Sensitivity of Fit Estimates to Bootstrap sample size, meta-aggregated across both estimation methods [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Estimated mean misfit of relationships between benchmark constructs by latent structure using Langrangian Multiplier test standardized expected parameter changes over bootstraps. Higher SEPC values indicate the model underestimates the inter-benchmark relationships, and lower values indicate overestimation. Labels above each plot indicate which two benchmarks and stars indicate levels of significance: ‘***… view at source ↗
Figure 7
Figure 7. Figure 7: Estimated mean misfit of relationships between benchmark constructs by latent structure using Langrangian Multiplier test standardized expected parameter changes after freeing interitem residual variances over bootstraps. Higher SEPC values indicate the hypothesized latent structure underestimates the inter-benchmark relationships, and lower values indicate overestimation. Labels above each plot indicate w… view at source ↗
Figure 8
Figure 8. Figure 8: Average Effect Size Changes to models based on single-degree Lagrangian Multiplier tests of residual variance. X axis represents the amount of variance explained implied by the score tests; more positive values indicate that the test is capturing more variation. Y axis represents the actual effect sizes; more positive and more negative values mean that the latent structure is underestimating and overestima… view at source ↗
Figure 9
Figure 9. Figure 9: Variance Decomposition diagram for four fully crossed facets of variation for Architecture (A), Benchmark (B), Contributor (C), and Deployment (D). We fit a fully crossed random-intercept/random￾slope LMM: yi = µ + βxi + X S∈S  u S gS (i) + v S gS (i)xi  + εi , (26) where S is the set of random-effect terms (main effects and interactions) included by the measurement design (e.g., A, B, C, D, AB, AC, . . … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 8 canonical work pages · 3 internal anchors

  1. [1]

    doi: 10.1002/ 9781118619179

    ISBN 978-0-471-01171-2 978-1-118-61917-9 978-1-118-61916-2 978-1-118-61903-2. doi: 10.1002/ 9781118619179. Brennan, R. L.Generalizability Theory. Springer, New York, NY , 2001. ISBN 978-1-4419-2938-9 978-1-4757- 3456-0. doi: 10.1007/978-1-4757-3456-0. URL http:// link.springer.com/10.1007/978-1-4757-3456-0. Brown, T. A.Confirmatory factor analysis for app...

    work page doi:10.1007/978-1-4757-3456-0 2001

  2. [2]

    doi: 10.2307/2683631

    ISSN 0003-1305. doi: 10.2307/2683631. URL https://www.jstor.org/stable/2683631. Publisher: [American Statistical Association, Taylor & Francis, Ltd.]. Cai, L. High-dimensional Exploratory Item Factor Analysis by A Metropolis–Hastings Robbins–Monro Algorithm. Psychometrika, 75(1):33–57, March 2010a. ISSN 1860-

    work page doi:10.2307/2683631

  3. [3]

    Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

    doi: 10.1007/s11336-009-9136-x. URL https: //doi.org/10.1007/s11336-009-9136-x. Cai, L. Metropolis-Hastings Robbins-Monro Algorithm for Confirmatory Item Factor Analysis.Journal of Educational and Behavioral Statistics, 35(3):307–335, June 2010b. ISSN 1076-9986. doi: 10.3102/ 1076998609353115. URL https://doi.org/10.3102/ 1076998609353115. Publisher: Amer...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1007/s11336-009-9136-x 2025

  4. [4]

    FotiosFitsilis, MariaKamilaki, BasilisGatos, VassilisKatsouros, andGeorgeMikros

    If your model has no model card or license tag, it is now in the "deleted" category & it won’t appear in the main view. A model with no explanation or license is not useful to the community., January 2024. URL https:// x.com/clefourrier/status/1742586339013337119. Cronbach, L. J. and Meehl, P. E. Construct validity in psychological tests.Psychological Bul...

    work page doi:10.1037/h0040957 2024

  5. [5]

    ISBN 978-1-4757-3990-9 978-1-4419-2323-3. Dunn, K. J. and McCray, G. The Place of the Bifactor Model in Confirmatory Factor Analysis Investigations Into Construct Dimensionality in Language Testing. Frontiers in Psychology, 11, July 2020. ISSN 1664-

    2020

  6. [6]

    URL https: //www.frontiersin.org/journals/psychology/ articles/10.3389/fpsyg.2020.01357/full

    doi: 10.3389/fpsyg.2020.01357. URL https: //www.frontiersin.org/journals/psychology/ articles/10.3389/fpsyg.2020.01357/full. Pub- lisher: Frontiers. Engel, R. F. Chapter 13 Wald, likelihood ratio, and Lagrange multiplier tests in econometrics. InHandbook of Econometrics, volume 2, pp. 775–826. Elsevier, January 1984. doi: 10.1016/S1573-4412(84)02005-5. UR...

    work page doi:10.3389/fpsyg.2020.01357 2020

  7. [7]

    Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact

    URL https://huggingface.co/spaces/ open-llm-leaderboard/open_llm_leaderboard. Groskurth, K., Bluemke, M., and Lechner, C. M. Why we need to abandon fixed cutoffs for goodness-of-fit indices: An extensive simulation and possible solutions.Behavior Research Methods, 56(4):3891–3914, June 2024. ISSN 1554-3528. doi: 10.3758/s13428-023-02193-3. URL https://doi...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.3758/s13428-023-02193-3 2024

  8. [8]

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    ISSN 1548-7660. doi: 10.18637/jss.v048.i02. URL https://doi.org/10.18637/jss.v048.i02. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . WinoGrande: An Adversarial Winograd Schema Chal- lenge at Scale, November 2019. URL http://arxiv. org/abs/1907.10641. arXiv:1907.10641 [cs]. Salaudeen, O., Reuel, A., Ahmed, A., Bedi, S., Robertson, Z., Sundar, ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18637/jss.v048.i02 2019

  9. [9]

    doi:10.1016/j.neucom

    ISSN 0925-2312. doi: 10.1016/j.neucom. 2025.132542. URL https://www.sciencedirect. com/science/article/pii/S092523122503214X. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y ., Chung, H. W., Chowdhery, A., Le, Q. V ., Chi, E. H., Zhou, D., and Wei, J. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them, Octo- ber 2022. URL ht...

    work page doi:10.1016/j.neucom 2025

  10. [10]

    Local asymptotic optimality follows from standard LAN arguments (Buse, 1973). Statistical conditions (typical SEM/ML assumptions).A sufficient set of conditions for Theorem D.1 (and its robust analogs) includes: 1.Identification:the constrained model is locally identified at the true parameter;I θθ is nonsingular

    1973

  11. [11]

    Interior point: ψ= 0 lies in the interior of the parameter space for the relevant parametrization (or use boundary- corrected tests if not)

  12. [12]

    Regularity/smoothness: ℓ is twice continuously differentiable in a neighborhood; interchange of differentiation and integration is valid

  13. [13]

    item redundancy

    Asymptotics: n→ ∞ with p fixed (or p increasing slowly under additional conditions);S is consistent for the population covariance ofy ⋆. RemarkD.2 (Finite-sample performance).Simulation studies show that MI statistics maintain approximately correct Type I error rates for n≥200 and moderate model complexity (p≤50 ), with scaled corrections essential for ca...

    2009

  14. [14]

    Randomly sample a small, computationally manageable set of items from each of the 6 benchmarks

  15. [15]

    Fit all six structural models to the tetrachoric correlation matrix of this item subset

  16. [16]

    This process yields distributions of fit indices and parameters, allowing us to perform a meta-analysis

    Store key model fit indices (e.g., scaled RMSEA, CFI), parameter estimates (Λ,Ψ ), and local-fit diagnostics (modification indices). This process yields distributions of fit indices and parameters, allowing us to perform a meta-analysis. We assess the central tendency and stability of each model’s performance, effectively integrating out the noise from an...

  17. [17]

    A bootstrap that happens to sample highly discriminating items produces a more precise estimate ofβ d than one dominated by low-discrimination items

    Heterogeneous precision.Different item draws yield different effective test lengths, item discrimination profiles, and conditioning on the latent trait. A bootstrap that happens to sample highly discriminating items produces a more precise estimate ofβ d than one dominated by low-discrimination items. Treating both equally wastes information

  18. [18]

    Naïve averaging allows these unstable estimates to contaminate the aggregate

    Occasional near-degeneracy.Some item subsets may produce near-singular Fisher information for particular parameters (e.g., a specific factor receiving very few high-discrimination items), inflating SE(b) by orders of magnitude. Naïve averaging allows these unstable estimates to contaminate the aggregate. Inverse-variance weighting addresses both issues: i...

  19. [19]

    true” variance to expected observed variance: G= σ2 true σ2true +σ 2error (24) The definitions of “true

    Fit.Estimate the target model (CFA structures for Method 1 via both MH-RM and DWLS; bifactor IRT with/without latent regression for Method 3 via MH-RM), obtaining estimates ˆξ(b) and standard errorsSE (b). 3.Weight and Aggregate via Mixed Effect Meta-regression (Method 1).Obtain statistical estimates based on (14). 4.Weight (Method 3).Computew (b) = 1/(SE...

    2015

  20. [20]

    It measures the strength of the average scaling trend (β) relative to its instability across the ecosystem’s facets

    Slope Signal-to-Noise Ratio (SNR β).This novel metric directly quantifies the reliability of the fixed-effect scaling law. It measures the strength of the average scaling trend (β) relative to its instability across the ecosystem’s facets. Definition G.2(Slope Signal-to-Noise Ratio).The SNR of the fixed-effect slope β is the ratio of the squared magnitude...

  21. [21]

    do models scale?

    Proportion of Slope Instability (PSI S).While SNR β quantifies thetotalinstability of the scaling law, PSI S diagnoses its sources by partitioning the total random slope variance. For each facet or interaction termS, we define: PSIS = σ2 S,1P S′ σ2 S′,1 .(31) PSIS is the proportion of the total instability in the scaling relationship that is attributable ...

  22. [22]

    scaling laws

    ˆβwith model-based and robust SE, andR β. 2.Ω x andΩ (−B) x (size-driven rank-order potential overall and benchmark-independent). 3.σ 2 β,eff andCV β (how much scaling varies across the ecosystem). These metrics separate three phenomena that are conflated in typical leaderboard analyses: (i) whether scale predicts performance on average, (ii) whether scal...

    2015

  23. [23]

    Model comparability:likelihood-based comparison between (i) and (ii) to verify that adding covariates/random effects improves fit without distorting the bifactor measurement structure

  24. [24]

    Noise-controlled scaling slopes:estimates { ˆβd}K d=0 and their uncertainty, aggregated across bootstraps via inverse- variance weighting

  25. [25]

    provenance variance

    Provenance variance: Σu as the contribution of contributor/provenance to latent variability after controlling for size and deployment. 4.Stability diagnostics:bootstrap distributions of ˆβd and (when applicable) ˆRd. H.1.4. RELIABILITY OF SCALING EFFECTS UNDER PSEUDO-REPLICATION The object of interest is the scaling law slope on a latent dimensiond∈ {0, ....

  26. [26]

    size is nearly all you need

    The residual variance drops by 36% (from 52.0 to 33.24 in the full model summary), indicating that model size explains a substantial portion of previously unexplained performance differences—without this control, architectural signal is confounded with scale. We calculate (33) to be Ωsize-all = 0.402, suggesting that about 40% of all observed variation is...