Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Improving LLM Leaderboards with Psychometrical Methodology

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Applying psychometric factor analysis to LLM leaderboards yields a more robust ranking than the raw average, and reveals that the average inflates weak models and deflates strong ones.

desk verdict A competent psychometric proof-of-concept whose headline ranking improvement is largely an artifact of the logit link; the real value is in the fit diagnostics and the call for better construct definitions. read the letter →

arxiv 2501.17200 v1 pith:2KRP3FZ4 submitted 2025-01-27 cs.CL cs.AIstat.AP

Denis Federiakin This is my paper · ORCID
classification cs.CLcs.AIstat.AP
keywords psychometricsfactoranalysisLLMleaderboardscontinuousresponsemodelbenchmarkevaluationitemtheoryHuggingFaceLeaderboardscores
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM leaderboards, which rank models by the simple average of benchmark scores, can be improved by treating benchmarks as test parcels and applying a unidimensional factor analysis borrowed from human psychometrics. Using data from two versions of the open LLM leaderboard, the author shows that a single latent ability factor accounts for most common variance across six very different benchmarks, and that factor scores derived from the model change the ordering of mid-tier models and carry uncertainty estimates the naive average hides. The headlined finding is that the raw average overestimates the weakest models and underestimates the strongest ones relative to the factor score, an inverted U-shaped relation the paper interprets as evidence that leaderboard averages mask diminishing returns in LLM scaling. A sympathetic reader would care because this is a concrete, low-cost fix: any leaderboard publishing per-benchmark averages already publishes the data needed to estimate factor scores.

What carries the argument

The central mechanism is the continuous response model (CRM) applied to benchmark parcels: each benchmark's average score $U_p$ is logit-transformed to $V_p = \mathrm{logit}(U_p)$, and a confirmatory factor model $V = \boldsymbol{\mu} + \boldsymbol{\Lambda}\theta + \boldsymbol{\varepsilon}$ is fitted with a single latent ability $\theta \sim \mathcal{N}(0,1)$. Because a logit link connects the observed bounded scores to the latent variable, this linear factor model approximates Samejima's continuous response model as derived by Wang and Zeng and formalized by Ferrando; the MAP estimator of $\theta$ yields factor scores on an unbounded interval scale with a closed-form posterior standard error. The model also produces residual covariances that reveal content-based dependencies among benchmarks beyond the general factor.

What would settle it

Estimate the same unidimensional factor model separately for subsets of models grouped by architecture family (e.g., dense decoders, mixture-of-experts, and vision-language models) and test for measurement invariance; if the factor loadings differ substantially across groups, the single factor score is not a common scale and the ranking correction is not valid for cross-architecture comparisons. Alternatively, fit the model at the task level rather than the benchmark-parcel level and check whether the inverted-U relation between average and factor score survives.

Watch

Extended reading notes

Core claim

The central claim is that a unidimensional confirmatory factor analysis applied to the logit-transformed per-benchmark averages (parcels) of the Hugging Face Leaderboard yields factor scores that rank LLMs more robustly and more meaningfully than the naive benchmark average, and that the relation between the two is an inverted U: the naive average slightly overestimates the weakest-performing models and slightly underestimates the strongest-performing models when compared with the factor score. The author demonstrates this on Leaderboard v.1 (3,792 models, six benchmarks) and v.2 (1,543 models, six benchmarks, with raw and anti-guessing-normalized scores), showing that a single latent factor explains substantial common variance in both versions, that residual correlations interpretable as content overlap remain after the factor is extracted, and that many adjacent models are not statistically significantly different in factor-score terms even though their benchmark averages differ. The paper explicitly notes that the v.1 model fits poorly and fails to converge under modification, and that measurement invariance across model architectures is an untested assumption.

Load-bearing premise

The load-bearing premise is that a single latent ability factor, with the same structure for every model architecture, really does underlie all six benchmark averages, so that the common variance the factor extracts is the meaningful signal and the residual variance is noise.

Editorial extensions

If this is right

  • If factor scores replace the average, leaderboard rankings of mid-tier models can shift substantially even though top models keep their order.
  • The inverted U relation implies that relying on the average systematically flatters the weakest models and undersells the strongest, distorting conclusions about scaling laws.
  • Reporting posterior standard errors alongside factor scores prevents over-interpretation of differences that are not statistically significant.
  • Because the model only needs per-benchmark averages, any leaderboard can adopt the methodology without re-running benchmarks at item level.
  • Using task-level parcels and anchor tasks common to successive leaderboard versions could make ability estimates comparable over time, tracking LLM progress despite changing benchmark content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own Study 1 uses factor scores from a model with poor fit; a natural extension is to test whether the inverted-U pattern persists when the model is estimated on task-level parcels or with multidimensional factors, or when measurement invariance across architecture families is enforced.
  • The residual covariances the paper interprets as content overlap (e.g., math benchmark correlating with MMLU, HellaSwag correlating negatively with TruthfulQA) suggest a cheap way to audit benchmarks: benchmarks that show large residual correlations with others after the general factor is removed are not pure measures of the general ability and should be interpreted with that caveat.
  • The analogy to variational autoencoders and collaborative filtering implies the factor-score approach could be extended to predict missing benchmark scores for partially evaluated models, effectively imputing leaderboard entries rather than dropping models with incomplete data.
  • The paper's discussion points toward a longitudinal 'AIQ' scale anchored by stable tasks; a testable intermediate step is to check whether Big Bench task-level parameters are indeed invariant across Leaderboard v.1 and v.2, as the proposed anchor design assumes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that the average score used by LLM leaderboards is a naive aggregation, and that applying a confirmatory factor-analytic psychometric model to the six benchmark averages reported by the Hugging Face Leaderboard yields factor scores that produce a more robust and meaningful ranking. In Studies 1 and 2 the authors fit unidimensional factor models (after logit-transforming the benchmark parcels) to two versions of the leaderboard, report poor-to-acceptable fit indices, and then greedily add residual covariances to improve fit. In Study 3 they compare the resulting MAP factor scores with the reported benchmark averages and interpret a mild inverted U-shaped relationship as evidence that the raw average overestimates weak models and underestimates strong ones. The paper also compares factor scores with model parameter counts and training emissions, and discusses the philosophical and methodological differences between benchmark development and human test development.

Significance. If the central claim were established, the paper would provide a useful psychometric perspective on LLM leaderboard construction and a principled alternative to simple averaging. The paper is clearly written in its exposition of factor-analytic machinery, and it is commendably transparent about several of its own limitations, including the poor fit of the Study 1 model and the untested measurement invariance assumption. However, the study's central claim is not currently supported by the evidence presented: the comparison in Study 3 does not distinguish factor analysis from a monotone transformation of the input scores, and no external validation or baseline aggregation is provided. The paper's methodological contribution—introducing a psychometric toolkit for benchmark parcels—is real, but its specific empirical assertion of an improved ranking requires additional analyses.

major comments (4)
  1. [Study 3, Eq. (13), Figures 1, 3, 4] The central claim that factor scores improve rankings is not supported by the reported comparisons, because the factor score is a deterministic affine function of a weighted average of logit-transformed benchmark scores. In the unidimensional case, Eq. (13) reduces to theta_hat_m = c * sum_i (lambda_i / psi_i) * (logit(U_im) - mu_i) with c > 0, so the factor-score ranking is identical to the ranking on a weighted average of logits. The raw leaderboard average is an unweighted average of raw accuracies. Since logit stretches small accuracies and compresses high ones, the scatterplot of factor scores against the raw average will generically display the reported inverted U-shaped pattern regardless of whether the factor model is correctly specified. To support the claimed advantage, the paper needs a baseline comparison (e.g., an equal-weight logit average, z-scored averages, or a principal-component score) and/or an external criterion such as predictive validity on held-out benchmarks or agreement with human judgments. Without such a comparison, Study 3 merely shows that a nonlinear transform changes the scale, not that it improves ranking.
  2. [Study 1, Section 5.1 and Study 3, Figure 1] The factor scores used for the Study 1 comparison are estimated from a model that the paper itself reports as poorly fitting (robust RMSEA = 0.304, TLI = 0.836) and whose attempted modification led to non-convergence, as stated in Section 5.3. The paper explicitly acknowledges this limitation and pivots to Leaderboard v.2, but Figure 1 and the associated discussion of the inverted U-shaped relationship are still presented as meaningful evidence. If the v.1 results are to be retained, the authors should either report them as exploratory only, or show that the key pattern is robust to alternative methods of score estimation, for example by using a model with acceptable fit or a non-parametric dimensionality reduction that does not impose the specific logit link.
  3. [Sections 5.1, 5.2.1, 5.2.2] The greedy addition of residual covariances, driven by modification indices until an acceptable fit is reached, is a data-driven post-hoc fitting procedure with only six observed parcels. The procedure has no out-of-sample validation, so the reported 'final' models are likely to capitalize on idiosyncratic correlations in the specific leaderboard snapshots. This matters because the residual correlations are then given substantive interpretations (e.g., HellaSwag ~ TruthfulQA negative correlation) and are used to support the general-factor conclusion. The authors should demonstrate stability of the residual covariance structure, for example through cross-validation, split-half analysis, or an alternative estimation (e.g., EFA followed by target rotation), and should temper the substantive claims about residual dependencies until such stability is shown.
  4. [Discussion and Limitations, Section 7] The paper identifies measurement invariance as a key untested assumption and acknowledges that different model architectures may have different 'world models', yet Figure 9 and the related architecture-by-architecture comparisons interpret differences in factor-score variance across architectures as if the factor model were invariant. If factor loadings or intercepts vary by architecture, the estimated factor scores are not commensurable across groups, and the architecture-level comparisons in Figure 9 are not interpretable in the claimed way. At a minimum, the authors should state this as a formal limitation of the architecture comparisons and, ideally, provide a preliminary invariance test (e.g., multigroup CFA by architecture family, or at least a factor-loading comparison for the largest families).
minor comments (5)
  1. [Section 2, pages 4-6] There is a duplicated paragraph discussing social constructivism and representativism (the sentences beginning 'This naturally raises the issue of social constructivism' and following appear twice), which should be removed for clarity.
  2. [Section 4.1] The subsection heading reads 'Analysis methodolody'; this should be corrected to 'methodology'.
  3. [Tables 1 and 2] The benchmark name 'Winograde' should be 'WinoGrande' for consistency with Section 4.2.
  4. [References] Several references are duplicated in the list: Marsman et al. (2018), Urban and Bauer (2021), and Yuan and Bentler (2000) each appear twice and should be consolidated.
  5. [Section 5.2.2] The word 'hypothetize' should be 'hypothesize'.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline inverted-U finding is a mathematical consequence of the logit link in Eq. (13), not an external validation; Study 3 compares two transformations of the same benchmark parcels.

  1. fitted input called prediction [Section 4.1, Eq. (13); Section 5.3 (Study 3)]
    "𝛉x!=(𝚲(𝚿#$𝚲+𝚵#$)#$𝚲(𝚿#$(𝑽!−𝝁)=𝚵𝚲((𝚲𝚵𝚲(+𝚿)#$(𝑽!−𝝁). (13) ... The comparison between the native LLM ranking from the Leaderboard (Benchmark Average) and the ranking of LLMs derived from the FA model reveals a slight inverted U-shaped dependency."

    By Eq. (1), V = logit(U); by Eq. (5) the CFA is fitted to V; Eq. (13) then gives the factor score as a fixed affine function of the logit-transformed parcel scores. The leaderboard average is the unweighted average of the raw U scores. Therefore the Study 3 scatterplot compares two deterministic functions of the same six parcels. The reported pattern (raw average overestimates weak models and underestimates strong ones relative to factor scores) is not an independent empirical discovery about LLM ability; it is determined by the model's choice of the logit link and by the fact that the factor score is estimated from the same parcels.

full rationale

The paper is transparent about its internal analyses, and there is no problematic self-citation chain: the author does not rely on prior work by Federiakin to justify the central premise. The unidimensional CFA, factor loadings, reliability coefficients, and residual covariances are empirical quantities estimated from the leaderboard data, so those parts are not circular. However, the central claim that factor scores provide a more robust and meaningful ranking is demonstrated almost entirely through Study 3, which compares factor scores with the benchmark average. That comparison is between two functions of the same input parcels: the factor score is, via Eq. (13), an affine function of logit(U), while the leaderboard average is an unweighted average of U. Consequently, the headline inverted-U relation is a mathematical consequence of the logit link and the model's estimation from the same data, not evidence of external validity or of a genuinely superior ranking. The paper honestly flags the poor fit of the Study 1 model and the untested measurement invariance across architectures; those are limitations on the empirical analysis, but they are not the circular step. The circularity is narrower: a fitted transformation of the same data is presented as a finding that the naive average 'overestimates' or 'underestimates' LLM ability relative to the factor score, with no external benchmark or baseline aggregation to establish which quantity is closer to the truth. This is a partial circularity: the factor model has independent statistical content, but the specific 'improvement' claimed in the abstract and conclusion reduces, by construction, to the logit rescaling of the leaderboard parcels.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small number of fitted factor-analytic parameters (loadings, intercepts, residual variances, residual covariances) and on the substantive assumption that a single latent ability underlies all six benchmarks. The greedy residual covariance selection is a model-fitting choice that shapes the final model. No new entities (forces, particles, dimensions) are introduced.

free parameters (5)
  • Factor loadings (lambda_p) for six benchmarks = Estimated per benchmark, e.g., Study 1 initial: ARC 13.928, HellaSwag 15.239, MMLU 14.890, TruthfulQA 6.451…
    Estimated from the covariance matrix; these loadings define the factor scores and are central to the ranking.
  • Intercepts (mu_p) = Estimated per benchmark, e.g., Study 2b IFEval -0.399, MATH -4.032
    Location parameters in the mean structure; needed for the factor score formula.
  • Residual variances = Estimated per benchmark in each model
    Residual variances influence the posterior variance of factor scores and the shrinkage toward the mean.
  • Residual covariances added greedily = Study 1: 2 residual covariances; Study 2a: 5; Study 2b: 4, each with estimated values
    Added by a greedy algorithm to improve model fit; the specific set is data-driven and changes the estimated factor structure and interpretations.
  • B-spline knots for trend lines = Three internal knots at 25, 50, 75 percentiles; additional knot at 70B parameters
    Chosen by hand to match the parameter-count distribution; affects the visual trend lines in Study 3 and the conclusions about scaling plateaus.
assumptions (4)
  • standard math Multivariate normality of observed variables V and residuals, with linear factor model (Eq. 5-6)
    Required for maximum likelihood estimation in factor analysis; the paper uses MLR to correct for non-normality of bounded proportions.
  • domain assumption Six benchmark averages are parcels that measure a single common latent ability
    The entire factor-score ranking assumes a unidimensional factor structure; Study 1's poor fit (RMSEA=0.304) challenges this assumption.
  • domain assumption Factor structure is invariant across all LLM architectures and training regimes
    The paper discusses measurement invariance and DIF as limitations, but the factor model is applied to all models pooled together.
  • ad hoc to paper Added residual covariances reflect substantively meaningful dependencies rather than sampling noise
    The greedy algorithm selects residual covariances that improve fit, and the authors interpret them in terms of benchmark content; these are post-hoc and not cross-validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving LLM Leaderboards with Psychometrical Methodology." pith.science (2026). https://pith.science/paper/2KRP3FZ4

@misc{pith2026250117200,
  author       = {Pith},
  title        = {Pith review of: Improving LLM Leaderboards with Psychometrical Methodology},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2KRP3FZ4}},
  note         = {Machine review of arXiv:2501.17200}
}
read the original abstract

The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance. These benchmarks resemble human tests and surveys, as they consist of sets of questions designed to measure emergent properties in the cognitive behavior of these systems. However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined. The most prominent benchmarks are often grouped into leaderboards for convenience, aggregating performance metrics and enabling comparisons between models. Unfortunately, these leaderboards typically rely on simplistic aggregation methods, such as taking the average score across benchmarks. In this paper, we demonstrate the advantages of applying contemporary psychometric methodologies - originally developed for human tests and surveys - to improve the ranking of large language models on leaderboards. Using data from the Hugging Face Leaderboard as an example, we compare the results of the conventional naive ranking approach with a psychometrically informed ranking. The findings highlight the benefits of adopting psychometric techniques for more robust and meaningful evaluation of LLM performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

    cs.CL 2026-03 unverdicted novelty 7.0 of 10

    HumorRank ranks nine LLMs on textual humor using GTVH-grounded pairwise tournaments and Adaptive Swiss aggregation on the SemEval-2026 MWAHAHA dataset, finding that comedic mechanism mastery matters more than scale.

Reference graph

Works this paper leans on

22 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    cognitive

    Table 1 Parameter estimates from the initial model Benchmark Factor loading Intercept Residual variance UnStd Std UnStd Std UnStd Std Est. S.E. Est. S.E. Est. S.E. ARC 13.928 0.146 0.997 54.003 0.227 3.867 1.017 0.735 0.005 HellaSwag 15.239 0.258 0.932 73.862 0.266 4.517 35.186 1.247 0.132 19 MMLU 14.89 0.139 0.918 52.381 0.263 3.231 41.094 1.31 0.156 Tru...

    work page 2023

  2. [2]

    50 Samejima, F. (1973). Homogeneous case of the continuous response model. Psychometrika, 38(2), 203-219. https://doi.org/10.1007/BF02291114 Samejima, F. (1974). Normal ogive model on the continuous response level in the multidimensional latent space. Psychometrika, 39, 111-121. https://doi.org/10.1007/BF02291580 Samejima, F. (1997). Graded response model...

    arXiv 1973

  3. [4]

    Table 4 Residual correlations in the revised model in the order of addition Benchmark Factor loading Intercept Residual variance UnStd Std UnStd Std UnStd Std Est. S.E. Est. S.E. Est. S.E. IFEval 0.671 0.029 0.593 -0.399 0.031 -0.353 0.828 0.166 0.648 BBH 0.433 0.008 0.983 -0.131 0.012 -0.297 0.006 0.002 0.033 Math 1.491 0.087 0.496 -4.032 0.082 -1.342 6....

    work page 2023

  4. [5]

    1 The initial unidimensional model calibrated on the older leaderboard dataset, exhibited relatively poor model fit under Maximum Likelihood Robust estimator (Yuan & Bentler, 2000)

    Results 5.1 Study 1 – Analysis of the Hugging Face Leaderboard v. 1 The initial unidimensional model calibrated on the older leaderboard dataset, exhibited relatively poor model fit under Maximum Likelihood Robust estimator (Yuan & Bentler, 2000). Specifically, SRMR = 0.054, robust RMSEA = 0.304 (90% CI for RMSEA = [0.294, 0.314]), CFI = 0.901, TLI = 0.83...

    work page 2000

  5. [6]

    Which LLM is the best?

    Conclusion With the emergence of ChatGPT, the rise of AI tools for information processing has led to the development of myriad LLMs. Correspondingly, the question “Which LLM is the best?” has become increasingly pressing. To address this, multiple benchmarks have been 35 developed to quantify the performance of various LLMs and rank them based on their (c...

    work page 2003

  6. [7]

    general cognitive ability

    Discussion and Limitations This study opens several new directions for further research, particularly in the application of psychometric methodologies to estimate benchmark quality. The traditional approach to benchmark development assumes that all items, regardless of their quality, should 37 be included in benchmarks, as LLMs – as neural networks (i.e.,...

    work page 2024

  7. [8]

    References AERA, APA, NCME (American Educational Research Association, American Psychological Association, National Council on Measurement in Education). (2014). The Standards for Educational and Psychological Testing. AERA. Aitchison, J. & Brown, J. A. C. (1957). The lognormal distribution. London: Cambridge University Press. Akaike, H. (1974). A new loo...

    arXiv 2014

  8. [11]

    outliers

    have attempted to use human psychological tests to estimate the traits of LLMs. However, this approach has been criticized by others (see the review by Löhn et al., 2024). A central concern is that LLMs lack constructs and psychological traits in the sense that humans possess them. While we generally agree with this critique, we do not believe it renders ...

    work page 2024

Show all 22 references
  1. [20]

    https://doi.org/10.1186/s13040-023-00339-9 Mislevy, R. J. (2024). Are sum scores a great accomplishment of psychometrics or intuitive test theory? Psychometrika, 1-5. https://doi.org/10.1007/s11336-024-10003-8 Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence‐c...

    2024

  2. [21]

    L., Domingue, B

    Wu, M., Davis, R. L., Domingue, B. W., Piech, C., & Goodman, N. (2020). Variational item response theory: Fast, accurate, and expressive. arXiv preprint. https://doi.org/10.48550/arXiv.2002.00276 Yuan, K. H., & Bentler, P. M. (2000)

    2020 doi arXiv

  3. [22]

    Sociological methodology, 30(1), 165-200

    Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. Sociological methodology, 30(1), 165-200. https://doi.org/10.1111/0081-1750.00078 Yuan, K. H., & Bentler, P. M. (2000). Three likelihood-based methods for mean and covariance...

    2000

  4. [25]

    http://dx.doi.org/10.1038/s44271-023-00026-9 Eshima, N. (2022). An Introduction to Latent Class Analysis – Methods and Applications. Springer Nature. https://doi.org/10.1007/978-981-19-0972-6 Evans, N. J. (2019). Assessing the practical differences between model selection meth...

    2022 doi arXiv

  5. [95]

    https://doi.org/10.3390/mca27060095 Robitzsch, A. (2023). Regularized generalized logistic item response model. Information, 14(6),

    2023 doi

  6. [118]

    https://doi.org/10.3969/j.issn.1002-0829.2012.02.010 Cai, L., & Monroe, S. (2014). A New Statistic for Evaluating Item Response Theory Models for Ordinal Data. CRESST Report

    2014

  7. [306]

    https://doi.org/10.3390/info14060306 Rose, N. (2010). Maximum likelihood and Bayes modal ability estimation in two-parametric IRT models: Derivations and implementation. Schriften zur Bildungsf. Rosseel, Y., Jorgensen, T. D., De Wilde, L., Oberski, D., Byrnes, J., Vanbrabant, ...

    2010 doi arXiv

  8. [421]

    https://www.rasch.org/rmt/rmt92a.htm Little, T. D. (2024). Longitudinal structural equation modeling. Guilford Publications. Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structur...

    2024

  9. [839]

    https://files.eric.ed.gov/fulltext/ED555726.pdf Chalmers, R

    National Center for Research on Evaluation, Standards, and Student Testing (CRESST). https://files.eric.ed.gov/fulltext/ED555726.pdf Chalmers, R. P. & Ng, V. (2017). Plausible-Value Imputation Statistics for Detecting Item Misfit. Applied Psychological Measurement, 41, 372-387...

    2017 doi arXiv

  10. [1978]

    stems from the tendency of such criteria to oversimplify the data-generating model (Evans, 2019), particularly in the context of IRT (Robitzsch, 2022). Importantly, none of the factor loadings on the general factor became insignificant or showed a substantial decrease in stand...

    2023

  11. [2021]

    These questions are administered in a 5-shot manner

    – a set of 1,767 MC questions based on the Winograd Schema Challenge (Levesque et al., 2012), designed to measure commonsense reasoning from in-sentence context. These questions are administered in a 5-shot manner. 17 • GSM8K (Cobbe et al.,

    2012

  12. [2022]

    utilitarian

    further support these “utilitarian" approaches to data modeling strategies. Although the clear interpretation of the estimated latent variables is challenging due to the vague definitions of what the original benchmarks aim to measure, these latent variables offer several adva...

    2024

  13. [2023]

    for the statistical programming language R (v. 4.3.0). FA reliability was estimated using the semTools package (v. 0.5-6; Jorgensen et al., 2022). 4.2 Data The data was retrieved from the Hugging Face Leaderboard (Beeching et al., 2023), where the performance of various models...

    2022

  14. [2024]

    A clear ceiling effect indicates that further increases in the number of parameters in LLMs results in increasingly diminishing improvements in model performance

    has reached a performance ceiling. A clear ceiling effect indicates that further increases in the number of parameters in LLMs results in increasingly diminishing improvements in model performance. However, this trend is most apparent in the normalized factor scores and most c...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.