Pith. sign in

REVIEW 2 major objections 5 minor 24 references

Within any single LLM, stochastic sampling produces no above-noise cross-question structure; across 24 diverse models, four structured dimensions emerge.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:40 UTC pith:J2PEM7G5

load-bearing objection The within-model null result is a careful empirical contribution; the across-model 'four dimensions' claim rests on a null that ignores per-model ability spread, so the headline gap is unproven. the 2 major comments →

arxiv 2607.20464 v1 pith:J2PEM7G5 submitted 2026-05-18 cs.AI

Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs

Izhar Ali This is my paper
classification cs.AI
keywords stochastic samplingself-consistencyepistemic uncertaintyMarchenko–Pasturrandom matrix theorymodel ensembledimensionality gapselective prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Does running one language model many times at high temperature reveal what it does not know, the way running many different models does? The paper answers no, and measures the gap. Using a Marchenko–Pastur random-matrix test on the run-by-question correctness matrix, it finds that within any single model the correctness patterns are statistically indistinguishable from independent coin flips—at most one eigenvalue at the noise edge, always within sampling fluctuations—across five model families and three benchmarks. Across a 24-model ensemble run once each at temperature zero, four eigenvalues exceed the noise edge, unmatched by any of 500 matched-difficulty null draws. The practical payoff: two external models predict when a model is right better than 100 stochastic samples from the model itself, at about 1/40th the compute.

Core claim

The paper's central claim is that stochastic temperature sampling inside a single LLM is epistemically shallow: the variation it produces is per-question noise, not cross-question structure. Formally, the K×N correctness matrix of one model across repeated runs is statistically indistinguishable from independent Bernoulli draws under the Marchenko–Pastur null—the top eigenvalue sits at or below the noise edge λ+=(1+√γ)^2 within Tracy–Widom fluctuations, for all five families and three benchmarks tested. In contrast, the same test applied to 24 diverse models run once each at temperature zero yields four eigenvalues above the edge, a result unmatched by any of 500 matched-difficulty Monte Car

What carries the argument

The Marchenko–Pastur (MP) null test on the correlation matrix of a run-by-question correctness matrix. One model's K stochastic runs at temperature τ are coded 0/1 per question, restricted to borderline questions (pass rate in (ε,1−ε)); the eigenvalue spectrum of the question correlation matrix is compared to the MP law with upper edge λ+=(1+√γ)^2, γ=Nb/K, and the standardized largest eigenvalue z follows a Tracy–Widom F1 distribution. Eigenvalues above λ+ indicate cross-question coupling that independent temperature noise cannot explain. The paper uses the count of such eigenvalues ('MP signal count') as the primary statistic, with parallel analysis as a cross-check.

Load-bearing premise

The load-bearing premise is Assumption A.1: that correctness on each question is independent of every other question within a run and across runs for a single model—a domain assumption about the sampling pipeline, not a theorem, and one that a binary right/wrong matrix may be too coarse to test.

What would settle it

Run the same Marchenko–Pastur test on the log-probability (or semantic-cluster) correlation matrix within one model; if any eigenvalue exceeds the MP edge λ+ beyond Tracy–Widom noise, the claim that stochastic sampling is epistemically shallow in the cross-question sense is wrong.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Self-consistency cannot substitute for ensemble diversity: deeper sampling from a single model cannot recover a cross-question dimensionality that is not there.
  • Selective prediction is cheaper and better with a few diverse models: two peers run once each beat 100-sample self-consistency at roughly 1/40th the inference cost (AUROC 0.81 vs 0.71).
  • The claim extends beyond MMLU: HellaSwag and GSM8K chain-of-thought also show zero above-noise eigenvalues within single models.
  • The across-model count (four) is a lower bound, not an asymptote, at 24 models; the true rank of cross-model disagreement is at least four and likely larger.
  • Research on uncertainty quantification should budget for model diversity over sample count; the paper frames this as the 'dimensionality gap.'

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The null test uses binary correctness, so it is blind to structure in wrong-answer preferences, log-probabilities, or semantic clusters; a model could carry rich cross-question uncertainty invisible to this probe. The paper acknowledges this in its limitations, but the 'no structure' conclusion must be read as 'no structure in right/wrong labels.'
  • The per-question independence assumption is a domain assumption, not a theorem; if it fails in other setups (e.g., shared prompts or non-independent sampling), the within-model null could be violated and the gap might narrow.
  • A natural extension is to run the same MP test on log-probability or semantic-cluster matrices within one model; if structure appears there, the claim would shift from 'temperature adds no structure' to 'temperature adds no structure in correctness.'
  • The 'four' across-model dimensions is a lower bound; scaling to a larger, more diverse ensemble would likely reveal more dimensions, implying the epistemic gap between within-model and across-model structure is even wider.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper compares stochastic sampling within a single LLM to a diverse ensemble of models using a Marchenko–Pastur (MP) random-matrix test on binary correctness matrices. For five model families on MMLU, HellaSwag, and GSM8K, the run×question correctness matrix yields at most one eigenvalue above the MP noise edge, all within Tracy–Widom fluctuations. Across a 24-model ensemble on MMLU, four eigenvalues exceed the edge, while a matched-difficulty independent-Bernoulli null never produces more than one in 500 Monte Carlo draws. The authors interpret this as a 'dimensionality gap': self-consistency gives per-question uncertainty but no detectable cross-question structure, while diverse ensembles surface multidimensional disagreement. A secondary selective-prediction experiment reports that two peer models outperform 100-sample self-consistency at roughly 1/40th the compute cost.

Significance. If the central claim holds, the paper makes a useful contribution to LLM uncertainty quantification: it offers a structural test for whether stochastic sampling can substitute for model diversity, and it reports a clean within-model null result across multiple families and benchmarks. The paper is candid about several limitations (binary correctness, MP power, SmolLM2 exception, across-model ceiling) and includes Monte Carlo calibrations, a Horn parallel-analysis cross-check, and robustness sweeps. However, the across-model comparison uses a null that does not preserve observed per-model accuracy spreads, so the headline 'dimensionality gap' (≤1 within vs. 4 across) is not currently established to the standard the paper claims.

major comments (2)
  1. [§2.3, §3.3, §2.4] The across-model null is not row-sum-preserving. Under the null of independent columns with per-question pass rates matched, the 24 model rows are exchangeable and each model's total score is Binomial with mean Σ p_i; the expected spread in model accuracy is only a few percentage points, yet the observed ensemble spans 40.2–77.4% (§2.4). Systematic per-model ability differences—a single general factor—generate positive correlations across all questions and can, in finite samples, push several empirical eigenvalues above the MP edge. Thus the four above-edge eigenvalues in §3.3 may reflect one strong ability dimension rather than four independent disagreement dimensions. Because the within-model null (Assumption A.1) makes row sums genuinely random, the ≤1 vs. 4 comparison is apples-to-oranges. Please rerun the across-model test with a null that preserves observed row sums (e.g., a bipart
  2. [Abstract, §3.1, §5] The paper acknowledges in §5 that the within-model test uses binary correctness, but the abstract and §3.1 state 'no detectable cross-question structure' without that qualifier. The MP test is blind to structure in wrong-answer preferences, log-probabilities, semantic clusters, or per-question uncertainty beyond the right/wrong bit. Appendix D's local-preference analysis is still on the binary matrix. Please scope all such claims to 'no detectable structure in the binary correctness matrix' in the main text and abstract, and consider whether the title's 'epistemically shallow' overstates what a binary-correctness null can establish. This is not a rejection issue because the limitation is explicitly stated, but the wording currently overreaches.
minor comments (5)
  1. [§2.2 vs §3.3] §2.2 designates the Tracy–Widom standardized z as the primary null test and calls the eigenvalue count a 'coarse summary', but §3.3 calls the MP count 'primary' for the across-model result. Clarify which statistic is pre-specified as primary and report both consistently for all tables.
  2. [Appendix D] The permutation test summary ('no pairwise associations above the 99th percentile of the null, 200 permutations per model') lacks a multiple-testing correction across the large number of question pairs, and 200 permutations gives a coarse null. Report corrected thresholds or a larger permutation count.
  3. [Table 1 / §3.1] For HellaSwag, Qwen2.5-7B has only Nb=15 borderline questions with K=50; the MP/TW asymptotics at γ=0.3 and such small dimension are not reliable. The overall conclusion is supported by other rows, but this row should be flagged as underpowered or removed from the 'no structure' claim.
  4. [Reproducibility] No code or data availability statement is provided. Given that the paper's contribution is purely empirical, releasing the sampling pipeline, question subsets, and null-draw code would substantially aid verification.
  5. [Fig. 1c] The caption text 'Split-half pi on Qwen2.5-7B' appears to be a typo; should be 'Split-half reliability' or 'Split-half Pearson correlation'.

Circularity Check

0 steps flagged

No significant circularity: the MP tests are null-hypothesis tests with external mathematical grounding; observed eigenvalue counts are data, not fitted outputs.

full rationale

No circular step is present. The within-model claim rests on a standard spectral null: Assumption A.1 posits independent Bernoulli columns, and Proposition A.2 derives the Marchenko-Pastur/Tracy-Widom null from external theorems (Pillai & Yin 2012; Bao et al. 2012). The observed eigenvalue counts are not constructed from the null; they are empirical rejections or failures to reject. Estimating per-question pass rates from the same data and then selecting borderline questions is explicitly handled in the proof sketch of Proposition A.2 via a union bound, so this is not a definitional circularity. The across-model matched-difficulty Bernoulli null fits per-question pass rates, but the predicted quantity (number of eigenvalues above the MP edge) is not a mathematically forced function of those rates; it is a Monte Carlo null distribution, and the observed count of 4 is an empirical comparison. There is no load-bearing self-citation: prior factor-analytic work is cited only as external context, and the novel within-model result is not justified by those citations. The skeptic's concern that the across-model null does not preserve per-model accuracy spreads is a real methodological limitation: a single general ability factor might inflate several eigenvalues, so the interpretation of the count 4 as multi-dimensional is contestable. But that is a correctness/validity objection, not a reduction of the paper's derivation to its own inputs. Section 5 also candidly lists limitations (binary correctness, MP power, across-model ceiling, post-training), which supports that the findings are empirical rather than manufactured by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The main free parameter is the borderline threshold ε (0.05), but the robustness sweep shows the result is not sensitive to it. The key hidden assumption is that binary correctness on a fixed set of prompts captures all cross-question structure; if wrong, the 'shallow' conclusion is an artifact of the probe, not a property of the model.

free parameters (3)
  • borderline threshold ε = 0.05
    Defines which questions are 'borderline' and enter the MP test; the choice affects the eigenvalue spectrum and the test's power. Swept in Appendix B.2, but the main result uses 0.05.
  • number of samples K (100 within-model, 1 across-model) = 100 / 1
    The within-model test uses K=100 samples, which controls the MP γ and the statistical power to detect structure. The across-model test uses M=24 (one run each). These are experimental choices, not fitted numbers, but they are free parameters in the test design.
  • temperature τ for sampling = 1.0
    The main within-model result uses τ=1.0; τ=1.5 also gives zero signal. This is a design choice, not fitted, but the result could depend on τ.
axioms (3)
  • domain assumption Assumption A.1: correctness indicators are independent Bernoulli draws across runs (within a model) and approximately independent across questions.
    This is the null being tested, but it is also assumed for the MP limit. If the samples are not independent (e.g., due to shared KV cache or prompt order), the null is misspecified.
  • standard math Marchenko–Pastur and Tracy–Widom convergence for a binary matrix with column-heterogeneous variances.
    Prop A.2 cites Pillai & Yin and Bao et al. for edge universality of correlation matrices with column-dependent variances. This is a strong but plausible theorem.
  • domain assumption Binary correctness is a sufficient probe for cross-question epistemic structure.
    The paper only records right/wrong. Richer signals (log-probs, answer-choice patterns, semantic clusters) could contain structure invisible to this binary test, as the paper concedes in Limitations.
invented entities (1)
  • No new entities no independent evidence
    purpose:
    The paper introduces no new particles, forces, or mathematical objects.

pith-pipeline@v1.3.0-alltime-deepseek · 10374 in / 6504 out tokens · 51119 ms · 2026-08-02T13:40:11.889523+00:00 · methodology

0 comments
read the original abstract

When a language model gives different answers on repeated runs, does that variation reveal what it does not know? Self-consistency turns the variation into a per-question uncertainty estimate via majority voting. But does the same variation reveal cross-question structure -- related questions flipping together, the way a diverse ensemble does? We compare two regimes on the same questions: one model run $100$ times at $\tau=1$ versus an ensemble of $24$ LLMs run once each at $\tau=0$. A Marchenko--Pastur random-matrix test separates signal from sampling noise on both sides. Within any single model, at most one dimension rises above noise across five families and three benchmarks (MMLU, HellaSwag, GSM8K). Across the ensemble, four eigenvalues clear the noise edge, while a matched-difficulty Bernoulli null produces at most one in $500$ Monte Carlo draws. Self-consistency gives accurate per-question uncertainty but no detectable cross-question structure; only a diverse ensemble surfaces what a model does not know.

Figures

Figures reproduced from arXiv: 2607.20464 by Izhar Ali.

Figure 1
Figure 1. Figure 1: The dimensionality gap. (a) Eigenvalue spectrum of the correctness correlation matrix on borderline MMLU questions (Qwen2.5-7B, K = 100, Nb = 121). No eigenvalue exceeds the MP noise edge λ+ = 4.41 (red dashed). (b) Same-metric comparison: MP signal count is ≤1 for every one of five families on three tasks but 4 for the 24-model ensemble (τ = 0), unmatched by any of 500 matched-difficulty independent-Berno… view at source ↗
Figure 2
Figure 2. Figure 2: Cost-AUROC Pareto frontier for selective prediction on Qwen2.5-7B (τ = 0), 500 MMLU questions. Ensemble2 (AUROC 0.81) beats SC100 (0.71) at ∼ 1/40th the compute cost. Grey dotted: SC oracle upper bound (requires ground-truth labels at inference, not deployable). Ensemble Pareto-dominates opera￾tional SC across the entire cost range. Cost: Qwen-7B-equivalent forward passes. Per-question estimation. Split-ha… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 4 linked inside Pith

  1. [1]

    International Conference on Learning Representations (ICLR) , year=

    metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  2. [2]

    Maimon, Aviya and Cohen, Amir D. N. and Vishne, Gal and Ravfogel, Shauli and Tsarfaty, Reut , journal=

  3. [3]

    Yao, Louie Hong and Jarvis, Nicholas and Zhan, Tiffany and Ghosh, Saptarshi and Liu, Linfeng and Jiang, Tianyu , journal=

  4. [4]

    General Scales Unlock

    Zhou, Lexin and Pacchiardi, Lorenzo and Mart. General Scales Unlock. arXiv preprint arXiv:2503.06378 , year=

    Pith/arXiv arXiv

  5. [5]

    NeurIPS Workshop on Reliable and Responsible Foundation Models , year=

    Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification , author=. NeurIPS Workshop on Reliable and Responsible Foundation Models , year=

  6. [6]

    International Conference on Machine Learning (ICML) , year=

    Correlated Errors in Large Language Models , author=. International Conference on Machine Learning (ICML) , year=

  7. [7]

    Nature , year=

    Detecting hallucinations in large language models using semantic entropy , author=. Nature , year=

  8. [8]

    International Conference on Learning Representations (ICLR) , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  9. [9]

    Psychometrika , volume=

    A Rationale and Test for the Number of Factors in Factor Analysis , author=. Psychometrika , volume=

  10. [10]

    Mathematics of the USSR-Sbornik , volume=

    Distribution of Eigenvalues for Some Sets of Random Matrices , author=. Mathematics of the USSR-Sbornik , volume=

  11. [11]

    The Illusion of Stochasticity in

    Gu, Xiangming and De, Soham and Titsias, Michalis and Markeeva, Larisa and Veli. The Illusion of Stochasticity in. arXiv preprint arXiv:2604.06543 , year=

    Pith/arXiv arXiv

  12. [12]

    Scenario-independent Uncertainty Estimation for

    Wen, Zhangyue and Liu, Zifei and Tian, Zhenyu and Pan, Shirui and Huang, Ziwei and Li, Donglin and Huang, Minlie , booktitle=. Scenario-independent Uncertainty Estimation for

  13. [13]

    Dropout as a

    Gal, Yarin and Ghahramani, Zoubin , booktitle=. Dropout as a

  14. [14]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  15. [15]

    International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. International Conference on Learning Representations (ICLR) , year=

  16. [16]

    Monte Carlo Temperature: a robust sampling strategy for

    Cecere, Nicola and Bacciu, Andrea and Fern. Monte Carlo Temperature: a robust sampling strategy for. Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP) at NAACL , year=

  17. [17]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  18. [18]

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin , booktitle=

  19. [19]

    Annals of Statistics , volume=

    Edge universality of correlation matrices , author=. Annals of Statistics , volume=. 2012 , note=

    2012

  20. [20]

    Bao, Zhigang and Pan, Guangming and Zhou, Wang , journal=. Tracy--

  21. [21]

    arXiv preprint arXiv:2207.05221 , year=

    Language Models (Mostly) Know What They Know , author=. arXiv preprint arXiv:2207.05221 , year=

    Pith/arXiv arXiv

  22. [22]

    Manakul, Potsawee and Liusie, Adian and Gales, Mark JF , booktitle=

  23. [23]

    What Uncertainties Do We Need in

    Kendall, Alex and Gal, Yarin , booktitle=. What Uncertainties Do We Need in

  24. [24]

    Machine Learning , volume=

    Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods , author=. Machine Learning , volume=