Pith. sign in

REVIEW 4 major objections 4 minor 50 references

This paper proves an information-theoretic reliability ceiling for generative tasks and derives a max-form scaling law that recovers Chinchilla scaling as a special case.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:36 UTC pith:ZQOCIJP7

load-bearing objection Reliability ceiling is a clean, valid observation, but the Chinchilla-corollary claim is internally inconsistent and the derivation leans on unverified kernel assumptions. the 4 major comments →

arxiv 2607.14112 v1 pith:ZQOCIJP7 submitted 2026-05-08 cs.CL cs.AIcs.ITmath.IT

Information-Theoretic Limits of Reliability and Scaling in Language Models

Subhabrata Majumdar This is my paper
classification cs.CL cs.AIcs.ITmath.IT MSC 94A1768T5062G08
keywords reliability ceilingmutual informationscaling lawsChinchilla lawdependency kernelautoregressive degradationcatastrophic forgettinginformation-theoretic limits
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that LLM performance is bounded from above by task structure: every generative task has a reliability ceiling R* = I(X;Y)/H(Y), the fraction of output entropy resolvable from the input, that no architecture or scale can exceed. The gap to that ceiling decomposes into a closable part, from missing but obtainable context, and a permanent subjective part. The paper then derives a first-principles scaling law in which loss equals conditional entropy H(Y|X) plus the larger of a capacity error and a data error, with exponents fixed by the spectral shape of the task mixture. The empirical Chinchilla law emerges as the balanced case. A sympathetic reader would care because this turns vague claims about saturation into testable predictions about when scaling helps, when retrieval helps, and when evaluation rankings are basically noise.

Core claim

The central claim is that every generative task has a ceiling R* = I(X;Y)/H(Y): the fraction of output uncertainty resolvable from the input. By the data processing inequality on Y <- (X,C) <- X <- T <- Y-hat, no generative model, of any architecture or scale, can exceed this ceiling. The paper derives a max-form scaling law: L(N,D) = H(Y|X) + max(A/N^alpha, B/D^beta) + o(max(N^{-alpha},D^{-beta})), where H(Y|X) is the Bayes-optimal irreducible loss. Under balanced scaling, this reduces to the additive Chinchilla law. Autoregressive generation further degrades the ceiling at a rate controlled by the task's dependency kernel.

What carries the argument

Two primitives carry the argument. The reliability ceiling R* = I(X;Y)/H(Y) measures how much output entropy is resolvable from input; it is architecture-free via data processing. The dependency kernel, an LxL matrix K_Y(t,t') = I(y_t; y_{t'} | y_{-{t,t'}}, X), captures inter-token dependencies after conditioning on all other tokens and the input. The scaling law is carried by the spectrum of the task-averaged kernel: power-law eigenvalues and target projections give data exponent beta=(mu_bar-1)/nu_T and capacity exponent alpha=(mu_bar-1)/(d(nu_T+1)), combined as a max.

Load-bearing premise

The load-bearing premise is that the learned predictor's per-mode estimation error follows the exact kernel ridge regression formula err_k(D)=E[f_bar^2_k]/(1+D_eff lambda_k/sigma^2_noise), along with the ad hoc capacity budget M(N)=kappa*N^{1/d}; if either fails, the exponents in Theorem 7 are not established.

What would settle it

Estimate the task-mixture spectral decay nu_T and target regularity mu_bar from a corpus, then train models at multiple (N,D) pairs and check whether the loss follows H(Y|X)+max(A*N^{-alpha}, B*D^{-beta}) with alpha=(mu_bar-1)/(d(nu_T+1)) and beta=(mu_bar-1)/nu_T. A sharper check: scale D well beyond the balanced ratio and test whether loss plateaus at the max-form data term instead of continuing to decrease as an additive law would predict.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the ceiling theorem is right, no amount of scale, data, or architectural innovation can push a task's reliability above R*; high-entropy tasks like creative writing carry a permanent floor.
  • The max-form scaling law predicts diminishing returns from scaling data alone far beyond capacity, and vice versa: the scarcer resource dominates, so balanced scaling is optimal and the Chinchilla law is the balanced case.
  • Context enrichment such as retrieval, few-shot prompting, and tool use raises the ceiling only by closing the resolvable gap delta_r, so returns are bounded by delta_r*H(Y); tasks with a subjective gap cannot be pushed to perfect reliability.
  • Autoregressive degradation is governed by the dependency kernel: banded tasks like code and math are robust and verifiable, dense tasks like creative writing degrade exponentially faster, and per-step verification helps only where constraints are locally checkable.
  • Benchmark rankings on low-ceiling tasks are inherently unstable, so leaderboard comparisons on subjective tasks should not be read as precise statements of model quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the ceiling R* suggests evaluation should use distributional or multi-reference metrics whenever the subjective gap delta_u is positive; a single ground-truth score conflates model quality with task ambiguity.
  • Beyond the paper: the same spectral machinery could yield scaling prediction for in-context learning and test-time compute, since Theorem 11's bound on context enrichment applies to any additional input, including reasoning tokens.
  • Beyond the paper: the dependency kernel could be estimated directly from human-written corpora via conditional mutual information, giving a training-free predictor of where compounding errors or hallucinations will concentrate in a task.
  • Beyond the paper: the forgetting bound proportional to sqrt(1-rho^2) could be used as a diagnostic before fine-tuning: compute kernel alignment between the target task and the model's existing task mixture to predict which skills will degrade.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an information-theoretic framework for generative language models. It defines a reliability ceiling R* = I(X;Y)/H(Y) and argues via the data processing inequality that no model can exceed it (Theorem 1). It then introduces a task-averaged dependency kernel and, under power-law spectral assumptions, derives a max-form scaling law L(N,D) = H(Y|X) + max(A N^{-α}, B D^{-β}) + lower order, with α = (μ̄−1)/(d(ν_T+1)) and β = (μ̄−1)/ν_T (Theorem 7). The paper claims the additive Chinchilla law follows as a special case along the balanced frontier (Corollary 8), and uses the framework to explain retrieval, verification, benchmark instability, and catastrophic forgetting.

Significance. If the scaling-law derivation were fully sound, the paper would make a substantial contribution: it would provide a task-theoretic interpretation of the three Chinchilla terms, explain the scarcity-bottleneck structure, and connect reliability ceilings to autoregressive error propagation. The most defensible contribution is Theorem 1, a clean and genuinely architecture-free/scale-free DPI upper bound. The dependency-kernel concept is also suggestive and links task structure to both degradation and spectral scaling. However, the central derivation is conditional on strong, essentially phenomenological assumptions — kernel-ridge error for transformers, a power-law capacity budget with an inferred constant d, and power-law spectral forms — and the proof of Theorem 7 contains an internal inconsistency about the learned-mode contribution. These issues are load-bearing, so the paper in its present form does not establish its headline claims.

major comments (4)
  1. [§B.4 Step 5 and §B.7; Corollary 8] The proof of Theorem 7 is internally inconsistent. In Appendix B.4, Step 5, for the regime 1 < μ̄ ≤ ν_T + 1 — which the paper identifies as the language-modeling regime — the learned-mode sum satisfies S_≤(D) = Θ(D^{-β}), and the text explicitly says both learned and unlearned modes contribute to the leading prefactor. Yet the proof of Theorem 7 in Appendix B.7 discards the same learned-mode sum as o(D^{-β}_{eff}) when splitting at min(k*, M_eff). This is a same-order term being dropped, not a lower-order one. Consequently, Eq. (8) and the derivation of its prefactor B are not established as written. Moreover, Corollary 8 cannot follow from Eq. (8): along AN^{-α} ≍ BD^{-β}, max and sum differ by Θ(max), not o(max), so the additive equality with the same A and B is invalid. The authors should either prove dominance of the tail, or absorb the learned-mode contribution into a redefined B an
  2. [§B.5, Assumption 8; §6] The capacity exponent α depends on an ad hoc budget M(N) = κN^{1/d} with inverse-eigenvalue cost σ_k^{-1}. The architecture constant d is not derived: the paper calls d≈2 a 'naive estimate,' and the comparison with the empirical α/β ratio is used to conclude d≈1. Thus the 'first-principles' claim for α is not supported; the exponent is effectively calibrated to the Chinchilla data through d. This needs to be either derived from an architecture model or explicitly treated as a phenomenological parameter, with the theorem restated as conditional on that parameter.
  3. [§B.4, Assumption 7] The data exponent β rests on the assumption that a transformer's mode-wise estimation error is exactly the kernel ridge error err_k(D) = E[f̄_k^2]/(1 + D_eff λ̄_k/σ²). This is exact for kernel ridge regression, but modern autoregressive transformers are not in the lazy/kernel regime. The paper's justification — that power-law exponents are 'qualitatively correct' outside the kernel regime — is not a proof. Since β is a central prediction of Theorem 7, the manuscript should either provide empirical evidence on actual LLMs for this error decomposition or clearly limit the theorem's scope to kernel-regime models.
  4. [§B.2, Proposition 2] The proof of Proposition 2(i) is not rigorous. It introduces an unspecified constant q in pflip,t − ε*_t ≥ (1−ε*_t)q, then defines pflip(Δt) := q·TV, and asserts q ≥ 1/√2 without derivation. The relationship between the conditional probability pflip,t and the function pflip(Δt) is also unclear. This proposition underlies the autoregressive-degradation claims in Sections 4, 5, and 7, so the proof needs to be made precise or the bound stated as an additional assumption.
minor comments (4)
  1. [Eq. (8)] The exponent in Eq. (8) uses ν_τ in one place and ν_T elsewhere; this should be ν_T consistently.
  2. [Proposition 2(iii)] There is a typographical error in the definition of γ(L): an extra parenthesis in γ(L) := exp(−c∑Δt)).
  3. [Appendix C.2] The text says 'I(y_y; y_t′|...)' where 'y_y' should be 'y_t'. Also, the proxy used is unconditional MI while the caption says 'Conditional MI'; this discrepancy should be stated more explicitly.
  4. [§6] The statement that α/β ≈ 0.95 is 'consistent with d≈1 and moderately large ν_T' is imprecise: with d=1, the identity α/β = ν_T/(ν_T+1) forces ν_T ≈ 19. Please state the implied ν_T value explicitly.

Circularity Check

1 steps flagged

The only true circular step is the definitional 'unresolvable' subjective floor; the scaling law is a conditional derivation, and the additive-Chinchilla proof flaw is a non-circular mathematical issue.

specific steps
  1. self definitional [Section 3 (Task taxonomy, after Definition 2); echoed in Appendix D.1, Theorem 11]
    "Cu is unresolvable context: subjective information that has no fixed value because the 'correct' output depends on the reader, the cultural moment, or aesthetic judgments that cannot be specified even in principle. ... The subjective gap δu cannot be closed by any intervention, because the information it represents does not have a determinate value."

    The 'permanent floor' δu is true by construction: Cu is defined as unresolvable, so I(Cu;Y|X,Cr) is by stipulation the part of the gap that no intervention can close. Appendix D.1 formalizes this by assuming C̃⊥Cu|(X,Cr), i.e., auxiliary context is assumed not to reveal the quantity already labeled unresolvable. This is a definitional restatement rather than an information-theoretic derivation. The separate ceiling R*=I(X;Y)/H(Y) in Theorem 1 is, however, an independent DPI consequence.

full rationale

Most of the derivation chain is self-contained and not circular. Theorem 1 follows directly from the data-processing inequality on the stated Markov chain, with no fitted parameters. The max-form scaling law in Theorem 7 is a conditional theorem: given Assumptions 2, 3, 7, and 8 (power-law spectra, power-law target regularity, kernel-ridge error decomposition, and a capacity budget), the exponents are computed, not fitted to the Chinchilla data; the later consistency check with empirical α/β is not a prediction forced by a fit. The reliability ceiling, dependency kernel analysis, and autoregressive degradation bounds are also independent mathematical developments. The one clear circular element is the definitional 'unresolvable' subjective floor described above. Separately, there is a non-circularity concern: Appendix B.7 discards the learned-mode sum as o(D^{−β}), citing Step 5 of the Theorem 4 proof, while B.4 Step 5 states that in the relevant regime S≤(D)=Θ(D^{−β}), the same order as S>(D). This makes the proof of Corollary 8 (additive Chinchilla recovery) unsupported. I flag it as a correctness flaw, not as a reduction-to-inputs, so it does not raise the circularity score.

Axiom & Free-Parameter Ledger

4 free parameters · 9 axioms · 3 invented entities

The central claims rest on several unmeasured spectral parameters and a capacity-budget assumption with a data-inferred constant. The reliability ceiling itself is self-contained and standard, but the scaling-law derivation is conditional on domain assumptions that are plausible yet not established for LLMs.

free parameters (4)
  • d (architectural overhead) = ≈1 (inferred from empirical α/β≈0.95)
    Appears in α = (μ̄−1)/(d(νT+1)). The paper does not derive d; it infers d≈1 from corrected Chinchilla estimates, making the capacity exponent partly fitted.
  • νT (mixture spectral decay)
    Assumed power-law eigenvalue decay of the task mixture kernel. No measurement is provided for LLM task mixtures, so β = (μ̄−1)/νT is left uninstantiated.
  • μ̄ (target regularity)
    Assumed power-law decay of target projections. Not measured; appears in both exponents.
  • κ (capacity scale)
    Scale constant in M(N)=κN^{1/d}; affects the prefactor A but not the exponent. Unknown and not estimated.
axioms (9)
  • standard math Data processing inequality and Markov chain Y↔(X,C)↔X↔T↔Ŷ
    Used in Theorem 1 for the reliability ceiling. Standard information-theoretic background.
  • domain assumption Model observes only X, never C; T⊥C|X
    Defines the generative setup and ensures the ceiling is model-independent. Reasonable for the stated setting.
  • domain assumption Assumption 1: continuity of conditional distributions, KL ≤ ω(d)
    Used in Proposition 2 to bound autoregressive drift. The existence and shape of ω are not derived; for natural language this is a strong smoothness assumption.
  • domain assumption Assumption 2: power-law mixture eigenspectrum σ̄_k ∼ σ̄_1 k^{−νT}
    Needed for both scaling exponents. Justified by empirical covariance spectra in other domains, but not specifically measured for LLM tasks.
  • domain assumption Assumption 3: power-law target regularity E[f̄²_k] ∼ f̄₀² k^{−μ̄}
    Needed for both scaling exponents. Standard in spectral bias literature but assumed here for language tasks without direct verification.
  • domain assumption Assumption 5: task–data alignment of covariance eigenvalues
    Used in Theorem 4 to make the data exponent position-independent. Strong alignment between task mixture and training distribution is not guaranteed in practice.
  • ad hoc to paper Assumption 7: kernel-ridge mode-wise error formula
    The entire learning-curve derivation assumes the trained model behaves as kernel ridge regression on the mixture covariance. This is not established for pretrained transformers outside the lazy regime.
  • ad hoc to paper Assumption 8: capacity budget M(N)=κN^{1/d} with inverse-eigenvalue cost
    Introduced specifically to derive the capacity exponent. The cost model σ̄_k^{−1} per mode is not derived, and d is effectively fitted to Chinchilla data.
  • domain assumption Theorem 11 assumption: auxiliary context C̃⊥C_u | (X, C_r)
    Ensures RAG cannot reduce the subjective gap. Plausible but restrictive: some auxiliary context can carry subjective information beyond C_r.
invented entities (3)
  • Latent context C (decomposed into C_r and C_u) no independent evidence
    purpose: Encodes all output-relevant information absent from X; C_r is resolvable, C_u is subjective.
    Defined to make H(Y|X) nonzero and to make the 'subjective floor' follow by construction. No operational measurement procedure is given.
  • Dependency kernel K_Y(t,t′) = I(y_t; y_t′ | y_{−{t,t′}}, X) no independent evidence
    purpose: Quantifies inter-token correlation structure governing autoregressive degradation.
    The paper's Figure 2 uses unconditional pairwise MI as a proxy, not the conditional kernel. No direct evidence for the conditional quantity is provided.
  • Task mixture kernel K̄ with power-law eigenspectrum no independent evidence
    purpose: Defines shared spectral structure across tasks; determines scaling exponents.
    Its power-law form is assumed (Assumption 2), and its parameters are not measured for LLM task mixtures.

pith-pipeline@v1.3.0-alltime-deepseek · 28628 in / 17068 out tokens · 190587 ms · 2026-08-02T14:36:31.154297+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are evaluated as though perfect reliability is achievable for any task given sufficient scale. We show this assumption is information-theoretically unjustified. Every generative task has a reliability ceiling that no model can exceed, determined by how much output uncertainty is resolvable from observable context. The gap decomposes into a resolvable component closable with additional context and a subjective component inherent to task ambiguity. Autoregressive generation further degrades this ceiling at a rate governed by the task's dependency kernel, which quantifies inter-token correlations in the output. From these two primitives, we derive a first-principles scaling law where LLM performance is bottlenecked by the scarcer resource: training data or model capacity. This law recovers the Chinchilla scaling law as a special case and provides a structural account of when scaling improves reliability. Beyond scaling, our framework unifies diverse practical phenomena, such as the benefits of retrieval-augmentation and the spectral mechanics of catastrophic forgetting. Our work formalizes the resource-complexity tradeoffs that govern model performance across domains, offering a unified theory of performance limits in generative language models.

Figures

Figures reproduced from arXiv: 2607.14112 by Subhabrata Majumdar.

Figure 1
Figure 1. Figure 1: Best benchmark scores by task type. Bins of [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: I(yt; yt ′ ) for public benchmarks, with L = 32. Coding (HumanEval, MBPP) and math (GSM8K) tasks show banded or block diagonal structure, whereas ambiguous tasks (CNN/DailyMail and WritingPrompts) have dense kernels with smaller elements. Dependency-modulated degradation Proposition 2 shows that autoregressive degradation is gov￾erned by cumulative drift P t ∆t, but treats drift as a single aggregate quant… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 22 linked inside Pith

  1. [1]

    Akhtar, A

    M. Akhtar, A. Reuel, P. Soni, S. Ahuja, et al. When ai benchmarks plateau: A systematic study of benchmark saturation, 2026. URLhttps://arxiv.org/abs/2602.16763

    Pith/arXiv arXiv 2026

  2. [2]

    Arora, D

    A. Arora, D. Jurafsky, C. Potts, and N. D. Goodman. Bayesian scaling laws for in-context learning, 2025. URLhttps://arxiv.org/abs/2410.16531

    arXiv 2025

  3. [3]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, et al. Program synthesis with large language models,

  4. [4]

    Bahri, E

    Y . Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma. Explaining neural scaling laws.arXiv preprint arXiv:2102.06701, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    P. L. Bartlett, P. M. Long, G. Lugosi, and A. Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063–30070, 2020

    2020

  6. [6]

    Belkin, D

    M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116 (32):15849–15854, 2019

    2019

  7. [7]

    Besiroglu, E

    T. Besiroglu, E. Erdil, M. Barnett, and J. You. Chinchilla scaling: A replication attempt.arXiv preprint arXiv:2404.10102, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Bordelon, A

    B. Bordelon, A. Canatar, and C. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. InInternational Conference on Machine Learning, pages 1024–1034, 2020

    2020

  9. [9]

    Brown, J

    B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V . Le, C. Ré, and A. Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    R. C. Bunescu and R. J. Mooney. A shortest path dependency kernel for relation extraction. InProceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 724–731, 2005

    2005

  11. [11]

    Caballero, K

    E. Caballero, K. Gupta, I. Rish, and D. Krueger. Broken neural scaling laws, 2023. URL https://arxiv.org/abs/2210.14891

    Pith/arXiv arXiv 2023

  12. [12]

    Cagnetta, H

    F. Cagnetta, H. Kang, and M. Wyart. Learning curves theory for hierarchically composi- tional data with power-law distributed features, 2025. URL https://arxiv.org/abs/2505. 07067

    2025

  13. [13]

    Cagnetta, A

    F. Cagnetta, A. Raventós, S. Ganguli, and M. Wyart. Deriving neural scaling laws from the statistics of natural language, 2026. URLhttps://arxiv.org/abs/2602.07488

    Pith/arXiv arXiv 2026

  14. [14]

    Canatar, B

    A. Canatar, B. Bordelon, and C. Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks.Nature Communications, 12:2914, 2021

    2021

  15. [15]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, et al. Evaluating large language models trained on code,

  16. [16]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, et al. Training verifiers to solve math word problems, 2021. URLhttps://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  17. [17]

    URLhttps://arxiv.org/abs/2107.03374

    Pith/arXiv arXiv

  18. [18]

    A. Fan, M. Lewis, and Y . Dauphin. Hierarchical neural story generation, 2018. URLhttps: //arxiv.org/abs/1805.04833. 10

    Pith/arXiv arXiv 2018

  19. [19]

    Dell’Acqua, E

    F. Dell’Acqua, E. McFowland III, E. R. Mollick, H. Lifshitz-Assaf, K. Kellogg, S. Rajaman, L. Krayer, F. Candelon, and K. R. Lakhani. Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School Technology & Operations Mgt. Unit Working Paper No. 24-013, 2023

    2023

  20. [20]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Goldfeld, E

    Z. Goldfeld, E. van den Berg, K. Greenewald, I. Melnyk, N. Nguyen, B. Kingsbury, and Y . Polyanskiy. Estimating information flow in deep neural networks. InInternational Conference on Machine Learning, pages 2299–2308, 2019

    2019

  22. [22]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, et al. Training compute-optimal large language models. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY , USA, 2022. Curran Associates Inc. ISBN 9781713871088

    2022

  23. [23]

    Havrilla and W

    A. Havrilla and W. Liao. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data, 2024. URL https://arxiv.org/abs/2411.06646

    Pith/arXiv arXiv 2024

  24. [24]

    S. Lei, Z. Cheng, K. Jia, and D. Tao. Revisiting llm reasoning via information bottleneck, 2025. URLhttps://arxiv.org/abs/2507.18391

    Pith/arXiv arXiv 2025

  25. [25]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, et al. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  26. [26]

    J. Li, Z. Sun, B. Liang, L. Gui, and Y . He. Cue: an uncertainty interpretation framework for text classifiers built on pre-trained language models. InProceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI ’23. JMLR.org, 2023

    2023

  27. [27]

    B. Li, H. Wang, and H. Wilkinson. POEMetric: The last stanza of humanity. InThe Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=9VkJ058cTa

    2026

  28. [28]

    L. Mazur. LLM creative story-writing benchmark, 2025. URL https://github.com/ lechmazur/writing. Accessed: April 2026

    2025

  29. [29]

    Liang, R

    P. Liang, R. Bommasani, T. Lee, et al. Holistic evaluation of language models.Transactions on Machine Learning Research, 2023

    2023

  30. [30]

    M. A. Mohsin, M. Umer, A. Bilal, et al. On the fundamental limits of LLMs at scale.arXiv preprint arXiv:2511.12869v2, 2025

    arXiv 2025

  31. [31]

    O. C. Mesner and C. R. Shalizi. Conditional mutual information estimation for mixed, discrete and continuous data.IEEE Trans. Inf. Theor., 67(1):464–484, Jan. 2021. ISSN 0018-9448. doi: 10.1109/TIT.2020.3024886. URLhttps://doi.org/10.1109/TIT.2020.3024886

    arXiv 2021

  32. [32]

    A. K. Nayak and L. R. Varshney. An information theory of compute-optimal size scaling, emergence, and plateaus in language models.IEEE Journal of Selected Topics in Signal Processing, 19(7):1338–1348, Oct. 2025. doi: 10.1109/JSTSP.2025.3626264

    arXiv 2025

  33. [33]

    R. J. Moore, S. An, F. Ahmed, and J. P. Gala. Nc-bench: An llm benchmark for evaluating conversational competence, 2026. URLhttps://arxiv.org/abs/2601.06426

    arXiv 2026

  34. [34]

    Panferov, A

    A. Panferov, A. V olkova, I.-V . Modoranu, V . Egiazarian, M. Safaryan, and D. Alistarh. Uni- fied scaling laws for compressed representations. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id= 24wDPGiDzA

    2026

  35. [35]

    S. J. Paech. Eq-bench: An emotional intelligence benchmark for large language models, 2024. URLhttps://arxiv.org/abs/2312.06281

    Pith/arXiv arXiv 2024

  36. [36]

    Schaekermann, J

    M. Schaekermann, J. Beaton, E. Habber, A. Lim, K. Larson, and E. Law. Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work. InProceedings of the ACM on Human-Computer Interaction, 2018. 11

    2018

  37. [37]

    A. M. Saxe, Y . Bansal, J. Doapello, et al. On the information bottleneck theory of deep learning. InInternational Conference on Learning Representations, 2018

    2018

  38. [38]

    Shwartz-Ziv and N

    R. Shwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017

    Pith/arXiv arXiv 2017

  39. [39]

    A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointer-generator networks, 2017. URLhttps://arxiv.org/abs/1704.04368

    Pith/arXiv arXiv 2017

  40. [40]

    Spearman

    C. Spearman. The proof and measurement of association between two things.The American Journal of Psychology, 15(1):72–101, 1904

    1904

  41. [41]

    Simon, D

    J. Simon, D. Kunin, A. Atanasov, E. Boix-Adserà, B. Bordelon, J. Cohen, N. Ghosh, F. Guth, A. Jacot, M. Kamb, D. Karkada, E. J. Michaud, B. Ottlik, and J. Turnbull. There will be a scientific theory of deep learning, 2026. URLhttps://arxiv.org/abs/2604.21691

    Pith/arXiv arXiv 2026

  42. [42]

    L. G. Valiant. A theory of the learnable.Communications of the ACM, 27(11):1134–1142, 1984

    1984

  43. [43]

    Tishby, F

    N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. InProceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, pages 368–377, 1999

    1999

  44. [44]

    Wang and R

    C. Wang and R. Sennrich. On exposure bias, hallucination and domain shift in neural machine translation.arXiv preprint arXiv:2005.03642, 2020

    Pith/arXiv arXiv 2005

  45. [45]

    Walha, S

    N. Walha, S. G. Gruber, T. Decker, Y . Yang, A. Javanmardi, E. Hüllermeier, and F. Buettner. Fine-grained uncertainty decomposition in large language models: A spectral approach, 2025. URLhttps://arxiv.org/abs/2509.22272

    arXiv 2025

  46. [46]

    Z. Xu, S. Jain, and M. Kankanhalli. Hallucination is inevitable: An innate limitation of large language models.arXiv preprint arXiv:2401.11817, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Y . Wu, J. Mei, M. Yan, C. Li, S. Lai, Y . Ren, Z. Wang, J. Zhang, M. Wu, Q. Jin, and F. Huang. Writingbench: A comprehensive benchmark for generative writing, 2025. URL https:// arxiv.org/abs/2503.05244

    arXiv 2025

  48. [48]

    Z. Yang, Z. Qi, Z. Ren, Z. Jia, H. Sun, X. Zhu, and X. Liao. Exploring information processing in large language models: Insights from information bottleneck theory, 2025. URL https: //arxiv.org/abs/2501.00999. A Task Verifiability and Computational Complexity The task taxonomy of Section 3 admits a complexity-theoretic interpretation, connecting fully ver...

    Pith/arXiv arXiv 2025

  49. [49]

    G. Yang. Tensor programs II: Neural tangent kernel for any architecture.arXiv preprint arXiv:2006.14548, 2020

    Pith/arXiv arXiv 2006

  50. [2021]

    URLhttps://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv