{"id":"641591b9-5cb0-44f7-8735-3c07a9f19fbf","arxiv_id":"2504.18048","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Under gradient and log-probability insensitivity assumptions, SGLD-based local learning coefficient estimates cannot distinguish a sequence distribution from its mode-truncated effective version.","lead":"This paper gives a mathematical way to split the patterns in text data into strong and weak modes, and argues that a popular complexity measure for neural networks, the local learning coefficient, only sees the strong modes. This matters because it explains why that measure can be stable in practice even when the network has not reached a true minimum of its loss.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Theorem 7.4 bound is not shown to be small: it diverges as M approaches γ/(nβ), and A,B are never estimated, so the practical 'cannot distinguish' claim is unsupported.","rationale":"I read the paper's core as Theorem 7.4 plus the interpretation in Section 8 as a statement about what LLC estimates actually measure. The mathematical development in Sections 3-6 appears consistent, and the SVD/mode construction is well-motivated. The load-bearing link is the quantitative magnitude of the theorem's error bound. The reader's weakest assumption pointed to the unverified insensitivity constants and M<30; I agree partially and sharpen it: even conditional on M<30, the bound has a 1/(γ/(nβ)−M) singular factor, and B appears multiplied by nβ. Remark 7.5 is honest about M but does not discuss A,B or the amplification. A concrete numerical test on a small Pythia model with the paper's own hyperparameters would settle whether the bound can be small. Until then, the appropriate verdict is conditional acceptance: the theory is plausible and the caveats are acknowledged, but the central practical claim is not yet supported. I do not see an internal flaw requiring rejection.","tokens_in":30562,"tokens_out":7277,"duration_ms":78679,"concrete_test":"Run the following check on a small Pythia model (e.g., Pythia-70m or 160m): with the LLC hyperparameters of Remark 7.5, execute the SGLD chain used for LLC estimation and, at sampled points, estimate M via the largest Hessian eigenvalue of the minibatch loss (e.g., PyHessian or Lanczos). Also estimate A and B by computing the norms in Definitions 6.3 and 6.4 for a corpus-derived q(χ) truncation. Then evaluate the right-hand side of (75). If M≥30, the theorem's hypothesis fails for these hyperparameters; if the RHS exceeds the Monte Carlo error of λ̂ (or a small fraction such as 10% of λ̂), the practical 'cannot distinguish' claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLC estimates in practice measure the effective distribution requires the bound in Theorem 7.4 to be smaller than the estimator's resolution. Setting ξ=κ=0, the bound (75) is (ϵmax/ϵmin)·nβQA/(γ/(nβ)−M) + 2nβB. Under the paper's own LLC hyperparameters (Remark 7.5: nβ=10, γ=300, ϵmax=ϵmin), the condition Mnβ∈(γ−2/ϵmax,γ) becomes M<30 and the A-prefactor is 100Q/(30−M). If M=25, each unit of A contributes 20Q; if M=29, it contributes 100Q. The paper cites a ResNet max Hessian eigenvalue of 150 and admits no transformer Hessian estimates, so M could easily be too large for the condition, and even if M<30 the prefactor strongly amplifies any uncertainty in A. Moreover A and B are not measured anywhere: B enters as 2nβB=20B, so unless B≲0.01 the second term is already at a scale that could matter for LLC values. The theorem itself is internally coherent, but the abstract and conclusion assert a practical fact. To derive it one needs the entire bound (75) to be small; the paper provides no numerical or empirical evidence for this, and Remark 7.5 only addresses M. This is a quantitative gap, not a contradiction: the bridge from a conditional theorem to the unconditional practical claim is missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a tensor-decomposition framework for conditional sequence distributions, defines a Hilbert-space basis of 'modes' via SVD of the conditional-probability operator C_{k,l}, and introduces a coarse-grained 'effective true distribution' q^(χ) by truncating modes. It then introduces two formal conditions—-gradient-insensitivity and log-probability-insensitivity (Definitions 6.3 and 6.4)—-and proves, in Theorems 7.3 and 7.4, that under these conditions plus Lipschitz and hyperparameter assumptions, SGLD-based LLC estimates of the true loss L and of the truncated loss L^(χ) are close. The authors conclude that LLC estimates computed in practice characterize the effective, mode-truncated distribution rather than the full sequence distribution, and that the inverse temperature β acts as a resolution dial. The paper also reports empirical SVD modes computed on the Pile with a Pythia tokenizer.","tokens_in":30940,"tokens_out":6185,"duration_ms":68587,"significance":"If the main theorem could be turned into a quantitative statement about real transformers, the paper would make a useful conceptual contribution to singular learning theory and to the interpretation of LLC measurements in language models: it would explain why LLC estimates can be stable at parameters that are not strict minimizers of the population loss, and it would connect mode truncation to tempered posterior sampling. The tensor-algebraic setup is clean and the proofs of Theorems 7.3 and 7.4 are internally coherent given their hypotheses. The paper is also honest in Remark 7.5 about the strength of the Lipschitz assumption. The central limitation is that the insensitivity constants A and B and the Lipschitz constant M are never measured or bounded for transformers, and the theorem's bound is not shown to be smaller than the resolution of real LLC estimates. The practical conclusion in the abstract and conclusion therefore remains a conditional statement rather than an established fact.","major_comments":[{"comment":"The quantitative gap between the theorem and the paper's central claim is load-bearing and unresolved. The bound in Eq. (75) with ξ=κ=0 is (ϵmax/ϵmin)·nβQA/(γ/(nβ)−M) + 2nβB. Under the paper's own hyperparameters (Remark 7.5: nβ=10, γ=300, ϵmax=ϵmin) and the plausible estimate Q≈10, the first term is 100A/(30−M), which is 20A at M=25 and 100A at M=29, and the second term is 20B. For the theorem to support the statement that 'LLC calculated in practice characterises the effective rather than the true distribution,' the entire bound must be small compared with the resolution of the LLC estimator. The manuscript provides no estimates of A or B, no transformer Hessian spectral bound for M, and no comparison of the bound with estimator variance. Remark 7.5 concedes that the M<30 condition would not be realistic if the Lipschitz constant were orders of magnitude larger and that no Hessian estimates for large transformers were found. This is a fixable but essential gap: the abstract and conclusion assert a practical fact that the theorem, as quantified, does not deliver.","section":"§7, Theorem 7.4 and Remark 7.5"},{"comment":"The two insensitivity conditions are effectively a formal restatement of the conclusion they are used to prove. Gradient-insensitivity for constant A requires that the pairing of Ψ_{w∗} with q−q^(χ) have norm less than A over W′, and log-probability-insensitivity similarly bounds ⟨Φ(w∗), q−q^(χ)⟩. For q^(χ) obtained by mode truncation, these conditions say precisely that the model's gradients and log-probabilities do not see the truncated modes. The proof then propagates this insensitivity through SGLD. Since A and B are never measured, the theorem is conditional on the very phenomenon it is meant to establish. The motivation in §6.1 appeals to finite capacity but gives no capacity-dependent bound, and the plausibility argument in §6.2 relies on the auxiliary assumptions that all large modes are absolute bigrams, that q(στ)≤2^{−(k+l)H}, and that ∥P_{≤χ}(q)−q^(χ)∥ is small; these are stated as reasonable rather than verified. A direct measurement of A, B, and the gradient/log-probability differences for a small transformer would be needed to make the bridge from the conditional theorem to the practical claim.","section":"§6, Definitions 6.3 and 6.4"},{"comment":"The threshold χ is not given an operational characterization. The effective distribution q^(χ) is defined as the KL projection of q onto the submanifold P_{≤χ}, which is not the same as the orthogonal projection P_{≤χ}(q); Remark 4.18 explicitly notes that the naive truncation need not be a probability distribution. The main theorem is stated for an arbitrary q^(χ) satisfying the insensitivity conditions, but the paper never proves or estimates for which mode cutoffs χ those conditions hold with small A and B. Consequently, the 'data-dependent threshold' mentioned in the abstract remains unspecified, and the theorem cannot be used to predict, for a given model and dataset, which modes will be invisible to LLC estimation.","section":"§4.5 and Theorem 7.4"},{"comment":"The asymptotic regime of the theorem is not tied to a scaling of β with n. The theorem says 'there exists large enough n' while imposing Mnβ∈(γ−2/ϵmax,γ). If β is taken as the standard β0/logn from Watanabe (2013), then nβ grows with n and the condition eventually fails because γ/(nβ)→0. If instead nβ is held fixed at the practical value 10 used in Remark 7.5, then β shrinks as 1/n and it is unclear whether the Glivenko-Cantelli and local-minimum hypotheses of LLC estimation are satisfied in the asymptotic regime. The proof needs a precise statement of how n, β, m, and the insensitivity constants scale; otherwise the bound in Eq. (73) is not a well-defined asymptotic guarantee.","section":"§7, Theorem 7.4 and Remark 7.5"}],"minor_comments":[{"comment":"The word 'minises' in Definition 4.16 should be 'minimizes'.","section":"§4.5, Definition 4.16"},{"comment":"The text refers to 'Theorem 4.16' when citing the construction of q^(χ); this should be 'Definition 4.16'.","section":"§7, paragraph before Eq. (61)"},{"comment":"The probability lower bound 1−12T(T+1)δ−κ can be negative or meaningless for large T unless T is restricted, for example Tδ<1; the theorem should state this restriction explicitly.","section":"§7, Eq. (73)"},{"comment":"The notation would be clearer if the dependence of A and B on W′, q, and q′ were made explicit, since the constants are used later without restating their domain of validity.","section":"§6, Definitions 6.3 and 6.4"},{"comment":"The empirical modes are computed from smoothed, filtered conditional probability matrices, not from the exact q; this is described in Appendix B, but a sentence in Section 5.2 reminding the reader that these are approximate empirical modes would improve clarity.","section":"§5.2 and Appendix B"},{"comment":"The tokenized example is not typeset as a quoted string and is hard to read; using a monospaced or quoted format would improve presentation.","section":"§3.2, Example 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound conditional core: the tensor-mode framework is clearly developed and Theorems 7.3 and 7.4 appear internally correct under their hypotheses. The problem is that the manuscript's headline practical claim is not supported by the quantified bound, because A, B, and M are unmeasured for transformers and the bound is not compared with estimator resolution. I would support publication after the authors either (i) add empirical estimates of A, B, M (or a controlled proxy experiment on a small transformer) and show that the bound in Eq. (75) is small relative to the estimator's resolution, or (ii) explicitly reframe the contribution as a conditional theorem about effective potentials, removing the unconditional practical claim from the abstract and conclusion. The current version sits between a theorem paper and an empirical claim paper without fully delivering either."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious theory paper with a clean conditional theorem and an abstract that overstates what the theorem licenses. The genuinely new piece is Theorem 7.4: under gradient- and log-probability-insensitivity, SGLD-based LLC estimates cannot distinguish the original potential from the mode-truncated one. The effective true distribution construction in Section 4.5 is also a useful formalization, and the paper is honest about building on existing tensor/SVD and Markov-model literature rather than pretending those tools are new.\n\nThe proof of the theorem looks internally coherent. I did not find a load-bearing gap in the derivation given the assumptions. The discussion of inverse temperature as a resolution dial is suggestive and worth taking seriously. Remark 7.5 is unusually candid: it flags that the M < 30 condition is unchecked for transformers, that no Hessian spectral estimates for large transformers could be found, and that the bound would be unrealistic if the Lipschitz constant were orders of magnitude larger.\n\nThe soft spots are real but not fatal. First, the bridge from theorem to practice is missing. The constants A and B are never measured, and the bound in Theorem 7.4 has a denominator (γ/(nβ) − M) that blows up as M approaches the allowed limit. Even with plausible values, the prefactor can amplify a modest A into a bound much larger than typical LLC differences, so the practical claim that LLC estimates \"in practice\" characterize the effective distribution is not actually supported. The stress-test note's arithmetic has a minor slip (the A-prefactor is 10Q/(30−M) under the paper's own hyperparameters, not 100Q/(30−M)), but the qualitative point stands. Second, the abstract and conclusion assert a practical fact, while the theorem is explicitly conditional; the paper would be more accurate if the claims were restated as conditional on the insensitivity constants being small enough. Third, the empirical mode illustrations come with no code or data, so they are not independently reproducible. Fourth, the log-probability-insensitivity motivation leans on assumptions about entropy rates and context length that are plausible but not verified for real transformers.\n\nI want to be clear: none of this undermines the conditional theorem itself. The paper is a legitimate contribution to the SLT literature, and the citation pattern is solid. The right next step is not rejection but revision that either supplies empirical estimates of A, B, and M for actual transformers, or carefully rephrases the practical claims as conditional. Who is this for? Researchers working on LLC estimation, singular learning theory, and mode-based interpretability. I would send it to a serious referee, and I would probably cite it as a conditional formalization, but I would not yet cite it as an established fact about transformer loss landscapes.","headline":"A serious, honestly-written conditional theorem linking mode truncation to LLC insensitivity, with a practical claim that outruns what the assumptions currently support.","tokens_in":31423,"tokens_out":2412,"would_cite":true,"duration_ms":26669,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62B10","15A69"],"pacs":[],"model":"deepseek-v4-flash","headline":"Local learning coefficient estimates from SGLD characterise a mode-truncated 'effective' data distribution rather than the full transformer sequence distribution.","keywords":["sequence models","local learning coefficient","singular value decomposition","effective distribution","stochastic gradient Langevin dynamics","mode truncation","singular learning theory","transformers"],"falsifier":"Measure the spectral norm of the minibatch Hessian along an SGLD chain in a large transformer: if the resulting Lipschitz constant $M$ is not below $\\gamma/(n\\beta)$ (about 30 under the paper's Pythia-style settings), the central hypothesis $Mn\\beta<\\gamma$ fails and the theorem's conclusion does not apply.","tokens_in":30340,"feed_emoji":"📉","tokens_out":12449,"duration_ms":111512,"temperature":0.7,"pith_summary":"The paper develops a geometric account of what local learning coefficient (LLC) estimates measure when they are computed on transformer sequence models. Singular value decomposition of the conditional sequence-probability tensor produces principal modes, and truncating the small-amplitude modes yields an effective true distribution. The main theorem states that, when a model is insensitive to the truncated modes in both its gradients and its log-probabilities, SGLD-based LLC estimates cannot distinguish the original loss from the loss of the truncated distribution. The paper concludes that in practice LLC values describe the geometry of the effective distribution, with the inverse temperature of the sampler acting as a resolution dial.","feed_headline":"Learning coefficient estimates measure a truncated version of the data","feed_subtitle":"A theorem shows LLC values from SGLD describe an effective, mode-truncated distribution, not the full sequence distribution.","key_machinery":"The load-bearing object is the mode decomposition of the conditional-probability map $C_{k,l}: V_k\\to \\mathbb{R}^{\\Sigma^l}$, defined by $C_{k,l}(x)=\\sum_y q(y|x)y$, where $V_k$ is the $q$-weighted Hilbert space on length-$k$ contexts. SVD produces right vectors $v_\\alpha$, left vectors $u_\\alpha$, and singular values $s_\\alpha$; the index $\\alpha$ is a mode with propensity $s_\\alpha^2$, and truncating to modes $\\alpha\\le\\chi$ followed by KL projection gives the effective true distribution $q^{(\\chi)}$. The proof couples two SGLD chains driven by the empirical gradient $\\nabla L_m$ and the truncated gradient $\\nabla L_m^{(\\chi)}$, and uses the two insensitivity conditions together with Lipschitz continuity to bound both the chain distance and the difference of the resulting LLC estimates.","core_discovery":"The paper's central discovery is that the geometry probed by SGLD-based LLC estimation is the geometry of an effective distribution obtained by truncating the small modes of the data. Starting from the conditional-probability map $C_{k,l}(x)=\\sum_{y\\in\\Sigma^l} q(y|x)y$, the paper performs an SVD in a natural Hilbert space; the singular components are the modes, whose propensities are the squared singular values, and retaining modes up to a cutoff $\\chi$ defines an effective true distribution $q^{(\\chi)}$. Theorem 7.4 then shows that if the model is gradient-insensitive and log-probability-insensitive to $q-q^{(\\chi)}$ on a region containing the SGLD chain, and if the relevant losses are Lipschitz with $Mn\\beta<\\gamma$, the SGLD-based LLC estimates computed from $L$ and from $L^{(\\chi)}$ cannot be distinguished within a bound controlled by the insensitivity constants. The paper's conclusion is that LLC estimates calculated in practice characterise the effective, mode-truncated distribution rather than the full sequence distribution.","pith_inferences":["If the theorem survives contact with real transformers, systematic $\\beta$-scans of LLC estimates become a way to map the mode spectrum of a model's training distribution, since lower $\\beta$ sees only dominant modes and higher $\\beta$ resolves rarer ones.","The result implies that comparing LLC values across models or training runs is only meaningful when SGLD hyperparameters are matched, because the effective mode cutoff that an estimate sees depends on those hyperparameters.","The paper's connection to tensor renormalisation group suggests a testable extension: hierarchical or entanglement-guided truncations of the fundamental tensor could yield effective potentials that make LLC estimation both more accurate and more interpretable.","A direct experimental check on synthetic sequence distributions with known mode spectra could measure how much LLC estimates change when small modes are removed at fixed $\\beta$; large changes would indicate the insensitivity constants are bigger than the theorem requires."],"forward_implications":["LLC estimates on transformer language models should be read as measuring the geometry of an effective, mode-truncated distribution rather than the full sequence distribution.","Stable LLC estimates at parameters that are not local minimisers of the population loss become less puzzling, because such parameters can be stationary points of the effective potential $L^{(\\chi)}$.","The inverse temperature $\\beta$ acts as a resolution dial: raising $\\beta$ lets the estimate count contributions from rarer modes, while lowering $\\beta$ restricts the estimate to the dominant modes.","Reported LLC values carry meaning only relative to the SGLD hyperparameters, since the bound depends on $n\\beta$, $\\gamma$, the step sizes, and the Lipschitz constants of the loss.","The mode decomposition offers a principled coarse-graining of sequence data, and the leading empirical modes correspond to interpretable patterns such as punctuation and common subword collocations."],"supporting_citations":[{"why":"Defines the SGLD-based LLC estimator whose values Theorem 7.4 compares.","marker":"(Lau et al., 2024)"},{"why":"Supplies the singular learning theory definition of the learning coefficient and the free-energy expansion the estimator targets.","marker":"(Watanabe, 2009)"},{"why":"Provides the inverse-temperature scaling result that justifies the LLC estimator as a valid estimate of the learning coefficient.","marker":"(Watanabe, 2013)"},{"why":"Introduces the stochastic gradient Langevin dynamics sampler used to approximate the local posterior.","marker":"(Welling & Teh, 2011)"},{"why":"Provides the Glivenko\\u2013Cantelli uniform convergence results used to pass from empirical to population losses in Lemma 7.1.","marker":"(Vaart, 1998)"},{"why":"Gives the uniqueness of the KL projection used to define the effective true distribution $q^{(\\chi)}$.","marker":"(Amari, 2009)"},{"why":"Supplies the language condition relating fundamental tensors across sequence lengths, used to justify the joint distribution framework.","marker":"(Mumford, 1994)"},{"why":"Reports transformer LLC measurements and the non-minimiser puzzle that motivates the effective-potential interpretation.","marker":"(Hoogland et al., 2024)"},{"why":"Provides the Pythia model and tokenizer whose training setup grounds the numerical plausibility check in Remark 7.5.","marker":"(Biderman et al., 2023)"},{"why":"Offers Hessian spectral-norm estimates for a trained network used to calibrate whether the $M<30$ Lipschitz bound is plausible.","marker":"(Yao et al., 2020)"}],"fun_headline_variants":["LLC estimates probe a mode-truncated data geometry","SGLD learning coefficients see truncated sequence modes","Mode truncation explains LLC estimates in sequence models","LLC measures effective data not full distribution","Truncated modes define what SGLD sees in learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hinges on the assumption that, in the region where SGLD runs, a transformer's gradients and log probabilities barely change when the small modes are removed, and that the loss is smooth enough that its gradient's Lipschitz constant $M$ satisfies $Mn\\beta<\\gamma$; with the paper's Pythia-style hyperparameters this requires $M<30$, which the paper does not measure for large transformers.","fun_headline_variants_meta":{"raw":{"variants":["LLC estimates probe a mode-truncated data geometry","SGLD learning coefficients see truncated sequence modes","Mode truncation explains LLC estimates in sequence models","LLC measures effective data not full distribution","Truncated modes define what SGLD sees in learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3267,"prompt_tokens":907,"completion_tokens":2360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2285}},"tokens_in":523,"tokens_out":2360,"duration_ms":15596,"temperature":1.0,"reasoning_tokens":2285,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:26:27.098719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the spectral norm of the minibatch Hessian along an SGLD chain in a large transformer: if the resulting Lipschitz constant $M$ is not below $\\gamma/(n\\beta)$ (about 30 under the paper's Pythia-style settings), the central hypothesis $Mn\\beta<\\gamma$ fails and the theorem's conclusion does not apply.","supporting_citations":[{"cited_title":"Algebraic Geometry and Statistical Learning Theory","cited_arxiv_id":null,"evidence_quote":"Supplies the singular learning theory definition of the learning coefficient and the free-energy expansion the estimator targets."},{"cited_title":"A Widely Applicable Bayesian Information Criterion","cited_arxiv_id":null,"evidence_quote":"Provides the inverse-temperature scaling result that justifies the LLC estimator as a valid estimate of the learning coefficient."},{"cited_title":"and Teh, Y","cited_arxiv_id":null,"evidence_quote":"Introduces the stochastic gradient Langevin dynamics sampler used to approximate the local posterior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Glivenko\\u2013Cantelli uniform convergence results used to pass from empirical to population losses in Lemma 7.1."},{"cited_title":"Information Geometry and Its Applications: Convex Function and Dually Flat Manifold, pp.\\ 75--102","cited_arxiv_id":null,"evidence_quote":"Gives the uniqueness of the KL projection used to define the effective true distribution $q^{(\\chi)}$."}],"review_version":1}