{"id":"88a99bf1-2544-49c6-b216-aab67878bcb5","arxiv_id":"2506.05136","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Neural LMs learn languages with lower m-local entropy more easily, suggesting a shared sensitivity to local statistical structure with human learners.","lead":"The paper introduces m-local entropy, a measure of how well the previous m-1 words predict the next word, and shows that languages with higher m-local entropy are harder for neural language models to learn. It gives researchers a quantitative way to connect the inductive biases of AI language models to the local statistical constraints thought to shape human language processing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Controlled PFSA experiment lacks significance testing; many Table 3 correlations are near zero and may not survive FDR correction, so the 'clear positive correlation' claim is unsubstantiated.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and the rationale already notes the absence of error bars and significance testing. However, the reader's weakest_assumption focuses on the n-gram estimator validation gap in Experiment 1. I agree that gap is real, but the more load-bearing issue for the central claim is the statistical reliability of Experiment 2, the only experiment where m-local entropy is computed exactly and global entropy is controlled via DKL. If the Table 3 correlations are mostly compatible with zero after proper significance testing, then the paper's headline claim loses its strongest controlled support, and the natural-language experiment alone cannot carry it because of the estimator concern. I do not recommend REJECT because the direction is consistent across all 45 cells, and the paper honestly labels its analysis as correlational. The correct action is to require proper statistical reporting and tempering of the 'clear positive correlation' language, which is exactly what the CONDITIONAL verdict demands. My concrete test would settle whether the controlled correlations are reliably nonzero, thereby either validating or undermining the strongest form of the central claim.","tokens_in":18377,"tokens_out":15254,"duration_ms":196637,"concrete_test":"For each cell in Table 3, compute the exact p-value of the Pearson correlation and apply Benjamini-Hochberg FDR correction; in addition, run a permutation test that respects the 5x5 topology-by-weighting design (e.g., permute m-local entropy values within each topology or weighting, or fit a mixed-effects model with random intercepts for topology and weighting). Report the number and percentage of cells that remain significant after correction. If a large share do not survive, the 'clear positive correlation' claim in Section 4.2 is not supported by the controlled experiment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central controlled evidence (Experiment 2) is summarized by Pearson correlations in Table 3, but no significance tests, confidence intervals, or error bars are reported. Within each (|Q|, |Sigma|) cell, the 25 PFSAs are generated as the full cross of 5 topologies and 5 weightings, so the effective number of independent replicates is smaller than 25. Several coefficients are near zero: e.g., LSTM with |Sigma|=64 has r = 0.12-0.23 across all m; LSTM with |Q|=24, |Sigma|=64 has r about 0.12; Transformer with |Q|=24, |Sigma|=32, m=2 has r = 0.29. With n = 25 (or effectively less, given the factorial structure), such values are not significantly different from zero. The paper's statement in Section 4.2 that there is 'a clear positive correlation across both architectures and all values of m = 2, 3, 4, 5' is therefore not established by the reported statistics. The uniform positive sign across all 45 cells is suggestive, but the effect magnitude is weak in important regimes, and the controlled experiment does not quantify the uncertainty in the correlation estimates. This is load-bearing because Experiment 2 is the setting where m-local entropy is computed exactly and global entropy is controlled; if its correlations are not reliably positive, the central claim rests mainly on Experiment 1, whose estimator is itself validated only on small alphabets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces m-local entropy, an information-theoretic measure of a language's local predictability derived from lossy-context surprisal, and studies whether this measure predicts the learnability of probabilistic languages by neural language models. In Experiment 1, the authors apply bijective, length-preserving perturbations to a natural language corpus (BLLIP) to create counterfactual corpora with the same global and next-symbol entropy but different m-local entropy, and they train LSTM and Transformer LMs on them, finding strong positive correlations between estimated m-local entropy and next-symbol cross-entropy. In Experiment 2, they generate random PFSAs and compute exact m-local entropy and KL divergence, again reporting positive correlations. The authors interpret these findings as evidence that neural LMs share an inductive bias toward information locality with human language processing.","tokens_in":18574,"tokens_out":8324,"duration_ms":95217,"significance":"The paper makes a valuable conceptual contribution by proposing a quantifiable, theoretically grounded measure—m-local entropy—and testing it with carefully designed perturbations that control global entropy. The use of bijective length-preserving transformations is elegant, and the PFSA experiments allow exact computation, going beyond purely qualitative perturbation studies. The manuscript includes machine-checkable closed-form derivations for PFSA quantities and releases code, which strengthens reproducibility. If the reported correlations are robust, the finding would meaningfully connect information-theoretic characterizations of language with neural LM inductive biases. However, the strength of the evidence is currently overstated by the absence of inferential statistics in the controlled experiment and the unvalidated large-vocabulary estimator in the natural-language experiment.","major_comments":[{"comment":"The claim of a 'clear positive correlation between m-local entropy and DKL across both architectures and all values of m = 2, 3, 4, 5' is not supported by the reported statistics. The table gives Pearson correlation coefficients but no p-values, confidence intervals, or other uncertainty measures. With n = 25 PFSAs per cell, and with an effective sample size smaller than 25 due to the 5×5 factorial construction (5 topologies × 5 weightings), coefficients such as r = 0.119–0.137 for the LSTM at |Σ| = 64, or r ≈ 0.12 for the LSTM with (|Q|, |Σ|) = (24, 64), are not significantly different from zero. The uniform positive sign across 45 configurations is suggestive, but a sign test or meta-analytic summary is needed to quantify the evidence. As written, the central claim of Experiment 2—the setting that controls global entropy and computes m-local entropy exactly—rests on an unquantified pattern of small positive coefficients.","section":"Section 4.2, Table 3"},{"comment":"The n-gram estimator of m-local entropy is validated only on PFSAs with alphabets of size up to 48, yet it is applied to the BLLIP corpus, whose vocabulary is several orders of magnitude larger. At m = 4 and m = 5, the estimator's error on small alphabets is already non-negligible (1.49% and 7.61% MRE at 200K strings), and data sparsity in a large-vocabulary setting will be far worse; the reported R² ≈ 0.92 in Experiment 1 could be inflated or distorted by estimator bias. The authors should provide validation on a large-alphabet setting with known ground truth, or at least report bootstrap confidence intervals for the estimated m-local entropy values. Relatedly, the text in §3.2.1 states that the estimator achieves 'relative error below one percent up to m = 4,' but Table 1 reports MRE = 1.49% for m = 4 at 200K strings, contradicting that sentence.","section":"Section 3.2.1, Table 1"}],"minor_comments":[{"comment":"The sentence 'We term this fixed-context uncertainty measure m-local local entropy' contains a duplicated word; it should be 'm-local entropy.'","section":"Section 2.2.2"},{"comment":"The summation notation in the proofs is garbled (e.g., 'w (qi yk/w − − − →q′ ∈ δ, 33g)'); the expressions should be rewritten as explicit sums over transitions with weight w.","section":"Appendix A, Eqs. (33g) and (37i)"},{"comment":"Consider adding p-values or significance stars to the correlation coefficients, and note in the caption that all values are positive but not all are individually significant given the small and structured sample.","section":"Table 3 caption"},{"comment":"Please report the vocabulary size of the BLLIP-SM corpus used in the experiment, so readers can assess the sparsity of the n-gram counts for m = 4 and m = 5.","section":"Section 3.2"},{"comment":"When defining DKL, it would be helpful to explicitly state that the next-symbol entropy H(Y | →Y) is computed exactly from the PFSA rather than estimated from the sample.","section":"Section 4.1.2"},{"comment":"The transition-weight initialization is hard to parse: clarify that the 0.001 term is a smoothing constant added to every entry and how it interacts with the exponential weights.","section":"Appendix B, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious contribution with a clean experimental framework and strong reproducibility practices. The main risk is statistical overstatement: the controlled PFSA experiment needs proper significance testing or confidence intervals, and the large-vocabulary estimator needs validation or bias diagnostics. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper has a genuinely useful idea—isolate m-local entropy as a quantitative predictor of neural LM learnability while controlling for global entropy—and the natural-language experiment is clean and compelling. But the PFSA experiment, which is supposed to be the controlled confirmation, reports correlations without any significance testing, and several of those correlations are close to zero. The 'clear positive correlation' claim in Section 4.2 is not backed by the numbers as reported.\n\nWhat's new: m-local entropy itself is old—it's average m-gram surprisal under a fixed window—but the paper's contribution is the controlled framework. The bijective length-preserving perturbations are a nice trick: they guarantee equal global and next-symbol entropy while varying local predictability. The PFSA closed-form computations are careful and the code is released. The limitations section honestly states the correlational nature.\n\nWhere it gets soft: Table 3 has no confidence intervals or p-values. In several cells, particularly LSTM with |Sigma|=64, r is around 0.12–0.23; with n=25 (and effectively fewer independent replicates, given the 5x5 topology-weighting structure) those are not distinguishable from zero. The uniform positive sign across all 45 cells is suggestive, but it doesn't establish a 'clear' correlation in every configuration. The estimator validation in Table 1 is only on alphabets up to 48 symbols, while the English experiment uses vocabularies of tens of thousands; that gap is real but probably minor given the 5M-token corpus.\n\nMy take: the central direction is likely correct—neural LMs do seem sensitive to local predictability, and the natural-language result is strong. But the paper overstates what the PFSA experiment shows. A revision that adds significance tests or bootstrapped intervals, and moderates the Section 4.2 wording, would make this a solid contribution. It deserves peer review; it shouldn't be desk-rejected. I'd bring it to reading group.","headline":"Solid controlled study with an overstated headline; the PFSA correlations need significance tests before the 'clear' claim holds.","tokens_in":19181,"tokens_out":3315,"would_cite":true,"duration_ms":35698,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68Q45","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes m-local entropy as a predictor of neural language model learnability and shows that higher local uncertainty makes Transformer and LSTM language models perform worse, even when global entropy is held fixed.","keywords":["m-local entropy","inductive bias","neural language models","information locality","lossy-context surprisal","probabilistic finite-state automata","learnability","Transformer and LSTM"],"falsifier":"Construct two corpora (or two PFSA families) that match in estimated m-local entropy for all m = 2,3,4,5 and in global entropy, but differ in other structural properties, and show they yield clearly different LM cross-entropy or KL divergence; that would break the claim that m-local entropy is the operative predictor. A complementary check is to recompute the natural-language correlation with a much larger held-out sample or a different estimator and see whether the trend survives.","tokens_in":18128,"feed_emoji":"🧠","tokens_out":7703,"duration_ms":81375,"temperature":0.7,"pith_summary":"Neural language models are usually judged by whether they learn a corpus, not by which statistical properties make a language easy or hard. This paper argues that one such property is local predictability, formalized as m-local entropy—the average uncertainty of the next symbol given only the preceding m-1 symbols. In two experiments, one on English corpora and one on synthetic languages generated by probabilistic finite-state automata, Transformer and LSTM language models perform worse as m-local entropy rises, even when global entropy (a standard yardstick of language complexity) is held constant. If the claim holds, local statistical structure is a genuine inductive bias shared by neural models and human language processors.","feed_headline":"Higher local entropy makes languages harder for neural LMs","feed_subtitle":"Transformer and LSTM models both struggle more as m-local entropy rises, even when global entropy is fixed.","key_machinery":"m-local entropy $H(Y | C)$ is the expected entropy of the next symbol $Y$ given a random context $C$ of length $m-1$, where contexts are weighted by their normalized infix probability in the language. It is the average lossy-context surprisal obtained when the memory representation keeps only the immediately preceding $m-1$ symbols. Crucially, unlike global entropy and next-symbol entropy, this quantity changes under the length-preserving, bijective perturbations used in Experiment 1, so it isolates local structure. The machinery also includes closed-form matrix computations for PFSAs that give the true m-local entropy, and an n-gram estimator (KenLM) used for natural language corpora.","core_discovery":"The paper's central claim is that neural LMs have an inductive bias favoring information locality: languages in which the near context strongly disambiguates the next symbol are systematically easier to learn. This is measured with m-local entropy, an expectation over contexts of the entropy of the next-symbol distribution after observing the last m-1 symbols. For natural language, length-preserving bijective perturbations create corpora with the same global and next-symbol entropy but different m-local entropy, and the estimated next-symbol cross-entropy of LSTM and Transformer models rises with m-local entropy. For PFSAs, where m-local entropy is computed exactly, the symbol-level KL divergence between the model and the target language correlates positively with m-local entropy for m = 2,3,4,5. The paper interprets this as evidence that neural LMs, like humans, are sensitive to the local statistical structure of a language.","pith_inferences":["A testable extension is to use m-local entropy as a regularizer during training: penalizing high local entropy may push models toward human-like processing, but the paper does not itself train with such a penalty.","If local entropy is causally linked to difficulty, then comparing two natural languages with similar global entropy but very different local entropy should predict which one a given neural LM learns faster—an experiment the paper does not run.","The shared sensitivity between LMs and humans raises a follow-up question: whether m-local entropy also predicts human reading times beyond the lossy-context surprisal literature, which would connect the result to psycholinguistic data.","The estimator validation gap suggests an easy stress test: compute m-local entropy on a large-vocabulary corpus with alternative estimators (e.g., neural n-gram models) and check whether the correlation with LM performance remains stable."],"forward_implications":["Languages with lower m-local entropy should be learnable with less data and fewer updates, giving a quantitative target for curriculum learning or training-set selection.","Perturbation experiments can now be reasoned about on a continuous axis: local shuffles produce a graded continuum of difficulty rather than a binary human-like versus non-human-like split.","Because global entropy is controlled, differences in LM performance across languages can be attributed to local structure, making m-local entropy a useful control variable in cross-lingual learnability studies.","The PFSA setup gives a test bed for other inductive biases, since any candidate measure that can be computed exactly on automata can be checked against the same controlled learning curves."],"supporting_citations":[{"why":"Provides the lossy-context surprisal theory from which m-local entropy is derived as the finite-context special case.","marker":"Futrell et al. (2020)"},{"why":"Establishes information-theoretic locality properties of natural language that motivate the measure's focus on local predictability.","marker":"Futrell (2019)"},{"why":"The prior perturbation study whose transformations this paper refines into a controlled entropy continuum.","marker":"Kallini et al. (2024)"},{"why":"Supplies the dependency locality theory framing why local distance between highly informative elements shapes processing difficulty.","marker":"Gibson (2001)"},{"why":"Baseline showing which probabilistic regular languages are easy for neural LMs, the PFSA testbed this paper extends.","marker":"Borenstein et al. (2024)"},{"why":"KenLM is the n-gram estimator used to compute m-local entropy on natural-language corpora.","marker":"Heafield (2011)"},{"why":"The theorem connecting next-symbol entropy to global entropy justifies using next-symbol cross-entropy while controlling global entropy.","marker":"Malagutti et al. (2024)"}],"fun_headline_variants":["Local entropy predicts LM difficulty","Neural LMs prefer local context","Information locality shapes LM learning","High m-local entropy stumps Transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the n-gram estimate of m-local entropy, validated on small synthetic alphabets, remains accurate for English corpora with vocabularies of tens of thousands of word types; if sparse-data noise dominates there, the observed correlation could be an artifact of estimation error rather than a true property of the languages.","fun_headline_variants_meta":{"raw":{"variants":["Local entropy predicts LM difficulty","Neural LMs prefer local context","Information locality shapes LM learning","High m-local entropy stumps Transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1192,"prompt_tokens":897,"completion_tokens":295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":513,"tokens_out":295,"duration_ms":3893,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:23:24.400599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct two corpora (or two PFSA families) that match in estimated m-local entropy for all m = 2,3,4,5 and in global entropy, but differ in other structural properties, and show they yield clearly different LM cross-entropy or KL divergence; that would break the claim that m-local entropy is the operative predictor. A complementary check is to recompute the natural-language correlation with a much larger held-out sample or a different estimator and see whether the trend survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KenLM is the n-gram estimator used to compute m-local entropy on natural-language corpora."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The theorem connecting next-symbol entropy to global entropy justifies using next-symbol cross-entropy while controlling global entropy."}],"review_version":1}