Pith. sign in

REVIEW 2 major objections 6 minor 38 references

Information Locality as an Inductive Bias for Neural Language Models

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes m-local entropy as a predictor of neural language model learnability and shows that higher local uncertainty makes Transformer and LSTM language models perform worse, even when global entropy is held fixed.

desk verdict Solid controlled study with an overstated headline; the PFSA correlations need significance tests before the 'clear' claim holds. read the letter →

arxiv 2506.05136 v1 pith:734RPJ35 submitted 2025-06-05 cs.CL

Taiga Someya , Anej Svete , Brian DuSell , Timothy J. O'Donnell , Mario Giulianelli , Ryan Cotterell This is my paper
classification cs.CL MSC 68T5068Q4594A17
keywords m-localentropyinductivebiasneurallanguagemodelsinformationlocalitylossy-contextsurprisalprobabilisticfinite-stateautomatalearnabilityTransformerandLSTM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neural language models are usually judged by whether they learn a corpus, not by which statistical properties make a language easy or hard. This paper argues that one such property is local predictability, formalized as m-local entropy—the average uncertainty of the next symbol given only the preceding m-1 symbols. In two experiments, one on English corpora and one on synthetic languages generated by probabilistic finite-state automata, Transformer and LSTM language models perform worse as m-local entropy rises, even when global entropy (a standard yardstick of language complexity) is held constant. If the claim holds, local statistical structure is a genuine inductive bias shared by neural models and human language processors.

What carries the argument

m-local entropy $H(Y | C)$ is the expected entropy of the next symbol $Y$ given a random context $C$ of length $m-1$, where contexts are weighted by their normalized infix probability in the language. It is the average lossy-context surprisal obtained when the memory representation keeps only the immediately preceding $m-1$ symbols. Crucially, unlike global entropy and next-symbol entropy, this quantity changes under the length-preserving, bijective perturbations used in Experiment 1, so it isolates local structure. The machinery also includes closed-form matrix computations for PFSAs that give the true m-local entropy, and an n-gram estimator (KenLM) used for natural language corpora.

What would settle it

Construct two corpora (or two PFSA families) that match in estimated m-local entropy for all m = 2,3,4,5 and in global entropy, but differ in other structural properties, and show they yield clearly different LM cross-entropy or KL divergence; that would break the claim that m-local entropy is the operative predictor. A complementary check is to recompute the natural-language correlation with a much larger held-out sample or a different estimator and see whether the trend survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that neural LMs have an inductive bias favoring information locality: languages in which the near context strongly disambiguates the next symbol are systematically easier to learn. This is measured with m-local entropy, an expectation over contexts of the entropy of the next-symbol distribution after observing the last m-1 symbols. For natural language, length-preserving bijective perturbations create corpora with the same global and next-symbol entropy but different m-local entropy, and the estimated next-symbol cross-entropy of LSTM and Transformer models rises with m-local entropy. For PFSAs, where m-local entropy is computed exactly, the symbol-level KL divergence between the model and the target language correlates positively with m-local entropy for m = 2,3,4,5. The paper interprets this as evidence that neural LMs, like humans, are sensitive to the local statistical structure of a language.

Load-bearing premise

The load-bearing premise is that the n-gram estimate of m-local entropy, validated on small synthetic alphabets, remains accurate for English corpora with vocabularies of tens of thousands of word types; if sparse-data noise dominates there, the observed correlation could be an artifact of estimation error rather than a true property of the languages.

Editorial extensions

If this is right

  • Languages with lower m-local entropy should be learnable with less data and fewer updates, giving a quantitative target for curriculum learning or training-set selection.
  • Perturbation experiments can now be reasoned about on a continuous axis: local shuffles produce a graded continuum of difficulty rather than a binary human-like versus non-human-like split.
  • Because global entropy is controlled, differences in LM performance across languages can be attributed to local structure, making m-local entropy a useful control variable in cross-lingual learnability studies.
  • The PFSA setup gives a test bed for other inductive biases, since any candidate measure that can be computed exactly on automata can be checked against the same controlled learning curves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to use m-local entropy as a regularizer during training: penalizing high local entropy may push models toward human-like processing, but the paper does not itself train with such a penalty.
  • If local entropy is causally linked to difficulty, then comparing two natural languages with similar global entropy but very different local entropy should predict which one a given neural LM learns faster—an experiment the paper does not run.
  • The shared sensitivity between LMs and humans raises a follow-up question: whether m-local entropy also predicts human reading times beyond the lossy-context surprisal literature, which would connect the result to psycholinguistic data.
  • The estimator validation gap suggests an easy stress test: compute m-local entropy on a large-vocabulary corpus with alternative estimators (e.g., neural n-gram models) and check whether the correlation with LM performance remains stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces m-local entropy, an information-theoretic measure of a language's local predictability derived from lossy-context surprisal, and studies whether this measure predicts the learnability of probabilistic languages by neural language models. In Experiment 1, the authors apply bijective, length-preserving perturbations to a natural language corpus (BLLIP) to create counterfactual corpora with the same global and next-symbol entropy but different m-local entropy, and they train LSTM and Transformer LMs on them, finding strong positive correlations between estimated m-local entropy and next-symbol cross-entropy. In Experiment 2, they generate random PFSAs and compute exact m-local entropy and KL divergence, again reporting positive correlations. The authors interpret these findings as evidence that neural LMs share an inductive bias toward information locality with human language processing.

Significance. The paper makes a valuable conceptual contribution by proposing a quantifiable, theoretically grounded measure—m-local entropy—and testing it with carefully designed perturbations that control global entropy. The use of bijective length-preserving transformations is elegant, and the PFSA experiments allow exact computation, going beyond purely qualitative perturbation studies. The manuscript includes machine-checkable closed-form derivations for PFSA quantities and releases code, which strengthens reproducibility. If the reported correlations are robust, the finding would meaningfully connect information-theoretic characterizations of language with neural LM inductive biases. However, the strength of the evidence is currently overstated by the absence of inferential statistics in the controlled experiment and the unvalidated large-vocabulary estimator in the natural-language experiment.

major comments (2)
  1. [Section 4.2, Table 3] The claim of a 'clear positive correlation between m-local entropy and DKL across both architectures and all values of m = 2, 3, 4, 5' is not supported by the reported statistics. The table gives Pearson correlation coefficients but no p-values, confidence intervals, or other uncertainty measures. With n = 25 PFSAs per cell, and with an effective sample size smaller than 25 due to the 5×5 factorial construction (5 topologies × 5 weightings), coefficients such as r = 0.119–0.137 for the LSTM at |Σ| = 64, or r ≈ 0.12 for the LSTM with (|Q|, |Σ|) = (24, 64), are not significantly different from zero. The uniform positive sign across 45 configurations is suggestive, but a sign test or meta-analytic summary is needed to quantify the evidence. As written, the central claim of Experiment 2—the setting that controls global entropy and computes m-local entropy exactly—rests on an unquantified pattern of small positive coefficients.
  2. [Section 3.2.1, Table 1] The n-gram estimator of m-local entropy is validated only on PFSAs with alphabets of size up to 48, yet it is applied to the BLLIP corpus, whose vocabulary is several orders of magnitude larger. At m = 4 and m = 5, the estimator's error on small alphabets is already non-negligible (1.49% and 7.61% MRE at 200K strings), and data sparsity in a large-vocabulary setting will be far worse; the reported R² ≈ 0.92 in Experiment 1 could be inflated or distorted by estimator bias. The authors should provide validation on a large-alphabet setting with known ground truth, or at least report bootstrap confidence intervals for the estimated m-local entropy values. Relatedly, the text in §3.2.1 states that the estimator achieves 'relative error below one percent up to m = 4,' but Table 1 reports MRE = 1.49% for m = 4 at 200K strings, contradicting that sentence.
minor comments (6)
  1. [Section 2.2.2] The sentence 'We term this fixed-context uncertainty measure m-local local entropy' contains a duplicated word; it should be 'm-local entropy.'
  2. [Appendix A, Eqs. (33g) and (37i)] The summation notation in the proofs is garbled (e.g., 'w (qi yk/w − − − →q′ ∈ δ, 33g)'); the expressions should be rewritten as explicit sums over transitions with weight w.
  3. [Table 3 caption] Consider adding p-values or significance stars to the correlation coefficients, and note in the caption that all values are positive but not all are individually significant given the small and structured sample.
  4. [Section 3.2] Please report the vocabulary size of the BLLIP-SM corpus used in the experiment, so readers can assess the sparsity of the n-gram counts for m = 4 and m = 5.
  5. [Section 4.1.2] When defining DKL, it would be helpful to explicitly state that the next-symbol entropy H(Y | →Y) is computed exactly from the PFSA rather than estimated from the sample.
  6. [Appendix B, Algorithm 1] The transition-weight initialization is hard to parse: clarify that the 0.001 term is a smoothing constant added to every entry and how it interacts with the exponential weights.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: m-local entropy is defined independently of trained LM difficulty, and the reported correlations are empirical measurements rather than fits.

full rationale

The central claim is an empirical correlation between a newly defined information measure, m-local entropy, and neural LM cross-entropy or KL divergence. The measure is defined in Eqs. (14)-(15) directly from the underlying probabilistic language model p, not from the trained LSTM or Transformer q. In Experiment 1, m-local entropy is estimated from corpora with an n-gram model (Eqs. 17-19) that is not fit to the neural LMs' outputs, and the perturbation design preserves global entropy by construction; in Experiment 2, m-local entropy is computed analytically for PFSAs (Lemma A.6) and compared with measured bDKL. No equation or fitted parameter is transformed into the quantity it is said to predict. The only author-overlap citation in the derivation chain is Eq. (13d), which attributes to Malagutti et al. (2024, Thm. 2.2) the proportionality between next-symbol entropy and global entropy; this is a parameter-free mathematical identity used for experimental control and does not assume the empirical correlation between m-local entropy and learnability. The paper's own Limitations section states that the analysis is correlational and that PFSAs are limited; these are epistemic caveats, not circularity. The limited validation of the n-gram estimator on small alphabets (Table 1) is a statistical robustness concern, not a definitional identity. No load-bearing step reduces to its own inputs, so the score reflects only the minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim does not depend on any fitted numeric parameter. The experimental design uses hand-chosen PFSA generation constants (e.g., Exp(0.1), 0.001 floor, smin=2) that shape the range of languages, but these do not enter the hypothesis being tested. The main burdens are domain assumptions about the validity of m-local entropy as the operative measure and about the n-gram estimator for natural language.

assumptions (4)
  • standard math The language model p has finite expected string length mu = E|Y| < infinity and finite entropy H(Y) < infinity, guaranteeing normalizability of prefix probabilities and the identities used in Eq. (13).
    Needed for the definition of next-symbol entropy and the proportionality H(Y|Y^<-)=H(Y)/(mu+1).
  • domain assumption m-local entropy as defined in Eq. (15) is a faithful operationalization of 'information locality' from lossy-context surprisal theory (Futrell et al., 2020).
    The paper connects its measure to a cognitive theory, but the link is conceptual, not proven; this is one specific choice among many possible locality metrics.
  • domain assumption Bijective length-preserving perturbations preserve global entropy exactly, so differences in LM performance across perturbed corpora are not attributable to global entropy differences.
    Relies on entropy invariance under bijections, which holds for the underlying distribution and is applied to the empirical corpus.
  • domain assumption The n-gram model used to estimate m-local entropy provides a faithful estimate for natural language corpora, extrapolating from the PFSA validation in Table 1.
    Validation is only on small alphabets (up to 48 symbols); large-vocabulary natural language may have different estimation error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Information Locality as an Inductive Bias for Neural Language Models." pith.science (2026). https://pith.science/paper/734RPJ35

@misc{pith2026250605136,
  author       = {Pith},
  title        = {Pith review of: Information Locality as an Inductive Bias for Neural Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/734RPJ35}},
  note         = {Machine review of arXiv:2506.05136}
}
abstract

Inductive biases are inherent in every machine learning system, shaping how models generalize from finite data. In the case of neural language models (LMs), debates persist as to whether these biases align with or diverge from human processing constraints. To address this issue, we propose a quantitative framework that allows for controlled investigations into the nature of these biases. Within our framework, we introduce $m$-local entropy$\unicode{x2013}$an information-theoretic measure derived from average lossy-context surprisal$\unicode{x2013}$that captures the local uncertainty of a language by quantifying how effectively the $m-1$ preceding symbols disambiguate the next symbol. In experiments on both perturbed natural language corpora and languages defined by probabilistic finite-state automata (PFSAs), we show that languages with higher $m$-local entropy are more difficult for Transformer and LSTM LMs to learn. These results suggest that neural LMs, much like humans, are highly sensitive to the local statistical structure of a language.

Figures

Figures reproduced from arXiv: 2506.05136 by the authors.

Figure 1
Figure 1. KL divergence (Transformer LM) as a func [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Scatter plots of next-symbol cross-entropy ( [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Scatter plots of symbol-level KL divergence (y-axis) versus [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Scatter plots of next-symbol cross-entropy (y-axis) versus [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers

    Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, and Yulia Tsvetkov. 2024. https://api.semanticscholar.org/CorpusID:269362467 Learning syntax without planting trees: Understanding when and why transformers generalize hierarchically . ArXiv, abs/2404.16367

    work page Pith review arXiv 2024

  4. [4]

    Nadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, and Ryan Cotterell. 2024. https://arxiv.org/abs/2406.04289 What languages are easy to language-model? A perspective from learning probabilistic regular languages . Preprint, arXiv:2406.04289

    arXiv 2024

  5. [5]

    Alexandra Butoi, Ghazal Khalighinejad, Anej Svete, Josef Valvoda, Ryan Cotterell, and Brian DuSell. 2025. https://openreview.net/forum?id=aWLQTbfFgV Training neural networks as recognizers of formal languages . In The Thirteenth International Conference on Learning Representations

    2025

  6. [6]

    Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. https://doi.org/10.35111/FWEW-DA58 BLLIP 1987-89 WSJ Corpus Release 1

    doi:10.35111/fwew-da58 2000

  7. [7]

    Noam Chomsky. 1957. https://doi.org/10.1515/9783112316009 Syntactic Structures

    doi:10.1515/9783112316009 1957

  8. [8]

    Noam Chomsky, Ian Roberts, and Jeffrey Watumull. 2023. https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html Noam chomsky: T he false promise of ChatGPT . The New York Times

    work page 2023

Show all 38 references
  1. [9]

    Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang

    Leshem Choshen, Ryan Cotterell, Michael Y. Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. https://arxiv.org/abs/2404.06214 T he 2nd BabyLM challenge: S ample-efficient pretraining on a developmentally plausib...

    2024 arXiv

  2. [10]

    Andrea De Varda and Marco Marelli. 2024. https://doi.org/10.18653/v1/2024.cmcl-1.3 Locally biased transformers better align with human reading times . In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 30--36, Bangkok, Thailand. Associati...

    2024 doi

  3. [11]

    Everaert, Marinus A.C

    Martin B.H. Everaert, Marinus A.C. Huybregts, Noam Chomsky, Robert C. Berwick, and Johan J. Bolhuis. 2015. https://doi.org/10.1016/j.tics.2015.09.008 Structures, not strings: Linguistics as part of the cognitive sciences . Trends in Cognitive Sciences, 19(12):729--743

    2015 doi

  4. [12]

    Richard Futrell. 2019. https://doi.org/10.18653/v1/W19-7902 Information-theoretic locality properties of natural language . In Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019), pages 2--15, Paris, France. Association for Computational Linguistics

    2019 doi

  5. [13]

    Richard Futrell. 2023. https://api.semanticscholar.org/CorpusID:262070319 Information-theoretic principles in incremental language production . Proceedings of the National Academy of Sciences of the United States of America, 120

    2023

  6. [14]

    Richard Futrell, Edward Gibson, and Roger P. Levy. 2020. https://doi.org/10.1111/cogs.12814 Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing . Cognitive Science, 44(3):e12814

    2020 doi

  7. [15]

    Richard Futrell and Michael Hahn. 2024. https://api.semanticscholar.org/CorpusID:269921516 Linguistic structure from a bottleneck on sequential information processing . ArXiv, abs/2405.12109

    2024

  8. [16]

    Edward Gibson. 1998. https://doi.org/10.1016/S0010-0277(98)00034-1 Linguistic complexity: L ocality of syntactic dependencies . Cognition, 68(1):1--76

    1998 doi

  9. [17]

    Edward Gibson. 2001. https://doi.org/10.7551/mitpress/3654.003.0008 The dependency locality theory: A distance-based theory of linguistic complexity . In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium. The MIT Press

    2001 doi

  10. [18]

    Michael Hahn, Judith Degen, and Richard Futrell. 2021. https://doi.org/10.1037/rev0000269 Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal . Psychological Review, 128(4):726--756

    2021 doi

  11. [19]

    Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022. https://doi.org/10.1073/pnas.2122602119 A resource-rational model of human processing of recursive linguistic structure . Proceedings of the National Academy of Sciences, 119(43):e2122602119

    2022 doi

  12. [20]

    John Hale. 2001. https://aclanthology.org/N01-1021/ A probabilistic E arley parser as a psycholinguistic model . In Second Meeting of the North A merican Chapter of the Association for Computational Linguistics

    2001

  13. [21]

    Kenneth Heafield. 2011. https://www.aclweb.org/anthology/W11-2123 K en LM : Faster and smaller language model queries . In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187--197, Edinburgh, Scotland. Association for Computational Linguistics

    2011

  14. [22]

    Sepp Hochreiter and J\" u rgen Schmidhuber. 1997. https://doi.org/10.1162/neco.1997.9.8.1735 Long short-term memory . Neural Comput., 9(8):1735–1780

    1997 doi

  15. [23]

    Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. https://doi.org/10.18653/v1/2020.acl-main.158 A systematic assessment of syntactic generalization in neural language models . In Proceedings of the 58th Annual Meeting of the Association for Computationa...

    2020 doi

  16. [24]

    Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. https://doi.org/10.18653/v1/2024.acl-long.787 Mission: Impossible language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...

    2024 doi

  17. [25]

    Kingma and Jimmy Lei Ba

    Diederik P. Kingma and Jimmy Lei Ba. 2015. https://arxiv.org/abs/1412.6980 A dam: A method for stochastic optimization . In The Third International Conference for Learning Representations, San Diego, California, USA

    2015 arXiv

  18. [26]

    Roger Levy. 2008. https://doi.org/10.1016/j.cognition.2007.05.006 Expectation-based syntactic comprehension . Cognition, 106(3)

    2008 doi

  19. [27]

    Luca Malagutti, Andrius Buinovskij, Anej Svete, Clara Meister, Afra Amini, and Ryan Cotterell. 2024. https://doi.org/10.18653/v1/2024.naacl-long.382 The role of n -gram smoothing in the age of neural networks . In Proceedings of the 2024 Conference of the North American Chapte...

    2024 doi

  20. [28]

    Thomas McCoy, Robert Frank, and Tal Linzen

    R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. https://doi.org/10.1162/tacl_a_00304 Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks . Transactions of the Association for Computational Linguistics, 8:125--140

    2020 doi

  21. [29]

    Mitchell

    Tom M. Mitchell. 1980. https://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/pubs/NeedForBias_1980.pdf The need for biases in learning generalizations . Technical Report CBM-TR 5-110, Rutgers University, New Brunswick, New Jersey, USA

    1980

  22. [30]

    Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, and Ethan Wilcox. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.179 On the role of context in reading time prediction . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3042--...

    2024 doi

  23. [31]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  24. [32]

    Steven Piantadosi. 2024. https://lingbuzz.net/lingbuzz/007180 Modern language models refute Chomsky ’s approach to language . LingBuzz Published In: Edward Gibson & Moshe Poliak (eds.), From fieldwork to linguistic theory: A tribute to Dan Everett (Empirically Oriented Theoret...

    2024 doi

  25. [33]

    Jonathan Rawski and Jeffrey Heinz. 2019. https://api.semanticscholar.org/CorpusID:267932762 No free lunch in linguistics or machine learning: Response to pater . Language, 95:e125 -- e135

    2019

  26. [34]

    C. E. Shannon. 1948. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x A mathematical theory of communication . The Bell System Technical Journal, 27(3):379--423

    1948

  27. [35]

    William Timkey and Tal Linzen. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.582 A language model with limited memory capacity captures interference in human sentence processing . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705--872...

    2023 doi

  28. [36]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neura...

    2017

  29. [37]

    Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. https://doi.org/10.18653/v1/P18-2117 On the practical computational power of finite precision RNN s for language recognition . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2:...

    2018 doi

  30. [38]

    Thomas McCoy

    Aditya Yedetore, Tal Linzen, Robert Frank, and R. Thomas McCoy. 2023. https://doi.org/10.18653/v1/2023.acl-long.521 How poor is the stimulus? E valuating hierarchical generalization in neural networks trained on child-directed speech . In Proceedings of the 61st Annual Meeting...

    2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.