Pith. sign in

REVIEW 4 major objections 5 minor 67 references

Model Connectomes: A Generational Approach to Data-Efficient Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a binary 'model connectome', distilled from six generations of magnitude pruning on a 4-billion-token corpus, works as an efficient prior for learning language from just 100 million tokens.

desk verdict A clean controlled experiment showing a generationally pruned binary mask helps low-data LM training, but the 'data-efficient prior' framing overreaches without a data-agnostic pruning control. read the letter →

arxiv 2504.21047 v1 pith:5NHFCXN2 submitted 2025-04-29 cs.LG cs.AI

Klemen Kotar , Greta Tuckute This is my paper · ORCID
classification cs.LGcs.AI
keywords modelconnectomeiterativepruningsparseinitializationlow-datalanguagemodelinglotterytickethypothesisreading-timealignmentfMRInetworkdataefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Artificial language models are usually trained once from a random initialization on a huge corpus, while biological brains carry inherited wiring from evolution. This paper tries to close that gap by adding an outer 'evolution' loop: a model is trained on a 4-billion-token corpus, then pruned across six generations to a sparse binary mask that records only whether each remaining connection is excitatory or inhibitory. That mask, called a model connectome, is then used to initialize a GPT-2 that trains on only 100 million tokens. The connectome model beats a randomly masked control and a dense randomly initialized control on validation loss, HellaSwag, and MMLU, and matches or exceeds them on human reading-time and brain-response alignment. If the result holds, it suggests that a highly compressed wiring diagram can substitute for a large part of the data a language model would otherwise need.

What carries the argument

The central object is the model connectome: a binary weight mask that retains only the sign of each surviving connection, with positive weights set to +0.02 and negative to −0.02. It is produced by an outer loop that trains a GPT-2 on the large corpus, prunes the lowest 20% of weights by magnitude, resets survivors to constant signs, and repeats for six generations, leaving 25% of weights active. The resulting mask is then used to initialize a model that undergoes the inner learning loop on the small corpus. The machinery works by forcing the second phase of training to optimize only the wiring that survived the generational pruning, effectively carving out a subspace that makes language learning more data-efficient. The two-control design isolates the mask's contribution from the mere effect of sparsity.

What would settle it

Train the same six-generation pruning procedure on a large non-linguistic corpus (or on shuffled tokens) and use the resulting mask to initialize a model trained on the 100M-token language corpus; if that mask performs as well as the FineWeb-derived connectome, the prior is not language-specific. Alternatively, derive the connectome on one broad-domain corpus and test it on a small developmental corpus from a clearly different domain; if transfer disappears, the effect is domain-specific rather than a general efficiency prior.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a model connectome—a sparse binary mask distilled by six generations of 20% magnitude pruning on a 4B-token language corpus—serves as an efficient prior for learning in low-data regimes. When the final 25%-dense mask with constant ±0.02 weights is used to initialize a GPT-2 trained on a separate 100M-token corpus, it outperforms both a randomly sampled mask with the same sparsity and a standard dense initialization, and it approaches the validation loss of a dense model trained on the full 4B tokens. The same ordering holds on HellaSwag and MMLU, and the connectome model is better or on par with controls on reading-time and fMRI alignment. The paper interprets this as evidence that information from a large corpus can be transmitted across generations in a compressed wiring form, narrowing the gap between single-generation artificial models and biologically evolved neural networks.

Load-bearing premise

The load-bearing premise is that iterative magnitude pruning on a large language corpus produces a mask whose usefulness transfers to a different, smaller sample of language; the paper validates this only on two disjoint shards of the same web-text distribution.

Editorial extensions

If this is right

  • A 124M-parameter model with only 31M active weights can outperform a dense 124M model trained on the same 100M-token dataset, so sparsity combined with an inherited mask is more than a compression trick.
  • A 417M-parameter pruned model trained only on the small dataset reaches a validation loss comparable to a 124M dense model trained on the full 4B-token corpus, suggesting the approach scales in data efficiency.
  • Because the connectome model exceeds the random-mask control, the specific structure of the mask, not just its sparsity, carries the benefit.
  • Behavioral and neural evaluations show the connectome model stays aligned with human reading times and language-network fMRI responses at least as well as dense controls, so the efficiency gain does not obviously trade off against biological plausibility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The study leaves open whether the connectome is a general language prior or a FineWeb-specific prior, since the large and small corpora are disjoint shards of the same dataset; testing on an out-of-domain developmental corpus would separate these.
  • Because the connectome records only sign and not magnitude, it resembles a wiring diagram, which suggests a natural test: whether the same mask can be transferred across architectures or model sizes, or composed with quantization to push compression further.
  • The generational loop could be extended with mutation or recombination between multiple parent masks, and if those variants improve on single-lineage pruning, the analogy to evolution becomes more than a framing device.
  • The brain-alignment results on small 100M-token models suggest that data-efficient pretraining might be a lever for building models that are more neurally plausible, but direct comparisons across pretraining corpora would be needed to confirm.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a generational learning framework for language models. A GPT-2-style model is trained on a 4B-token corpus for six generations, with 20% magnitude pruning per generation and sign-only reinitialization, yielding a sparse ternary 'connectome' mask that retains 25% of the weights. This mask is then used to initialize a model trained on a 100M-token corpus, and the resulting Connectome model is compared against a RandomConnectome control (random ternary mask) and a NoConnectome dense control on FineWeb validation loss, HellaSwag, MMLU, reading-time alignment, and fMRI alignment. The authors report that Connectome outperforms or matches controls on most metrics, with an additional exploratory 417M-parameter scaling result. The central claim is that a model connectome serves as an efficient prior for learning in low-data regimes.

Significance. If the central claim is established, the work would be a useful demonstration that sparse, sign-only initializations distilled from a larger corpus can provide a strong inductive bias for low-resource language modeling. The main experimental comparison is cleanly designed: three models with identical architecture and inner-loop schedule, four seeds, publicly available evaluation benchmarks, and released code. The use of external behavioral and neural benchmarks that were not used to fit the connectome is a strength, as is the explicit separation of an outer evolutionary loop and an inner learning loop. However, the current evidence does not yet support the strongest interpretations: the claim that information from the large corpus is transmitted through the connectome lacks an appropriate data-agnostic pruning control, and the data-efficiency claims are complicated by the actual number of token presentations under the reported training schedules. The paper is honest about its limitations, but those limitations are broader than the text acknowledges.

major comments (4)
  1. [Section 2.1.3 and Section 3.1] The Connectome-vs-RandomConnectome comparison controls for sparsity and ternary sign initialization, but not for the fact that the Connectome mask is the output of a six-generation iterative-pruning optimization trajectory on L. The central claim is that information from L is transmitted through the connectome, yet no control runs the identical pruning pipeline on a non-linguistic dataset (e.g., shuffled tokens, random labels, or a non-linguistic sequence dataset) and uses the resulting mask to initialize training on S. Without such a control, the observed advantage could be attributed to iterative pruning itself producing a useful sparse initialization, independently of any language-specific content in L. This control is necessary to support the information-transmission interpretation in Sections 1 and 4.
  2. [Sections 2.1.3, 2.1.4, and Figure 1A] The paper reports a 4B-token outer loop and a 100M-token inner loop, but the stated training schedules imply far more token presentations. With the standard GPT-2 context length of 1024 tokens, 7,000 iterations at batch size 512 consume about 3.67B tokens per generation, or about 22B tokens across six generations, and the inner-loop 2,000 iterations consume about 1.05B tokens, which is roughly 10.5 passes over the 100M-token S dataset. If a shorter context length was used, the paper must state it; as written, the ratios in the abstract and in Section 3.1.1 conflate unique corpus tokens with total token presentations. The data-efficiency claim should be qualified to the number of unique tokens or stated explicitly in terms of compute/epochs, and the paper should clarify what the reported numbers mean.
  3. [Section 3.1.1] The 417M scaling result is presented as a single curve with no seeds, no error bars, and no matched controls: there is no 417M dense model trained on S, no 417M RandomConnectome control, and no 417M dense model trained on L. The claim that a large pruned model trained only on S matches a small dense model trained on L therefore conflates model scale with the connectome effect. This result is central to the 'data efficiency' conclusion in the same section, so it needs matched controls and seed variability before it can support that interpretation.
  4. [Figure 1B (panel ii) and accuracy table] The MMLU scores are all at or below the 0.25 chance level for a four-choice benchmark: Connectome 0.247, RandomConnectome 0.235, NoConnectome 0.241. HellaSwag scores are only slightly above chance (0.276, 0.257, 0.269). Reporting these as evidence of 'strong performance as a language model beyond simply next-word prediction' is not supported without reporting chance level, the number of choices, and effect sizes that account for the near-chance regime. At minimum, the authors should temper the NLP claim or remove MMLU from the central evidence.
minor comments (5)
  1. [Section 2.1.3] The notation for the generation states is confusing: f^0fin, f^1_theta, f^5fin, and f^5_theta are introduced without a clear schema for distinguishing a generation's final trained state from its initialization. A small table or consistent superscript notation would help.
  2. [Appendix A.3] The compression calculation has a typo: the entropy is 1.06 bits per weight, but the text says '1.6 bits,' and 124M * 1.06 bits equals about 16.4 MB, not 16 MB. The qualitative point stands, but the numbers should be corrected.
  3. [References [24], [25], [26]] Several references contain placeholder author names such as 'FirstName', 'Mansheej Paul et al.', and 'Firstname Zheng'. These citations are incomplete and should be replaced with full author lists.
  4. [Section 2.1.4] It is not stated whether the RandomConnectome mask preserves the per-layer sparsity of the Connectome mask or only the global 25% sparsity. If the random mask is sampled globally, differences in layerwise sparsity could confound the comparison; this should be specified.
  5. [Figure 1D] There is a typo in the panel label ('across-layer alignement') and the panel references in the text are inconsistent: Figure 1D is described as having panels i and ii, but the caption and text refer to 'Figure 1D, panel ii' and later 'Appendix A.5' with a separate figure number. Please standardize the panel and figure references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the connectome is empirically constructed and tested against external benchmarks.

full rationale

The paper's central result rests on an empirical comparison. A binary mask is produced by iterative pruning on the 4B-token corpus L, then used to initialize a model trained on the disjoint 100M-token corpus S, and this model is evaluated on FineWeb validation loss, HellaSwag, MMLU, Futrell reading times, and the Tuckute et al. fMRI benchmark. None of these evaluation metrics is used to select the mask; the mask is fixed before any inner-loop training. The pruning and binary-sign initialization procedures are adopted from the external lottery-ticket literature ([23,29]), and the paper explicitly presents them as inherited premises rather than deriving them anew. The only self-citations are to public datasets and published methods (e.g., [52,53,46]); they provide evaluation data and procedures, not the claimed conclusion, and are externally falsifiable. The absence of a data-agnostic pruning control is an experimental-design limitation and a validity threat, but it is not circularity: the Connectome versus RandomConnectome comparison is a genuine empirical outcome rather than an identity by construction, and no equation in the paper reduces the claimed result to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical entities. The 'model connectome' is a name for a sparse binary weight mask. Free parameters are the pruning schedule components and the choice of layer-wise over global pruning, all set by hand or by pilot experiments rather than by theory.

free parameters (4)
  • Pruning fraction per generation = 0.2
    Chosen by hand; six generations at this rate produce 25% retained weights. No ablation justifies this specific rate or the number of generations.
  • Number of generations = 6
    Fixed schedule to reach target sparsity; not tuned or varied in experiments.
  • Binary initialization magnitude = 0.02
    Set equal to the standard deviation of the dense initialization, following prior work [29]; not tuned.
  • Layer-wise vs global pruning mode = Layer-wise
    Selected after pilot ablations showed global pruning performed significantly worse; this is a design choice informed by the authors' own pilot experiments (Section 2.1.3 footnote).
assumptions (4)
  • domain assumption FineWeb text corpora are an acceptable proxy for human language exposure for studying developmental-scale learning.
    The paper equates 100M tokens with a child's language exposure up to age 10 and uses next-word prediction as the learning task (Sections 1 and 2.1.2).
  • domain assumption A sparse binary mask with constant-magnitude weights can serve as a meaningful initialization for learning.
    Inherited from the lottery ticket hypothesis [23, 29]; the paper builds on this result rather than deriving it.
  • domain assumption Linear encoding models and Pearson correlations are valid measures of model-brain and model-behavior alignment.
    Used in brain alignment (ridge regression, Section A.4.3) and behavioral alignment (Section A.4.2), following prior work [52, 65, 66].
  • domain assumption The outer-loop dataset L being 40 times larger than S meaningfully represents the information available to evolution.
    The analogy assumes that more data in the outer loop corresponds to evolutionary scale, but no biological claim is tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model Connectomes: A Generational Approach to Data-Efficient Language Models." pith.science (2026). https://pith.science/paper/5NHFCXN2

@misc{pith2026250421047,
  author       = {Pith},
  title        = {Pith review of: Model Connectomes: A Generational Approach to Data-Efficient Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NHFCXN2}},
  note         = {Machine review of arXiv:2504.21047}
}
read the original abstract

Biological neural networks are shaped both by evolution across generations and by individual learning within an organism's lifetime, whereas standard artificial neural networks undergo a single, large training procedure without inherited constraints. In this preliminary work, we propose a framework that incorporates this crucial generational dimension - an "outer loop" of evolution that shapes the "inner loop" of learning - so that artificial networks better mirror the effects of evolution and individual learning in biological organisms. Focusing on language, we train a model that inherits a "model connectome" from the outer evolution loop before exposing it to a developmental-scale corpus of 100M tokens. Compared with two closely matched control models, we show that the connectome model performs better or on par on natural language processing tasks as well as alignment to human behavior and brain data. These findings suggest that a model connectome serves as an efficient prior for learning in low-data regimes - narrowing the gap between single-generation artificial models and biologically evolved neural networks.

Figures

Figures reproduced from arXiv: 2504.21047 by the authors.

Figure 1
Figure 1. A. Conceptual overview, see description in Sections 1 and 2. B. Performance evaluation on standard NLP benchmarks: FineWeb validation loss (panel i), HellaSwag and MMLU (panel ii). C. Alignment with human reading times on naturalistic stories. D. Model-brain alignment, flexibly mapping all units within each model layer to brain responses (panel i) or through a more stringent procedure which localizes language-select… view at source ↗
Figure 2
Figure 2. In the main text, we present model-brain alignment results using the top 1% language [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 47 canonical work pages

  1. [1]

    Love, Christopher J

    Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Christopher J. Cueva, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Mar- tin N. Hebart, Nathan Cloos, Nikolaus Kriegeskorte, Nori Jacoby, Qiuyi Zhang, Raja Marjieh,...

    work page 2024

  2. [2]

    Language in brains, minds, and machines

    Greta Tuckute, Nancy Kanwisher, and Evelina Fedorenko. Language in brains, minds, and machines. Annual Review of Neuroscience, 47, 2024

    work page 2024

  3. [3]

    Catalyzing next-generation artificial intelligence through neuroai

    Anthony Zador, Sean Escola, Blake Richards, Bence ¨Olveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, et al. Catalyzing next-generation artificial intelligence through neuroai. Nature communications, 14(1):1597, 2023

    work page 2023

  4. [4]

    A deep learning framework for neuroscience

    Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761–1770, 2019

    work page 2019

  5. [5]

    How learning can guide evolution.Complex Systems, 1(3):495–502, 1987

    Geoffrey E Hinton and Steven J Nowlan. How learning can guide evolution.Complex Systems, 1(3):495–502, 1987. 6 To appear at the ICLR 2025 Workshop on Representational Alignment (Re-Align)

    work page 1987

  6. [6]

    A critique of pure learning and what artificial neural networks can learn from animal brains

    Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 10(1):3770, 2019

    2019

  7. [7]

    Direct fit to nature: an evolutionary perspective on biological and artificial neural networks

    Uri Hasson, Samuel A Nastase, and Ariel Goldstein. Direct fit to nature: an evolutionary perspective on biological and artificial neural networks. Neuron, 105(3):416–434, 2020

    work page 2020

  8. [8]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017), December 2017

    work page 2017

Show all 67 references
  1. [9]

    What artificial neural networks can tell us about human language acquisition

    Alex Warstadt and Samuel R Bowman. What artificial neural networks can tell us about human language acquisition. In Algebraic structures in natural language, pages 17–60. CRC Press, 2022

    2022

  2. [10]

    Call for papers–the babylm challenge: Sample-efficient pretraining on a developmen- tally plausible corpus

    Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. Call for papers–the babylm challenge: Sample-efficient pretraining on a developmen- tally plausible corpus. arXiv preprint arXiv:2301.11796, 2023

    2023 arXiv

  3. [11]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018

    2018

  4. [12]

    Llama v3: Next-generation foundation language model, 2023

    Meta AI. Llama v3: Next-generation foundation language model, 2023. Unpublished release. Accessed on 2025-02-05

    2023

  5. [13]

    Deepseek llm: Advancing deep information retrieval with large language models, 2023

    DeepSeek Team. Deepseek llm: Advancing deep information retrieval with large language models, 2023. Online release/preprint. Accessed on 2025-02-05

    2023

  6. [14]

    Scaling laws for neural language mod- els, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, et al. Scaling laws for neural language mod- els, 2020

    2020

  7. [15]

    Incorporating context into language encoding models for fmri

    Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. Advances in neural information processing systems, 31, 2018

    2018

  8. [16]

    Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

    Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in neural information processing systems, 32, 2019

    2019

  9. [17]

    Martin Schrimpf, Jonas Kubilius, Michael J. Lee, N. Apurva Ratan Murty, Robert Ajemian, and James J. DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3):413–423, November 2020

    2020

  10. [18]

    Brains and algorithms partially converge in natural language processing

    Charlotte Caucheteux and Jean-R ´emi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022

    2022

  11. [19]

    Shared computational principles for language processing in humans and deep language models.Nature neuroscience, 25(3):369–380, 2022

    Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models.Nature neuroscience, 25(3):369–380, 2022

    2022

  12. [20]

    Gptq: Accu- rate post-training quantization for generative pre-trained transformers, 2022

    Lukas Nagel, Diego Acuna, Elmar Schmerling, Marion Schenk, and Erich Elsen. Gptq: Accu- rate post-training quantization for generative pre-trained transformers, 2022. arXiv preprint

    2022

  13. [21]

    Movement pruning: Adaptive sparsity by fine-tuning

    Victor Sanh, Thomas Wolf, and Sebastian Ruder. Movement pruning: Adaptive sparsity by fine-tuning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 195–202. Association for Computational Linguistics, 2020

    2020

  14. [22]

    Le, Geoffrey Hinton, and Jeff Dean

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In Proceedings of the International Conference on Learning Representations (ICLR), 2017

    2017

  15. [23]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019

    2019

  16. [24]

    Unmasking the lottery ticket hypothesis: What’s encoded in a winning ticket’s mask? In Proceedings of the International Conference on Learning Representations (ICLR), 2023

    Mansheej Paul et al. Unmasking the lottery ticket hypothesis: What’s encoded in a winning ticket’s mask? In Proceedings of the International Conference on Learning Representations (ICLR), 2023

    2023

  17. [25]

    Lottery ticket adaptation: Mitigating destructive interference in llms,

    Ashwinee Panda et al. Lottery ticket adaptation: Mitigating destructive interference in llms,

  18. [26]

    Lottery tickets in llms: Robustness to adversarial examples via binary masking

    FirstName Zheng and Others. Lottery tickets in llms: Robustness to adversarial examples via binary masking. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (ACL), Dublin, Ireland, 2022. Association for Computational Linguistics. Long paper

    2022

  19. [27]

    Sparse winning tickets are data-efficient image recognizers

    Mukund Varma T, Xuxi Chen, Zhenyu Zhang, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Sparse winning tickets are data-efficient image recognizers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Process...

    2022

  20. [28]

    Pruning neural networks without any data by iteratively conserving synaptic flow

    Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377–6389, 2020

    2020

  21. [29]

    Deconstructing lottery tickets: Zeros, signs, and the supermask

    Hattie Zhou et al. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pages 3592–3602, 2019

    2019

  22. [30]

    Complex com- putation from developmental priors

    D ´aniel L Barab´asi, Taliesin Beynon, ´Ad´am Katona, and Nicolas Perez-Nieves. Complex com- putation from developmental priors. Nature Communications, 14(1):2226, 2023

    2023

  23. [31]

    Encoding innate ability through a genomic bottleneck

    Sergey Shuvaev, Divyansha Lachi, Alexei Koulakov, and Anthony Zador. Encoding innate ability through a genomic bottleneck. Proceedings of the National Academy of Sciences , 121(38):e2409160121, 2024

    2024

  24. [32]

    Enhancing in- terpretability using human similarity judgements to prune word embeddings

    Natalia Flechas Manrique, Wanqian Bao, Aurelie Herbelot, and Uri Hasson. Enhancing in- terpretability using human similarity judgements to prune word embeddings. arXiv preprint arXiv:2310.10262, 2023

    2023 arXiv

  25. [33]

    Pruning sparse features for cognitive modeling

    Nhut Truong, Uri Hasson, et al. Pruning sparse features for cognitive modeling. In The 7th annual conference on Cognitive Computational Neuroscience, 2024

    2024

  26. [34]

    American parenting of language-learning children: Persisting differences in family-child interactions observed in natural home environments

    Betty Hart and Todd R Risley. American parenting of language-learning children: Persisting differences in family-child interactions observed in natural home environments. Developmen- tal psychology, 28(6):1096, 1992

    1992

  27. [35]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl ´ıˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processin...

    2024

  28. [36]

    modded-nanogpt: Speedrunning the nanogpt baseline, 2024

    Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @Grad62304977. modded-nanogpt: Speedrunning the nanogpt baseline, 2024

    2024

  29. [37]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11867–11878

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11867–11878. AAAI Press, 2019

    2019

  30. [38]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Zou, David Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. arXiv preprint

    2021

  31. [39]

    On the predictive power of neural language models for human real-time comprehension behavior

    Ethan G Wilcox. On the predictive power of neural language models for human real-time comprehension behavior. arXiv preprint arXiv:2006.01912, 2020

    2006 arXiv

  32. [40]

    Byung-Doh Oh and William Schuler. Why does surprisal from larger transformer-based lan- guage models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350, 2023

    2023

  33. [41]

    Word frequency and predictability dissociate in naturalistic reading

    Cory Shain. Word frequency and predictability dissociate in naturalistic reading. Open Mind, 8:177–201, 2024

    2024

  34. [42]

    A probabilistic earley parser as a psycholinguistic model

    John Hale. A probabilistic earley parser as a psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics, 2001

    2001

  35. [43]

    The effect of word predictability on reading time is loga- rithmic

    Nathaniel J Smith and Roger Levy. The effect of word predictability on reading time is loga- rithmic. Cognition, 128(3):302–319, 2013

    2013

  36. [44]

    The natural stories corpus: a reading-time corpus of english texts containing rare syntactic constructions

    Richard Futrell, Edward Gibson, Harry J Tily, Idan Blank, Anastasia Vishnevetsky, Steven T Piantadosi, and Evelina Fedorenko. The natural stories corpus: a reading-time corpus of english texts containing rare syntactic constructions. Language Resources and Evaluation , 55:63–7...

    2021

  37. [45]

    Instruction-tuning aligns llms to the human brain

    Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosse- lut. Instruction-tuning aligns llms to the human brain. arXiv preprint arXiv:2312.00575, 2023

    2023 arXiv

  38. [46]

    Brain-like lan- guage processing via a shallow untrained multihead attention network

    Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. Brain-like lan- guage processing via a shallow untrained multihead attention network. arXiv preprint arXiv:2406.15109, 2024

    2024 arXiv

  39. [47]

    Transformer-based language model surprisal pre- dicts human reading times best with about two billion training tokens

    Byung-Doh Oh and William Schuler. Transformer-based language model surprisal pre- dicts human reading times best with about two billion training tokens. arXiv preprint arXiv:2304.11389, 2023

    2023 arXiv

  40. [48]

    Large gpt-like models are bad babies: A closer look at the relationship between linguistic competence and psycholinguistic measures

    Julius Steuer, Marius Mosbach, and Dietrich Klakow. Large gpt-like models are bad babies: A closer look at the relationship between linguistic competence and psycholinguistic measures. arXiv preprint arXiv:2311.04547, 2023

    2023 arXiv

  41. [49]

    Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal’s fit to reading times.arXiv preprint arXiv:2402.02255, 2024

    Byung-Doh Oh, Shisen Yue, and William Schuler. Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal’s fit to reading times.arXiv preprint arXiv:2402.02255, 2024

    2024 arXiv

  42. [50]

    Scaling in cognitive modelling: A multilingual approach to human reading times

    Andrea De Varda and Marco Marelli. Scaling in cognitive modelling: A multilingual approach to human reading times. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 139–149, 2023

    2023

  43. [51]

    New method for fmri investigations of language: defining rois functionally in individual subjects

    Evelina Fedorenko, Po-Jang Hsieh, Alfonso Nieto-Casta ˜n´on, Susan Whitfield-Gabrieli, and Nancy Kanwisher. New method for fmri investigations of language: defining rois functionally in individual subjects. Journal of neurophysiology, 104(2):1177–1194, 2010

    2010

  44. [52]

    Driving and suppressing the human lan- guage network using large language models

    Greta Tuckute, Aalok Sathe, Shashank Srikant, Maya Taliaferro, Mingye Wang, Martin Schrimpf, Kendrick Kay, and Evelina Fedorenko. Driving and suppressing the human lan- guage network using large language models. Nature Human Behaviour, pages 1–18, 2024

    2024

  45. [53]

    The llm language network: A neuroscientific approach for identifying causally task-relevant units.arXiv preprint arXiv:2411.02280, 2024

    Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. The llm language network: A neuroscientific approach for identifying causally task-relevant units.arXiv preprint arXiv:2411.02280, 2024

    2024 arXiv

  46. [54]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  47. [55]

    Big self-supervised models are strong semi-supervised learners

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural informa- tion processing systems, 33:22243–22255, 2020

    2020

  48. [56]

    Pathnet: Evolution channels gradient de- scent in super neural networks

    Chrisantha Fernando, Daniil Banarse, Charles Blundell, Yori Zwols, Oriol Vinyals, Daniel Apthorp, Daan Wierstra, and Koray Kavukcuoglu. Pathnet: Evolution channels gradient de- scent in super neural networks. arXiv preprint arXiv:1701.08734, 2017

    2017 arXiv

  49. [57]

    Learning both weights and connections for efficient neural networks

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems (NIPS), 2015

    2015

  50. [58]

    Synaptic density in human frontal cortex-developmental changes and effects of aging

    Peter R Huttenlocher et al. Synaptic density in human frontal cortex-developmental changes and effects of aging. Brain Res, 163(2):195–205, 1979

    1979

  51. [59]

    Natural evolution strategies

    Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and J ¨urgen Schmidhu- ber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949–980, 2014

    2014

  52. [60]

    Designing neural net- works through neuroevolution

    Kenneth O Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen. Designing neural net- works through neuroevolution. Nature Machine Intelligence, 1(1):24–35, 2019

    2019

  53. [61]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  54. [62]

    Sparse feature circuits: Discovering and editing interpretable causal graphs in language mod- els

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language mod- els. arXiv preprint arXiv:2403.19647, 2024

    2024 arXiv

  55. [63]

    Root mean square layer normalization, 2019

    Xiang Zhang et al. Root mean square layer normalization, 2019. arXiv preprint. 9 To appear at the ICLR 2025 Workshop on Representational Alignment (Re-Align)

    2019

  56. [64]

    Roformer: Enhanced transformer with rotary position embedding

    Ruixiang Su, Xinjian Lu, Wei Li, Rui Xiong, Xiaoyu Li, Hao Chen, et al. Roformer: Enhanced transformer with rotary position embedding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  57. [65]

    Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018

    Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018

    2018

  58. [66]

    Hosseini, Nancy Kanwisher, Joshua B

    Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of lan- guage: Integrative modeling converges on predictive processing. Proceedings of the National Academy o...

    2021

  59. [2024]

    7 To appear at the ICLR 2025 Workshop on Representational Alignment (Re-Align)

    arXiv preprint, https://arxiv.org/abs/2406.16797v1. 7 To appear at the ICLR 2025 Workshop on Representational Alignment (Re-Align)

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.