REVIEW 2 major objections 5 minor 36 references
Finding Structure in Language Models
T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Language models encode abstract grammar and can generalize it from indirect evidence, while adjective-order preferences largely track corpus frequencies.
desk verdict Careful psycholinguistic-style thesis with a strong structural priming chapter and an honest adjective-order study; the FiCT 'indirect evidence' claim is plausible but not yet proven without a leakage audit or matched-deletion control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Priming Effect (PE) is the load-bearing metric: the difference in log probability of a target sentence conditioned on a congruent versus an incongruent prime, averaged over controlled prime-target corpora, which isolates structural persistence from lexical and semantic overlap. Filtered Corpus Training (FiCT) is the load-bearing method: train on a corpus with a linguistic construction removed, then evaluate with Targeted Syntactic Evaluations, with the accuracy delta between filtered and unfiltered training measuring generalisation from indirect evidence. The adjective-order analysis adds AOP-Δ, the log-probability difference between natural and swapped adjective order, correlated against relative n-gram counts extracted from the training corpus.
What would settle it
Run a FiCT experiment in which every sentence containing a prepositional-phrase-modified subject is removed from the training corpus, verified with an independent parser and n-gram search of the training files, then test the trained model on Targeted Syntactic Evaluations for that construction; if the model scores at chance, or if the filtered corpus is found to contain any direct example of the construction, the claim that models generalise from indirect evidence would be refuted.
Extended reading notes
Core claim
On its own terms, the central claim is that language models do not merely memorise training text: they build representations of abstract grammatical structure that persist across sentences and can be acquired from indirect evidence. The structural-priming experiments show that congruent primes raise the probability of a target structure even when prime and target share no lexical content, and that the effect increases with repeated exposure and decreases with distance, mirroring human production patterns. The Filtered Corpus Training experiments show that models trained on corpora from which a construction was removed can still make correct grammaticality judgements about that construction, and that Transformers' better perplexity does not translate into better linguistic generalisation than LSTMs. The adjective-order experiments complicate the picture: high accuracy on naturally occurring orders is largely predicted by simple n-gram counts, though context and unseen pairs reveal some abstraction beyond counts.
Load-bearing premise
The conclusion rests on the assumption that the filtering step removes every direct example of the target construction from training while leaving the rest of the distribution intact; if some examples leak through or the filtering changes correlated cues, the apparent generalisation could come from direct evidence.
Editorial extensions
If this is right
- If language models genuinely prime, structural information is active at inference time and can be studied through sentence-level probability contrasts rather than probes or fine-tuning.
- If FiCT results are correct, a language model can acquire a construction it never saw, so the language-modelling objective itself can support poverty-of-stimulus-style generalisation.
- Perplexity and linguistic generalisation dissociate: a Transformer's perplexity advantage does not imply better structural learning, so architecture comparisons should separate the two.
- Adjective-order results imply that claims about LM linguistic rules must be checked against training-data frequency baselines, since bigram counts alone reach around 90 percent accuracy on natural orders.
- Inverse-frequency and lexical-overlap effects in priming suggest LM predictions incorporate abstract expectations that mirror human production patterns.
Reading between the lines
- Beyond the paper, FiCT could be pointed at adjective order: remove every double-adjective phrase from training and test whether preferences persist; if they do, the signal must come from adjective-noun statistics or semantic classes rather than memorised collocations.
- The cumulative priming effect reported here, where more congruent primes monotonically increase the effect, resembles a minimal form of in-context learning; a direct test would compare priming curves to few-shot demonstrations on a held-out structure with matched lexical content.
- The strong bigram correlation in adjective order implies that interpretability claims about LM linguistic rules should routinely be checked against corpus-frequency baselines; otherwise human-like accuracy may be misread as human-like representation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The thesis investigates whether Transformer language models encode abstract grammatical structure, using three complementary strands: structural priming as a psycholinguistic paradigm (Chapters 2–3), analyses connecting training-data statistics to model behaviour including the Filtered Corpus Training (FiCT) intervention (Chapters 4–6), and controlled synthetic-language experiments for evaluating interpretability methods (Chapters 7–8). The central positive claims are that language models exhibit structural priming modulated by lexical, semantic, and frequency factors in ways partly analogous to humans, and that FiCT-trained models can judge constructions that were filtered out of their training data, indicating generalization from indirect evidence. The adjective-order chapter is more cautious, showing that simple bigram frequencies from The Pile predict most adjective-order preferences, with limited generalization to unseen pairs and with context effects that go beyond n-gram counts.
Significance. If the results hold, the thesis makes a substantial contribution to the debate on memorisation versus generalisation in language models and demonstrates the value of repurposing psycholinguistic paradigms in NLP. Concrete strengths include: the carefully constructed and publicly released Prime-LM and CAP corpora; the Priming Effect metric of Chapter 2, which compares two primes against the same target and thereby avoids the prior-perplexity confound of earlier adaptation-based measures; the use of intermediate checkpoints and batch-level n-gram counts in Chapter 4 to relate model behaviour directly to its training signal; the genuinely causal FiCT training-intervention design; and the FIDAM framework in Chapter 7, which explicitly addresses faithfulness of feature-interaction attributions. The thesis is also appropriately self-critical, for example in reporting that bigram statistics explain 90.3% of adjective-order accuracy and in showing that priming asymmetries are opposite in direction to typical human asymmetries.
major comments (2)
- [§5.3, Fig. 5.1, §5.5] The central claim of Chapter 5—that FiCT-trained models 'have never seen instances of their evaluation targets' and therefore generalize from indirect evidence—requires that the filters actually remove all direct evidence of each target construction. The visible text describes filters based on parsing and surface heuristics (Figure 5.1; §5.3) but reports no independent audit of the filtered corpora: parse errors, coordination, or alternative syntactic realizations could leave fragments or full instances of the target construction in the 'filtered' training data, making the evaluation items partially seen. The authors should quantify filter precision and recall on a held-out sample and, ideally, verify with an n-gram or dependency search that the TSE evaluation sentences or their structural instantiations do not appear in the filtered corpus. Without such an audit, the strong 'never seen' formulation is unsupported.
- [§5.5, §5.7] Even if the filters are perfectly precise, the accuracy-delta comparison between filtered and unfiltered models cannot separate indirect-evidence generalization from distribution shift induced by filtering. Removing all sentences containing PP-modified subjects, for example, also changes the frequencies of specific prepositions, noun-preposition bigrams, and subject lengths, which can shift baseline statistics for the TSE items. The chapter does not include a matched-deletion control in which an equal amount of unrelated, length- and frequency-matched material is removed. Adding such a control—or showing via re-weighting that the filtered corpus preserves the relevant marginal distributions—is necessary to support the conclusion that the measured accuracy deltas reflect genuine generalization to unseen constructions rather than a shift in the training distribution.
minor comments (5)
- [§2.3, Eq. (2.2)] Equation (2.2) writes the conditioning context as 'tx_i' in the final term; it should be the previous target tokens tx_{<i}, as correctly written in Eq. (3.2).
- [Chapter 2 captions] Several cross-references in Chapter 2 are stale: the captions of Figures 2.3–2.5 refer to '§7.1.1', '§7.2', '§7.3.1', '§7.3.2' and '§7.4', which should be Sections 2.6.1–2.6.5, and Table 2.1 is referenced as 'T able2.1'.
- [§3.2.1] There is a duplicated word in the sentence 'unless unless explicitly mentioned otherwise'.
- [Fig. 5.1] Figure 5.1 labels the method as 'FICT' in the overview box while the text consistently uses 'FiCT'; the capitalization should be standardised.
- [Fig. 3.2] The token labels in the legend of Figure 3.2 (e.g., 'NPO2 × NDO2') are difficult to parse; the caption would benefit from listing the corresponding word positions in the example sentence.
Circularity Check
No significant circularity: the thesis's conclusions rest on controlled corpora, external data statistics, and empirical interventions, not on outputs fitted to their own inputs.
full rationale
The derivation chain in this thesis does not reduce to its inputs. The Priming Effect (Eq. 2.1) is a defined measure, not a fitted parameter: positive s-PE is an empirical outcome on the Prime-LM corpus, and the corpus construction (§2.4) controls for lexical overlap and semantic similarity before the effect is measured. Chapter 3's regression uses model-computed surprisal and structural preference (Eqs. 3.1 and 3.4) as explanatory predictors of the same models' PE; because these are predictors in a multivariate mixed model rather than parameters fitted to reproduce PE, the significant coefficients are an empirical finding, and the paper itself flags (§3.7) that a data-distribution control would be needed to rule out indirect effects. Chapter 4 compares AOP-Δ (Eq. 4.1) against external n-gram counts from The Pile; the 90.3% bigram accuracy is independent of the model, and the 'unseen order' analysis uses checkpoint-level counts to define held-out items, so the limited generalisation claim is not equivalent to the corpus statistics. FiCT (Ch. 5) trains on filtered corpora and evaluates on targeted syntactic evaluations; the accuracy-delta comparison is an empirical intervention, and leakage or distribution shift, if present, would weaken the conclusion but are not circularity. The only self-citation with a methodological role is the author's diagNNose software (§2.5.1, §3.3.3), which is an implementation tool and not load-bearing for any of the claims. No equation is defined in terms of the quantity it purports to predict, and no fitted parameter is renamed as a prediction. Score 0.
Assumptions & free parameters
free parameters (3)
- Prime-LM corpus size per target structure =
15,000 prime-target pairs
- Semantic similarity threshold =
cosine similarity >= 0.4
- Number of primes per target =
10
assumptions (5)
- domain assumption Sentence probability factorizes as a product of token probabilities, and log-probability differences between congruent and incongruent primes measure structural priming (Equations 2.1 and 2.2).
- domain assumption The Prime-LM template vocabulary and manual verb and noun selection via USF norms, WordNet categories, and COCA frequency produce grammatical and semantically controlled stimuli.
- ad hoc to paper FiCT filters remove all direct evidence of target constructions from training data while preserving the distributional cues needed for indirect generalization.
- domain assumption Pythia intermediate checkpoints and batch-level n-gram counts correctly identify which adjective pairs a model has seen at a given point in training.
- domain assumption Targeted Syntactic Evaluations accuracy is a valid measure of linguistic generalization.
Cite this review
Pith. "Pith review of Finding Structure in Language Models." pith.science (2026). https://pith.science/paper/WM4YO2SX
@misc{pith2026241116433,
author = {Pith},
title = {Pith review of: Finding Structure in Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/WM4YO2SX}},
note = {Machine review of arXiv:2411.16433}
}
read the original abstract
When we speak, write or listen, we continuously make predictions based on our knowledge of a language's grammar. Remarkably, children acquire this grammatical knowledge within just a few years, enabling them to understand and generalise to novel constructions that have never been uttered before. Language models are powerful tools that create representations of language by incrementally predicting the next word in a sentence, and they have had a tremendous societal impact in recent years. The central research question of this thesis is whether these models possess a deep understanding of grammatical structure similar to that of humans. This question lies at the intersection of natural language processing, linguistics, and interpretability. To address it, we will develop novel interpretability techniques that enhance our understanding of the complex nature of large-scale language models. We approach our research question from three directions. First, we explore the presence of abstract linguistic information through structural priming, a key paradigm in psycholinguistics for uncovering grammatical structure in human language processing. Next, we examine various linguistic phenomena, such as adjective order and negative polarity items, and connect a model's comprehension of these phenomena to the data distribution on which it was trained. Finally, we introduce a controlled testbed for studying hierarchical structure in language models using various synthetic languages of increasing complexity and examine the role of feature interactions in modelling this structure. Our findings offer a detailed account of the grammatical knowledge embedded in language model representations and provide several directions for investigating fundamental linguistic questions using computational methods.
Figures
Figures from the paper (41 more)
Reference graph
Works this paper leans on
- [1]
- [2]
- [3]
-
[4]
16.9 14.5 Swapped order always preferred
Icterids are unusual in songbirds because they haveconsiderable sexualdimorphism. 16.9 14.5 Swapped order always preferred
- [5]
- [6]
- [7]
-
[8]
−5.8 −5.7 Context improves AOP
The daily averageattendance from January through November 2010 was 22,133 peo- ple a day. −5.8 −5.7 Context improves AOP
work page 2010
Show all 36 references
-
[9]
0.9 16.8
Being a spherical 3-manifold, it is the only homology 3-sphere, besides the 3-sphere itself, with a finite fundamental group. 0.9 16.8
-
[10]
−0.5 15.3
Since the 2015 Styria municipal structural reform, it is part of the municipality Birkfeld. −0.5 15.3
2015
-
[11]
0.2 15.4
I won it at a fair last night. 0.2 15.4
-
[12]
2.6 17.5 Context worsens AOP
I say that the man who does not play whist lays up asad old age for himself. 2.6 17.5 Context worsens AOP
-
[13]
12.3 4.0
This is an ability which may have been present in their last common ancestor in the Archaean. 12.3 4.0
-
[14]
Jacob, you’ve had a complex partial seizure, which can cause psychosis, including reli- gious psychosis. 7.3 0.3
-
[15]
6.2 −0.6
He was also a right-arm medium fast bowler with three wickets in test matches to his credit. 6.2 −0.6
-
[16]
4.0 −2.8 200 A.2
He wheeled round sharply, and distinguished her lying withhelpless outspreadarms on the couch. 4.0 −2.8 200 A.2. Adjective Order A.2.2 Correlations to Cognitive Predictors 1 16 256 3000 20000 143000 0.0 0.2 0.4Spearman's 1 2 3 Length 1 16 256 3000 20000 143000 1 2 3 PMI(a;n) 1...
2019
-
[17]
Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert- Threlkeld. 2021. Language models use monotonicity to assess NPI licensing . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4958–4969, Online. Association for C...
2021
-
[18]
Arabella Sinclair*, Jaap Jumelet*, Willem Zuidema, and Raquel Fernández. 2022. Structural persistence in language models: Priming as a window into abstract lan- guage representations. Transactions of the Association for Computational Linguistics, 10:1031–1050
2022
-
[19]
Jaap Jumelet and Willem Zuidema. 2023a. Feature interactions reveal linguistic structure in language models. In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 8697–8712, T oronto, Canada. Association for Computa- tional Linguistics
2023
-
[20]
Jaap Jumelet and Willem Zuidema. 2023b. T ransparency at the source: Evaluating and interpreting language models with access to the true distribution . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4354–4369, Singapore. Association for Computa...
2023
-
[21]
Jaap Jumelet, Willem Zuidema, and Arabella Sinclair. 2024b. Do language mod- els exhibit human-like structural priming effects? In Findings of the Association for Computational Linguistics ACL 2024, pages 14727–14742, Bangkok, Thailand and virtual meeting. Association for Comp...
2024
-
[22]
2024.Filtered corpus training (FiCT) shows that language models can generalize from indirect evidence
Abhinav Patil*, Jaap Jumelet*, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie W ang, Clevis Willrich, and Shane Steinert-Threlkeld. 2024.Filtered corpus training (FiCT) shows that language models can generalize from indirect evidence. Transac- tions of the Association for Com...
2024
-
[23]
Jaap Jumelet, Lisa Bylinina, Willem Zuidema, and Jakub Szymanik. 2024a. Black big boxes: Do language models hide a theory of adjective order? Under Review During my PhD I have also contributed to the following papers:
-
[24]
Jaap Jumelet. 2020. diagNNose: A library for neural activation analysis . In Pro- ceedings of the Third BlackboxNLP W orkshop on Analyzing and Interpreting Neural Networks for NLP, pages 342–350, Online. Association for Computational Linguis- tics
2020
-
[25]
2021.Language mod- elling as a multi-task problem
Lucas W eber, Jaap Jumelet, Elia Bruni, and Dieuwke Hupkes. 2021.Language mod- elling as a multi-task problem. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main V olume, pages 2049– 2060, Online. Association fo...
2021
-
[26]
T om Kersten, Hugh Mee W ong, Jaap Jumelet, and Dieuwke Hupkes. 2021. At- tention vs non-attention for a shapley-based explanation method. In Proceedings of Deep Learning Inside Out: The 2nd W orkshop on Knowledge Extraction and Integra- tion for Deep Learning Architectures, D...
2021
-
[27]
2022.The birth of bias: A case study on the evolution of gender bias in an English language model
Oskar V an Der W al, Jaap Jumelet, Katrin Schulz, and Willem Zuidema. 2022.The birth of bias: A case study on the evolution of gender bias in an English language model. In Proceedings of the 4th W orkshop on Gender Bias in Natural Language Processing (GeBNLP), pages 75–75, Sea...
2022
-
[28]
Lucas W eber, Jaap Jumelet, Paul Michel, Elia Bruni, and Dieuwke Hupkes. 2023. Curriculum learning with adam: The devil is in the wrong details. CoRR, abs/2308.12202 214 B. List of Publications
2023 arXiv
-
[29]
Jaap Jumelet, Michael Hanna, Marianne de Heer Kloots, Anna Langedijk, Char- lotte Pouw, and Oskar van der W al. 2023. ChapGTP, ILLC’s attempt at raising a BabyLM: Improving data efficiency by automatic task formation. In Proceedings of the BabyLM Challenge at the 27th Conferen...
2023
-
[30]
2023.Attribu- tion and alignment: Effects of local context repetition on utterance production and comprehension in dialogue
Aron Molnar, Jaap Jumelet, Mario Giulianelli, and Arabella Sinclair. 2023.Attribu- tion and alignment: Effects of local context repetition on utterance production and comprehension in dialogue. In Proceedings of the 27th Conference on Computational Natural Language Learning (C...
2023
-
[31]
Hosein Mohebbi, Jaap Jumelet, Michael Hanna, Afra Alishahi, and Willem Zuidema
-
[33]
Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem Zuidema, and Jaap Jumelet
-
[34]
In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4764– 4780, Mexico City, Mexico
DecoderLens: Layerwise interpretation of encoder-decoder transformers. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4764– 4780, Mexico City, Mexico. Association for Computational Linguistics
2024
-
[35]
2024.Interpretability of language models via task spaces
Lucas W eber, Jaap Jumelet, Elia Bruni, and Dieuwke Hupkes. 2024.Interpretability of language models via task spaces . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 4522– 4538, Bangkok, Thailand. Asso...
2024
-
[36]
Arabella Sinclair, Anastasia Klimovich-Gray, Jaap Jumelet, Nika Adamian, and Ag- nieszka Konopka. 2024. Structural priming in humans and large language models. Under Review 215 Structuur Ontrafelen uit T aalmodellen Samenvatting W anneer we spreken, schrijven of luisteren, cre...
2024
-
[2024]
In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Ab- stracts, pages 21–26, St
T ransformer-specific interpretability. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Ab- stracts, pages 21–26, St. Julian’s, Malta. Association for Computational Linguistics
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.