Pith. sign in

REVIEW 4 major objections 4 minor 43 references

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This systematic review of 337 studies finds that transformer language models reliably handle formal syntax like agreement but show weaker, more variable ability at the syntax-semantics interface.

desk verdict A transparent systematic review whose useful database and cautious body are betrayed by an abstract claiming a cross-linguistic result the paper never tests. read the letter →

arxiv 2601.19926 v2 pith:NT7ZDF6L submitted 2026-01-09 cs.CL cs.AI

Nora Graichen , Iria de-Dios-Flores , Gemma Boleda This is my paper
classification cs.CLcs.AI
keywords syntaxtransformerslanguagemodelsinterpretabilitysystematicreviewBLiMPsyntax-semanticsinterfaceprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This review of 337 studies and over 1,000 model results argues that transformer language models acquire a substantial amount of syntax from the language-modeling task alone. The clearest evidence is behavioral: models perform strongly on formal, form-oriented phenomena like agreement and part of speech, more weakly and variably on phenomena at the syntax-semantics interface such as binding, quantifiers, and island effects, and worse for languages with less digital support. Probing and mechanistic studies corroborate that syntactic information is present internally, but most existing work is observational and methodologically heterogeneous, so detailed computational mechanisms remain unclear. The central conclusion is that the field should stop asking whether transformers know syntax and instead work on why interface phenomena are harder, standardize evaluation practices, and broaden the empirical scope beyond English and BERT.

What carries the argument

The load-bearing instrument is the annotated database of 337 studies (1,015 model results) organized along five annotation dimensions, plus BLiMP—the Benchmark of Linguistic Minimal Pairs—as the common benchmark allowing quantitative comparison. BLiMP works by giving models pairs of sentences that differ only in grammaticality and checking whether the model assigns higher probability to the grammatical one. The database carries the descriptive weight: phenomenon categories, model types, languages, methods, and findings are extracted from each paper, and BLiMP per-category scores provide the only directly comparable cross-study measure, enabling the formal-syntax-versus-interface contrast and

What would settle it

Take the released database and hand-code a random sample of, say, 60 of the 337 papers on the five AI-assisted annotation variables; if the manual labels diverge from the AI-generated ones on the phenomena that define the central contrast (agreement versus binding and quantifiers), the reported gap could be an artifact. Independently, re-running the BLiMP analysis using per-category scores from all available studies rather than the 11 that reported them would show whether the formal-versus-interface ordering survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that transformer language models encode a non-trivial amount of syntactic knowledge, with a specific internal gradient: strong, often human-level performance on formal syntactic relations such as subject-verb, anaphor, and determiner-noun agreement, and weaker, more variable performance at the syntax-semantics interface, including binding, argument structure, NPI licensing, control/raising, quantifiers, and island effects. The evidence comes from a systematically collected database of 337 articles, analyzed through behavioral, probing, and mechanistic methods, with BLiMP used as the common metric for quantitative comparison. The authors present this as a tentativ

Load-bearing premise

The quantitative conclusions rest on the accuracy of the semi-automated annotation workflow that turned 337 papers into a structured database; the paper itself notes in Appendix C that the validation was rated by an AI model with a median score of 4.1 out of 5—the same class of model that generated the summaries—so any systematic distortion in those labels would propagate into the aggregate patterns.

Editorial extensions

If this is right

  • The syntax-semantics interface gap means researchers should expect model failure on binding, quantifiers, and island effects even when agreement looks solved; benchmarks that mix categories, like overall BLiMP scores, can mask this.
  • Because performance scales with training data and model size, larger models should continue to improve on formal syntax; whether the interface gap closes at scale is unknown, since no bidirectional models beyond roughly 1B parameters and 30B training tokens have been tested.
  • Current conclusions about transformer syntax are really conclusions about English and BERT: 91% of studies include English and 58% of results involve BERT or its variants, so cross-linguistic or architectural generalizations need new data.
  • The field should shift from observational probing to causal, mechanistic methods that connect internal representations to behavior, and should report per-category benchmark scores rather than overall averages.
  • Languages with less digital support show consistently lower performance, so multilingual evaluation is not just a coverage nicety but a substantive test of syntactic generality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the interface gap is real, a testable prediction follows that the authors do not draw: models should show larger degradation on binding and scope phenomena than on agreement when lexical items are replaced with nonce words, because interface tasks require more than surface-form matching.
  • The review's English/BERT skew implies the 'non-trivial syntactic knowledge' claim is safest as a statement about high-resource languages and masked encoders; extending to low-resource languages may reveal qualitative rather than merely quantitative differences.
  • The authors' standardization recommendation could be operationalized as a public reporting schema for benchmark results; if adopted, subsequent systematic reviews could move from narrative synthesis to meta-analysis with effect sizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper reports a systematic review of 337 articles on syntactic knowledge in transformer-based language models (TLMs), building a database of 1,015 individual results annotated for models, languages, syntactic phenomena, interpretability methods, and findings. The review's principal descriptive findings are that the literature is concentrated on English and BERT-like models; that behavioral, probing, and mechanistic methods are used in relatively balanced proportions; and that, based on an analysis of English BLiMP scores from 25 papers (11 with per-category results), TLMs perform well on formal syntactic phenomena such as agreement but more variably and less well on syntax-semantics interface phenomena such as binding, quantifiers, and island effects. The paper also contributes a set of recommendations for future work, including better reporting, standardization, and broader empirical coverage.

Significance. If its conclusions hold, the paper provides a valuable and much-needed quantitative synthesis of a large and heterogeneous literature, and it usefully reframes the field's central question from 'do transformers learn syntax?' to 'why are syntax-semantics interface phenomena harder and how should the empirical base be broadened?' The database and annotation scheme, if released, would be a significant community resource. However, the paper's main quantitative claims rest on a narrow evidentiary basis — English BLiMP scores from a small subset of papers — and one abstract-level generalization goes beyond the analyses actually reported. These issues are correctable but require substantive revision.

major comments (4)
  1. [Abstract; §3.3; Limitations] The abstract states: 'Performance is also consistently lower for languages with less digital support.' This claim is not supported anywhere in the body. Section 3.3 explicitly analyzes only English BLiMP scores, and the Limitations section states that the answer to RQ3 is based on one benchmark (BLiMP) and one language (English). No definition of 'digital support' is given, and no per-language performance comparison is reported. This sentence should be removed unless a corresponding analysis is added.
  2. [App. C.1; §2] The reliability of the LLM-based annotation workflow is load-bearing for all aggregate counts and qualitative conclusions, but the validation in App. C.1 is circular: the AI-generated summaries were scored by ChatGPT, the same type of model that produced them, with a median score of 4.1 and no reported human verification or inter-annotator agreement. Residual errors are acknowledged in App. C.2. Please provide a human-validated sample or a sensitivity analysis showing that annotation errors do not materially affect the main conclusions.
  3. [§3.3; Fig. 5 (right)] The central conclusion that formal syntax is easier than the syntax-semantics interface for TLMs is based on 11 papers reporting per-category English BLiMP scores, with descriptive boxplots and no statistical testing, no confidence intervals, and no control for model family, size, or training data. The body appropriately says 'tentatively conclude,' but the abstract states the conclusion categorically. Please either add quantitative uncertainty/statistical support or soften the abstract accordingly.
  4. [Abstract; §1] The abstract reports 'over 3,000 datapoints,' while the full text (§1, §4) consistently reports '1,015 individual results.' This is a substantial numerical discrepancy. If the 3,000 figure refers to a different unit (e.g., individual model-phenomenon pairs, or raw items rather than study-level results), this should be stated explicitly; as written, the abstract overstates the database size and undermines the trustworthiness of the reported statistics.
minor comments (4)
  1. [Limitations] Typo: 'language mdoels' should be 'language models.'
  2. [App. C.3; Fig. 13 caption] Caption refers to 'NotebookLLM,' but the text and reference use 'NotebookLM.' Please standardize.
  3. [Table 1] The 2025 row begins with 'Acs et al. (2024)', which appears to be a duplicate/relic; also reference 'N. Atox and M. Clark' lacks a year and publication venue in Table 1.
  4. [General] The database and analysis code are promised 'upon acceptance'; for a systematic review whose quantitative claims depend on the database, consider providing at least the aggregated data tables as supplementary material now.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the review's conclusions are aggregations of external primary studies, not reductions to the paper's own inputs.

full rationale

This paper is a systematic review rather than a derivation. The central claims—that TLMs encode nontrivial syntactic knowledge, that formal syntactic phenomena are handled better than syntax–semantics interface phenomena, and that the field over-relies on English and BERT—are presented as aggregations of a 337-study database built with explicit search, eligibility, and annotation procedures. The RQ3 conclusion is computed from English BLiMP scores in 25 external papers, plotted in Fig. 5, with human performance taken from Warstadt et al. (2020a); it is not fitted to a target or defined into existence. The six seed reviews are prior external work used for snowballing, not self-citations by the present authors, and the review explicitly does not include those reviews in the analyzed corpus. The AI-assisted annotation workflow is validated against a manually collected gold set; using ChatGPT to rate summary similarity is a reliability caveat, not a reduction of the conclusions to the model's own outputs. The abstract's claim that performance is 'consistently lower for languages with less digital support' is not derived in the body—the Limitations section states that the RQ3 answer is based only on one benchmark and one language—and the abstract's 'over 3,000 datapoints' conflicts with the full text's '1,015 individual results.' These are evidence-quality and reporting inconsistencies, not circularity under the defined patterns, and no specific equation or fitted parameter is shown to be equivalent to an input.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities: this is a review, not a model-building paper. The load-bearing assumptions are about the completeness of the literature search, the reliability of the AI-based annotation, and the appropriateness of BLiMP for the central comparison.

assumptions (3)
  • domain assumption The six prior narrative reviews provide a sufficiently comprehensive and unbiased seed set for snowballing to identify relevant literature.
    Candidate identification (§2, Fig. 1) starts from the references of six previous reviews. If those reviews have selection biases, the database inherits them; the authors did not fully validate coverage against an independent corpus.
  • ad hoc to paper The LLM-based annotation workflow (NotebookLM + ChatGPT) yields reliable structured data on model type, phenomena, materials, method, and findings.
    App. C.1: validation compared AI summaries against n=91 manual gold items and was 'AI-rated by ChatGPT... with a median score of 4.1'. The validation score is itself generated by the same kind of model used to produce the summaries, and residual errors are acknowledged in C.2.
  • domain assumption BLiMP is a valid instrument for answering RQ3 about syntactic knowledge in TLMs.
    §3.3 narrows the knowledge-status analysis to English BLiMP scores from 25 papers. The authors acknowledge BLiMP's limited coverage and the arbitrariness of its aggregate weighting (citing Kulmizev & Nivre 2022), so the conclusion 'formal syntax easier than syntax-semantics interface' is contingent on this benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models." pith.science (2026). https://pith.science/paper/NT7ZDF6L

@misc{pith2026260119926,
  author       = {Pith},
  title        = {Pith review of: The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NT7ZDF6L}},
  note         = {Machine review of arXiv:2601.19926}
}
read the original abstract

We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on over 3,000 datapoints spanning a wide range of syntactic phenomena, languages, models, and methods. We take the data to collectively show that TLMs encode a non-trivial amount of syntactic knowledge. Behavioral evidence shows strong performance on formal syntactic phenomena, but weaker and more variable performance on phenomena at the syntax-semantics interface. Performance is also consistently lower for languages with less digital support. Probing and mechanistic studies further support the presence of syntactic knowledge in TLMs. Yet, because most work remains observational and current approaches are methodologically heterogeneous, insight into the detailed computational mechanisms underlying syntactic processing remains limited. At the same time, the literature remains heavily concentrated on English and BERT-like models. We discuss the implications of our results and provide recommendations for future research.

Figures

Figures reproduced from arXiv: 2601.19926 by the authors.

Figure 1
Figure 1. Flowchart summarizing database creation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: Number of studies published each year (2025 until 31/07) categorized by model type. Right: 20 most frequently used TLMs. TLMs are counted once per study and grouped by their family or specific model name when available (e.g., one BERT count for a study that compares BERT-base/large/(un)cased). Overall, the database features 54 different TLM names and 609 instances without duplicates per study. cluding blog pos… view at source ↗
Figure 3
Figure 3. Left: Counts of syntactic phenomena by evaluated language setting (only English tested, English and other languages, and non-English). Right: Counts of syntactic phenomena investigated per study. in App. D.1) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Top: Percentage of methods over years. Bottom: Counts of experimental materials and model types by interpretability method (unknown experimen￾tal material with fewer than 10 counts excluded) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Left: BLiMP average scores plotted against the number of training tokens (log scale). Right: BLiMP scores from the 11 studies that provide at least one result per-category for English. Human performance taken from Warstadt et al. (2020a). Top ⋆ (Roberta large) and bott…
Figure 6
Figure 6. Figure 6: Distribution by model type and language coverage. ∼60% to 73%) evaluation, while causal and both model types were more often tested on phenomena like POS and other lexical properties and depen￾dency parsing and constituency (<20%). Regard [PITH_FULL_IMAGE:figures/full…
Figure 7
Figure 7. Figure 7: Heatmap of counts illustrating model type [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]
Figure 8
Figure 8. Figure 8: Counts heatmap of model language coverage [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]
Figure 10
Figure 10. Figure 10: Left: BLiMP (Warstadt et al., 2020a) average scores shown by model size and model type. If a paper reports scores for TLMs trained on varying sizes of training data, we include only the fully trained models in the boxplot. Right: Computed model rankings per phenomenon…
Figure 11
Figure 11. Figure 11: Distribution of studies in our database across syntactic phenomena, model type (bidirectional, causal, [PITH_FULL_IMAGE:figures/full_fig_p037_11.png]
Figure 12
Figure 12. Figure 12: Table from Warstadt et al. (2020a); minimal pairs from each of the 12 linguistic phenomenon categories covered by BLiMP. Column N records the number of different subphenomena exemplifying the relevant general phenomenon, 67 in total [PITH_FULL_IMAGE:figures/full_fig_…
Figure 13
Figure 13. Figure 13: Illustration of the AI-based workflow. Top: NotebookLLM (Google Labs, 2023) interaction depicting how the model assists with information extraction; numbers refer to the original source in the paper. Bottom: Example of a ChatGPT (OpenAI, 2025) session showing a query …
Figure 14
Figure 14. Figure 14: Instance of a ChatGPT (OpenAI, 2025) prompt showing our evaluation query with an example [PITH_FULL_IMAGE:figures/full_fig_p040_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 13 linked inside Pith

  1. [1]

    Name as reported in the paper, e.g., bert base uncased

  2. [2]

    Name enriched with post processing script: e.g.,family=“BERT” plus mention of size, BERT-base

  3. [3]

    bert base uncased

    Final example: list({string =“bert base uncased”,family=“BERT”, size= base,case= uncased}). Fine-tuned:fine tuned, indicates whether fine-tuning was performed. Language coverage: model language coverage •Monolingual:One language from scratch or continually pretrained (e.g., GPT2, BERT, ru- BERT). •Multilingual:Trained on multiple languages simultaneously ...

    2023

  4. [6]

    ArXiv:2402.18838

    When does word order matter and when doesn’t it?arXiv preprint. ArXiv:2402.18838. Jiali Cheng and Hadi Amiri. 2025. Linguistic Blind Spots of Large Language Models. InProceedings of the Workshop on Cognitive Modeling and Compu- tational Linguistics, pages 1–17, Albuquerque, New Mexico, USA. Association for Computational Lin- guistics. Daniil Cherniavskii,...

    arXiv 2025

  5. [7]

    Vittoria Dentella, Fritz G ¨unther, and Evelina Leivada

    Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias.Proceedings of the National Academy of Sciences of the United States of America, 120(51):e2309583120. Vittoria Dentella, Fritz G ¨unther, and Evelina Leivada

  6. [9]

    InProceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 375– 388, Punta Cana, Dominican Republic

    Not All Models Localize Linguistic Knowl- edge in the Same Place: A Layer-wise Probing on BERToids’ Representations. InProceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 375– 388, Punta Cana, Dominican Republic. Association for Computational Linguistics. Steven Y . Feng, Noah D. Goodman, and Michael...

    2024

  7. [10]

    InProceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 238–252, Singa- pore

    Not all layers are equally as important: Every Layer Counts BERT. InProceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 238–252, Singa- pore. Association for Computational Linguistics. Luke Gessler and Nathan Schneider. 2023. Syntac- tic Inductive Bias in Transformer Language Models: Especially Help...

    arXiv 2023

  8. [11]

    Computer Speech & Language, 71:101261

    BERT syntactic transfer: A computational ex- periment on Italian, French and English languages. Computer Speech & Language, 71:101261. Raffaele Guarasci, Stefano Silvestri, Giuseppe De Pietro, Hamido Fujita, and Massimo Esposito

Show all 43 references
  1. [12]

    Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar

    Assessing BERT’s ability to learn Italian syntax: a study on null-subject and agreement phenomena.Journal of Ambient Intelligence and Humanized Computing, 14(1):289–303. Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. 2021a. Bert & family eat word salad: Experiments with te...

    2010 arXiv

  2. [13]

    Finding Neurons in a Haystack: Case Stud- ies with Sparse Probing.Transactions on Machine Learning Research. John T. Hale and Milo ˇs Stanojevi ´c. 2024. Do LLMs learn a true syntactic universal? InProceedings of the 2024 Conference on Empirical Methods in Natu- ral Language P...

    2024 arXiv

  3. [15]

    InProceed- ings of the 21st Workshop on Multiword Expressions (MWE 2025), pages 7–13, Albuquerque, New Mex- ico, U.S.A

    Probing Internal Representations of Multi- Word Verbs in Large Language Models. InProceed- ings of the 21st Workshop on Multiword Expressions (MWE 2025), pages 7–13, Albuquerque, New Mex- ico, U.S.A. Association for Computational Linguis- tics. Josef Klafka and Allyson Ettinge...

    2025

  4. [16]

    InProceedings of the 57th Annual Meet- ing of the Association for Computational Linguis- tics, pages 5729–5739, Florence, Italy

    Empirical linguistic study of sentence em- beddings. InProceedings of the 57th Annual Meet- ing of the Association for Computational Linguis- tics, pages 5729–5739, Florence, Italy. Association for Computational Linguistics. Daria Kryvosheieva and Roger Levy. 2025. Controlled ...

    2025

  5. [17]

    Lower Perplexity is Not Always Human-Like. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 5203–5217, Online. Association for C...

    2024

  6. [19]

    Soo-Hwan Lee and Sebastian Schuster

    (AL)BERT Down the Garden Path: Psy- cholinguistic Experiments for Pre-trained Language Models.Korean Journal of English Language and Linguistics, 22:1033–1050. Soo-Hwan Lee and Sebastian Schuster. 2022. Can lan- guage models capture syntactic associations without surface cues?...

    2022

  7. [21]

    InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy

    Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Compu- tational Linguistics. R. Thomas McCoy...

    2024 arXiv

  8. [22]

    InProceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 518–528, Punta Cana, Dominican Republic

    Exploratory Model Analysis Using Data- Driven Neuron Representations. InProceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 518–528, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics. Subba Reddy...

  9. [23]

    Joint processing of linguistic properties in brains and language models. OpenAI. 2025. Chatgpt (gpt-4, gpt-5).https:// chat.openai.com. Large language model. Yulia Otmakhova, Karin Verspoor, and Jey Han Lau

    2025

  10. [24]

    InProceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 27–35, Seat- tle, Washington

    Cross-linguistic Comparison of Linguistic Feature Encoding in BERT Models for Typologi- cally Different Languages. InProceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 27–35, Seat- tle, Washington. Association for Compu...

    2021

  11. [25]

    Association for Compu- tational Linguistics

    Out of Order: How important is the sequen- tial order of words in a sentence in Natural Language Understanding tasks? InFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1145–1160, Online. Association for Compu- tational Linguistics. Tiago Pimen...

    2021

  12. [26]

    Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling. InProceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4375–4391, Seattle, United States. Association f...

    2022

  13. [27]

    Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg

    Grammaticality representation in ChatGPT as compared to linguists and laypeople.Humanities and Social Sciences Communications, 12(1):1–15. Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual Interventions Re- veal the Causal Effect of Relative C...

    2021

  14. [28]

    Rudolf Rosa and David Mareˇcek

    A Primer in BERTology: What We Know About How BERT Works.Transactions of the Asso- ciation for Computational Linguistics, 8:842–866. Rudolf Rosa and David Mareˇcek. 2019. Inducing Syn- tactic Trees from BERT Representations.arXiv preprint. ArXiv:1906.11511. Subhro Roy, Sam Tho...

    2019 arXiv

  15. [29]

    Carolin M

    Transformer Grammars: Augmenting Trans- former Language Models with Syntactic Inductive Biases at Scale.Transactions of the Association for Computational Linguistics, 10:1423–1439. Carolin M. Schuster and Simon Hegelich. 2022. From BERT‘s Point of View: Revealing the Prevailin...

    2022

  16. [30]

    Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1589–1598, Seattle, United Stat...

    2022

  17. [31]

    Dhanasekar Sundararaman, Vivek Subramanian, Guoyin Wang, Shijing Si, Dinghan Shen, Dong Wang, and Lawrence Carin

    A Latent-Variable Model for Intrinsic Prob- ing.Proceedings of the AAAI Conference on Artifi- cial Intelligence, 37(11):13591–13599. Dhanasekar Sundararaman, Vivek Subramanian, Guoyin Wang, Shijing Si, Dinghan Shen, Dong Wang, and Lawrence Carin. 2019. Syntax-Infused Transform...

    2019 arXiv

  18. [32]

    Anushka Swarup, Avanti Bhandarkar, Ronald Wilson, Tianyu Pan, and Damon Woodard

    Interpreting Language Models Through Knowledge Graph Extraction. Anushka Swarup, Avanti Bhandarkar, Ronald Wilson, Tianyu Pan, and Damon Woodard. 2025. From syn- tax to semantics: Evaluating the impact of linguis- tic structures on LLM-based information extraction. InProceedin...

    2025 arXiv

  19. [33]

    Quantity doesn’t buy quality syntax with neural language models. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5831–5837, Hong Kong, Chi...

    2019

  20. [34]

    InPro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 5604– 5620, Online and Punta Cana, Dominican Republic

    Controlled evaluation of grammatical knowl- edge in Mandarin Chinese language models. InPro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 5604– 5620, Online and Punta Cana, Dominican Republic. Association for Computational Lingui...

    2021

  21. [35]

    InProceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 932–948, Online and Punta Cana, Dominican Republic

    Frequency effects on syntactic rule learn- ing in transformers. InProceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 932–948, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. Leonie Weissweiler...

    2021

  22. [36]

    Shijie Wu and Mark Dredze

    Bigger is not always better: The impor- tance of human-scale language modeling for psy- cholinguistics.Journal of Memory and Language, 144:104650. Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Be- cas: The Surprising Cross-Lingual Effectiveness of BERT. InProceedings of the 20...

    2019

  23. [37]

    InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 7070–7083, Vienna, Austria

    A Systematic Study of Compositional Syn- tactic Transformer Language Models. InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 7070–7083, Vienna, Austria. Associa- tion for Computational Linguistics. Yi...

    2020

  24. [38]

    ArXiv:2311.08287

    How Well Do Large Language Mod- els Understand Syntax? An Evaluation by Ask- ing Natural Language Questions.arXiv preprint. ArXiv:2311.08287. Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuail- iang Zhang. 2020. LIMIT-BERT : Linguistics In- formed Multi-Task BERT. InFindings of...

    2020 arXiv

  25. [42]

    main findings

    for structured information extraction on 5 variables, model type, syntactic phenomena, ex- perimental materials, method name, and main syn- tactic findings, listed in Table 3. Each paper was first analysed using Note- bookLM (Google Labs, 2023) by posing a se- ries of targeted...

    2023

  26. [43]

    The AI-assisted summaries tend to be more de- tailed and insightful for keyword searches, pro- viding richer context

    on a scale from 0 (completely unrelated) to 5 (equivalent), with a median score of¯x= 4.1. The AI-assisted summaries tend to be more de- tailed and insightful for keyword searches, pro- viding richer context. By contrast, the manually collected information was generally shorte...

    2019

  27. [2016]

    Nelson F

    Assessing the ability of LSTMs to learn syntax-sensitive dependencies.Transactions of the Association for Computational Linguistics, 4:521– 535. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Lin- guistic knowledge and transferabilit...

    2019 arXiv

  28. [2017]

    Cognitive Science, 41(5):1202–1241

    Grammaticality, acceptability, and probabil- ity: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241. Jonghyun Lee and Jeong-Ah Shin. 2023. Decoding BERT’s Internal Processing of Garden-Path Struc- tures through Attention Maps.Korean Journal of En...

    2023

  29. [2019]

    Associa- tion for Computational Linguistics

    What Does BERT Learn about the Structure of Language? InProceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 3651–3657, Florence, Italy. Associa- tion for Computational Linguistics. Jae-young Jo and Sung-Hyon Myaeng. 2020. Roles and ...

    2020 arXiv

  30. [2020]

    InProceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4623–4637, Online

    On the Cross-lingual Transferability of Monolingual Representations. InProceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4623–4637, Online. As- sociation for Computational Linguistics. Nicholas Asher, Swarnadeep Bhar, Akshay Chatu...

    2023 arXiv

  31. [2021]

    InProceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 263–276, Online

    Syntactic Perturbations Reveal Representa- tional Correlates of Hierarchical Phrase Structure in Pretrained Language Models. InProceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 263–276, Online. Associ- ation for Computational Linguistics...

    2021

  32. [2022]

    Chiara Alzetta, Alessio Miaschi, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni

    Probing BERT on Syntactic Representation Paradigms: Dependency vs Constituency Tree. Chiara Alzetta, Alessio Miaschi, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. 2025. Parallel Trees: a novel resource with aligned de- pendencyand constituency syntactic repre...

    2025

  33. [2023]

    InProceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 6649– 6663, Toronto, Canada

    Prompting Language Models for Linguis- tic Structure. InProceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 6649– 6663, Toronto, Canada. Association for Computa- tional Linguistics. Gemma Boleda. 2025. Llms a...

    2025

  34. [2024]

    InProceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 14638–14663, Bangkok, Thailand

    CausalGym: Benchmarking causal inter- pretability methods on linguistic tasks. InProceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 14638–14663, Bangkok, Thailand. As- sociation for Computational Linguisti...

    2024

  35. [2025]

    in silico: Size mat- ters but Larger Language Models still do not com- prehend language on a par with humans due to impenetrable semantic reference.PLOS ONE, 20(7):e0327794

    Language in vivo vs. in silico: Size mat- ters but Larger Language Models still do not com- prehend language on a par with humans due to impenetrable semantic reference.PLOS ONE, 20(7):e0327794. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-...

    2019 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.