Pith. sign in

REVIEW 4 major objections 4 minor 43 references

The paper claims that as language models grow, their memorized factual knowledge improves roughly twice as fast as their core linguistic competence, making small language-first models a viable design.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Across 23 models from 135M to 32B parameters, internal factual knowledge scales about twice as fast with model size as linguistic competence, supporting modular small-model-plus-retrieval systems.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful broad empirical sweep, but the central slope comparison is not statistically established; the paper deserves revision, not publication as is. the 4 major comments →

arxiv 2509.02225 v1 pith:63O25YZO submitted 2025-09-02 cs.CL

Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?

Jaime Collado-Monta\~nez , L. Alfonso Ure\~na-L\'opez , Arturo Montejo-R\'aez This is my paper
classification cs.CL
keywords linguistic competencefundamental language modelsmodel scalingfactual knowledgeCEFRzero-shot evaluationLLM evaluationretrieval augmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that linguistic competence and factual memorization scale differently with model size, and that this difference justifies a modular architecture called the Fundamental Language Model (FLM). Across 23 models ranging from 135M to 32B parameters, the authors find that internal factual knowledge grows about twice as fast as linguistic competence when scores are regressed against log parameter count. If true, this means a relatively small model can carry the grammar, vocabulary, and meaning needed for fluent language, while external retrieval tools supply facts. That would make language systems cheaper, easier to update, and less prone to hallucinated memorized content.

Core claim

The paper's central claim is that internal factual knowledge scales significantly faster with model size than linguistic competence. Using zero-shot evaluation across seven model families and benchmarks for lexical, grammatical, and semantic competence, external factual knowledge, and internal factual knowledge, the authors fit linear regressions of score against log parameter count. The fitted slope for internal factual knowledge is 0.059, more than double the 0.029 slope for linguistic competence, and model size explains 81% of internal-knowledge variance but only about 50% of linguistic variance. Quartile-based Mann-Whitney U tests show significant gains from small to medium and small to

What carries the argument

The central object is the Fundamental Language Model (FLM) paradigm: a proposed modular architecture in which a compact model handles lexical, grammatical, and semantic competence while factual retrieval is delegated to external tools. The empirical machinery is a comparison of scaling slopes: average zero-shot benchmark scores for each competence are regressed against log parameter count, and the slopes and R² values are compared, with Mann-Whitney U tests on size quartiles providing significance checks.

Load-bearing premise

The claim depends on the chosen benchmarks being clean measures of grammar, vocabulary, and semantics that do not secretly rely on world knowledge, and on zero-shot performance being a fair proxy for core linguistic competence.

What would settle it

Run the same size sweep on a linguistic battery designed to exclude factual confounds, using models matched for training-token counts; if linguistic performance keeps climbing past 10B parameters with a slope comparable to factual recall, the FLM claim would collapse.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim is correct, models in the 3–7B parameter range can serve as linguistic cores without losing most language ability.
  • Factual recall, not language, is what drives the demand for very large monolithic models, so external retrieval can carry that load.
  • A modular FLM design would let developers update factual knowledge without retraining the language component.
  • The non-significant medium-to-large performance step implies diminishing returns beyond roughly 10B parameters on these linguistic benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The apparent gap could partly be a benchmark artifact: WiC, BLiMP, MNLI, and similar tasks may saturate at moderate sizes, so a less saturated linguistic battery might show steeper scaling.
  • Because model size often correlates with training-token volume, the steeper factual-knowledge slope may reflect greater data exposure rather than parameter count alone; matching token budgets across sizes would test this.
  • A multilingual extension could reveal that morphologically rich languages need larger models for the same grammatical competence, which would shift the practical FLM size estimate upward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces the Fundamental Language Model (FLM) paradigm, arguing that linguistic competence should be decoupled from factual memorization. It evaluates 23 models (135M–32B) from seven families on three composite scores: linguistic competence (WiC, BLiMP, RTE, MNLI, QQP), external factual knowledge (LAMBADA, BoolQ, COPA, MultiRC, ReCoRD), and internal factual knowledge (TriviaQA, TruthfulQA). Based on regression slopes of composites against log(size) and Mann-Whitney tests, it claims internal factual knowledge grows significantly faster than linguistic competence, and concludes compact FLMs with external retrieval are viable.

Significance. If the empirical dissociation were established, the FLM proposal would be a useful contribution to model design and to the scaling-law literature. The paper uses standard benchmarks, public models, and reports per-task results in appendices, which is commendable. However, the central finding rests entirely on a statistically fragile comparison of slopes on arbitrary composite scales. The significance of the FLM proposal therefore depends on the re-analysis suggested below.

major comments (4)
  1. [§4.1.1 and Figure 2] The central claim is based on comparing regression slopes of two composite scores (0.059 vs 0.029). These composites are not on a common scale: LC is an average of accuracies from WiC, BLiMP, RTE, MNLI, QQP, while IFK averages TriviaQA accuracy, TruthfulQA_gen BLEU/ROUGE values, MC1 accuracy, and MC2 normalized probabilities. A slope in composite units is not interpretable across composites; a monotone rescaling of one component (e.g., using percentages instead of fractions for BLEU) changes the slope ratio. The paper also reports no standard errors, confidence intervals, or significance test for the slope difference. The claim that knowledge grows 'significantly faster' is therefore not supported by the reported statistics.
  2. [§4.1.1, Table 1] The regression uses log(Size) pooled across seven model families, confounding size with architecture, token budget, and data mix. Table 1 shows large family effects: Qwen2.5-32B has IFK 0.4288, lower than OLMo-2-1124-13B (0.4799) and OLMo-2-0325-32B (0.5784). Without a within-family analysis or a mixed-effects model with family as a random effect, the reported slopes may be artifacts of which families are included at each size.
  3. [§4.1.2, Table 2] The Mann-Whitney tests do not directly test the paper's comparative claim. They test, for each competence separately, whether score distributions differ between size bins; they never compare LC versus IFK scaling. The result that Medium vs Large is non-significant even for IFK (p=0.078) is hard to reconcile with the claim that IFK continues to scale steeply. The median-split gaps in §4.2 (39.5% vs 18.3%) are descriptive and inherit the same composite-scale problem. A direct interaction test or a joint model of standardized scores is needed.
  4. [§3, §3.1, Limitations] The linguistic benchmarks are assumed to isolate CEFR-style competence from factual knowledge, but no evidence is given that RTE, MNLI, or QQP are free of world-knowledge/commonsense confounds, and the paper's Limitations section acknowledges that language understanding often requires world knowledge. Moreover, BLiMP scores in Appendix A are near ceiling (0.69–0.83), which compresses the LC scale and can manufacture a slower slope. Per-benchmark scaling analyses (or at least ceiling diagnostics) are needed before the dissociation can be attributed to linguistic competence as opposed to benchmark saturation.
minor comments (4)
  1. [§4.1.1] Report the exact R² for linguistic competence and external factual knowledge, not just 'about 50%'. Also provide the fitted intercepts and model counts.
  2. [Table 2] The text says the table reports U statistics and p-values, but only p-values and significance are shown. Include the U statistics and sample sizes.
  3. [Figure 2] Add confidence bands to the regression lines and use distinct markers/colors for model families so the reader can see the family-level structure that the pooled regression ignores.
  4. [§4.2] For the median-split comparison, state the median size, the number of models in each group, and the standard errors of the mean scores. The phrase 'only shows' should be 'shows only'.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical evaluation with external benchmarks; the central slope comparison is a statistical summary, not a derivation from its own inputs.

full rationale

The paper's central claim—that internal factual knowledge scales more steeply with model size than linguistic competence—is an empirical finding obtained by running external benchmarks (WiC, BLiMP, RTE, MNLI, QQP, LAMBADA, BoolQ, COPA, MultiRC, ReCoRD, TriviaQA, TruthfulQA) on off-the-shelf models and then regressing averaged scores against log(size). No parameter is fitted to a subset of data and then reported as a prediction of a closely related quantity; the composites are simple means of externally defined benchmark scores, not quantities defined in terms of the conclusion. The FLM paradigm is a framing device articulated in the introduction, not an input to the measurement, and the paper does not invoke any self-citation, uniqueness theorem, or ansatz smuggled in through prior work by the same authors. The slope comparison in §4.1.1 is a legitimate (if statistically fragile) description of the collected data; concerns about commensurability of composite scales, missing confidence intervals, and confounds between model family and training data are threats to validity and interpretation, not circularity. The paper's own Limitations section explicitly acknowledges that separating linguistic competence from factual knowledge is difficult, which further indicates that the benchmark choices are assumptions rather than disguised conclusions. Under the stated rules, where no specific equation reduces to an input by construction and no fitted input is renamed as a prediction, the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces the FLM paradigm as a conceptual proposal, but it does not postulate new physical entities, forces, or conserved quantities. The main analytical assumptions are about benchmark validity, evaluation comparability, and statistical modeling, all of which are domain assumptions rather than invented entities. No free parameters are fitted to produce the central claim beyond the regression slopes, which are themselves the result rather than an input.

axioms (4)
  • domain assumption The selected benchmarks (WiC, BLiMP, RTE, MNLI, QQP) validly operationalize CEFR lexical, grammatical, and semantic competence independent of factual knowledge.
    Section 3.1 states these benchmarks 'minimize the influence of external factual knowledge', but no evidence is given that they are not confounded. This is load-bearing for separating linguistic from factual scaling.
  • domain assumption Zero-shot evaluation with the LM Evaluation Harness is a reliable and comparable measure of each competence across heterogeneous model families.
    The paper uses default harness protocols (Section 3) but does not control for differences in prompting, tokenization, or training objectives across models, which could affect scores independently of true competence.
  • domain assumption log(Size) is the correct functional form for comparing scaling slopes across competences.
    Section 4.1.1 assumes logarithmic scaling based on prior scaling law references (Kaplan et al., 2020), but no model comparison or residual analysis is performed to justify this choice for these particular tasks.
  • standard math The Mann-Whitney U test assumptions (independent samples, comparable distributions) are satisfied by grouping 23 models into three size quartiles.
    The test is applied in Section 4.1.2, but with small group sizes (around 5-6 models per extreme quartile) and multiple comparisons without correction, the statistical conclusions are fragile.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?." pith.science (2026). https://pith.science/paper/63O25YZO

@misc{pith2026250902225,
  author       = {Pith},
  title        = {Pith review of: Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63O25YZO}},
  note         = {Machine review of arXiv:2509.02225}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models offer impressive language capabilities but suffer from well-known limitations, including hallucinations, biases, privacy concerns, and high computational costs. These issues are largely driven by the combination of linguistic competence and factual memorization within a single monolithic model. This paper introduces and empirically supports the Fundamental Language Model (FLM) paradigm, which advocates for smaller, linguistically competent models that offload factual retrieval to external tools. We evaluate models ranging from 135M to 32B parameters across three dimensions: linguistic competence, external factual knowledge, and internal factual knowledge. Our findings reveal that while both linguistic competence and factual knowledge improve with scale, internal factual knowledge grows significantly faster, suggesting that model size is more closely tied to memorization than to core language ability. These results support a modular approach to language modeling, where compact, linguistically proficient models serve as the foundation for tool-augmented systems. The FLM paradigm offers a path toward more efficient, interpretable, and sustainable NLP solutions.

Figures

Figures reproduced from arXiv: 2509.02225 by Arturo Montejo-R\'aez, Jaime Collado-Monta\~nez, L. Alfonso Ure\~na-L\'opez.

Figure 1
Figure 1. Figure 1: Scores achieved against model size in million parameters. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Linear regressions for each competence against [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 19 canonical work pages · 1 internal anchor

  1. [1]

    Deepak Bhaskar Acharya, Karthigeyan Kuppan, and B Divya. 2025. Agentic ai: Autonomous intelligence for complex goals--a comprehensive survey. IEEE Access

    work page 2025

  2. [2]

    01. AI. 2025. https://arxiv.org/abs/2403.04652 Yi: Open foundation models by 01.ai . Preprint, arXiv:2403.04652

    Pith/arXiv arXiv 2025

  3. [3]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werr...

    Pith/arXiv arXiv 2025

  4. [4]

    Nathan Atox and Mason Clark. 2024. Evaluating large language models through the lens of linguistic proficiency and world knowledge: A comparative study. Authorea Preprints

    work page 2024

  5. [5]

    Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, and Yejin Choi. 2025. https://arxiv.org/abs/2501.17805 International ai safety report . Preprint, arXiv:2501.17805

    Pith/arXiv arXiv 2025

  6. [6]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    work page 2024

  7. [7]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL

    work page 2019

  8. [8]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. https://api.semanticscholar.org/CorpusID:8587959 The pascal recognising textual entailment challenge . In Machine Learning Challenges Workshop

    work page 2005

  9. [9]

    Vittoria Dentella, Fritz Guenther, and Evelina Leivada. 2024. https://arxiv.org/abs/2404.14883 Language in vivo vs. in silico: Size matters but larger language models still do not comprehend language on a par with humans . Preprint, arXiv:2404.14883

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Jianmei Dong. 2022. https://doi.org/10.47852/bonviewGHSS2022030313 A study on the relationship between language and thought based on sapir-whorf . Journal of Global Humanities and Social Sciences, 3(3):83–85

    work page doi:10.47852/bonviewghss2022030313 2022

  11. [11]

    Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759

    Pith/arXiv arXiv 2023

  12. [12]

    Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, et al. 2025. When one llm drools, multi-llm collaboration rules. arXiv preprint arXiv:2502.04506

    Pith/arXiv arXiv 2025

  13. [13]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. https...

    doi:10.5281/zenodo.12608602 2024

  14. [14]

    J \'e r \^o me Garnier-Brun, Marc M \'e zard, Emanuele Moscato, and Luca Saglietti. 2024. How transformers learn structured data: insights from hierarchical filtering. arXiv preprint arXiv:2408.15138

    Pith/arXiv arXiv 2024

  15. [15]

    Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma

    doi:10.34740/kaggle/m/3301 2024

  16. [16]

    Gordon, Zornitsa Kozareva, and Melissa Roemmele

    Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2011. https://api.semanticscholar.org/CorpusID:434646 Choice of plausible alternatives: An evaluation of commonsense causal reasoning . In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning

    2011

  17. [17]

    Susan J Hespos and Elizabeth S Spelke. 2004. Conceptual precursors to language. Nature, 430(6998):453--456

    work page 2004

  18. [18]

    Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox

    Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. https://arxiv.org/abs/2412.05149 Findings of the second babylm challenge: Sample-efficient pretraining on developmentally plausible corpora . Preprint, arXiv:2412.05149

    Pith/arXiv arXiv 2024

  19. [19]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computational Linguistics

    2017

  20. [20]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    Pith/arXiv arXiv 2020

  21. [21]

    Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL)

    work page 2018

  22. [22]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459--9474

    2020

  23. [23]

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.01100

    Pith/arXiv arXiv 2025

  24. [24]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2022.acl-long.229 2022

  25. [25]

    Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. 2024. https://arxiv.org/abs/2407.16154 Ddk: Distilling domain knowledge for efficient large language models . Preprint, arXiv:2407.16154

    Pith/arXiv arXiv 2024

  26. [26]

    AI @ Meta Llama Team. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    Pith/arXiv arXiv 2024

  27. [27]

    Diego Marconi. 1997. Lexical competence. MIT press

    work page 1997

  28. [28]

    Diego Marconi. 2020. Semantic competence. In The Routledge Handbook of Philosophy of Skill and Expertise, pages 409--418. Routledge

    work page 2020

  29. [29]

    Radislav Millrood. 2014. https://doi.org/10.1016/j.sbspro.2014.10.147 Cognitive models of grammatical competence of students . Procedia - Social and Behavioral Sciences, 154:259--262. THE XXV ANNUAL INTERNATIONAL ACADEMIC CONFERENCE, LANGUAGE AND CULTURE, 20-22 October 2014

    work page doi:10.1016/j.sbspro.2014.10.147 2014

  30. [30]

    Vivi Nastase and Paola Merlo. 2024. Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification. In Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), pages 203--214

    work page 2024

  31. [31]

    Team OLMo. 2024. https://arxiv.org/abs/2501.00656 2 OLMo 2 Furious . Preprint, arXiv:2501.00656

    Pith/arXiv arXiv 2024

  32. [32]

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. https://arxiv.org/abs/1606.06031 The lambada dataset: Word prediction requiring a broad discourse context . Preprint, arXiv:1606.06031

    Pith/arXiv arXiv 2016

  33. [33]

    Julia M Penn. 2014. Linguistic relativity versus innate ideas: The origins of the Sapir-Whorf hypothesis in German thought, volume 120. Walter de Gruyter

    work page 2014

  34. [34]

    Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. https://doi.org/10.18653/v1/N19-1128 W i C : the word-in-context dataset for evaluating context-sensitive meaning representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and S...

    doi:10.18653/v1/n19-1128 2019

  35. [35]

    David Premack. 1959. Toward empirical behavior laws: I. positive reinforcement. Psychological review, 66(4):219

    work page 1959

  36. [36]

    Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842--866

    work page 2021

  37. [37]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36:55565--55581

    work page 2023

  38. [38]

    Julius Steuer, Marius Mosbach, and Dietrich Klakow. 2023. Large gpt-like models are bad babies: A closer look at the relationship between linguistic competence and psycholinguistic measures. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 142--157

    work page 2023

  39. [39]

    Falcon-LLM Team. 2024 a . https://huggingface.co/blog/falcon3 The falcon 3 family of open models

    work page 2024

  40. [40]

    Qwen Team. 2024 b . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  41. [41]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. https://doi.org/10.1162/tacl\_a\_00321 Blimp: The benchmark of linguistic minimal pairs for english . Transactions of the Association for Computational Linguistics, 8:377--392

    doi:10.1162/tacl 2020

  42. [42]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. http://aclweb.org/anthology/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112--...

    2018

  43. [43]

    Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. https://arxiv.org/abs/1810.12885 Record: Bridging the gap between human and machine commonsense reading comprehension . CoRR, abs/1810.12885

    Pith/arXiv arXiv 2018

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.