Pith. sign in

REVIEW 3 major objections 5 minor 45 references

Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BERT has systematic but uneven knowledge of NPI licensing, and the evaluation method changes what knowledge appears.

desk verdict Useful multi-method comparison shows evaluation method shapes conclusions about BERT's NPI knowledge, despite weak label validation and missing error bars. read the letter →

arxiv 1909.02597 v2 pith:YSYNXCZP submitted 2019-09-05 cs.CL

Alex Warstadt , Yu Cao , Ioana Grosu , Wei Peng , Hagen Blix , Yining Nie , Anna Alsop , Shikha Bordia
show 8 more authors
Haokun Liu Alicia Parrish Sheng-Fu Wang Jason Phang Anhad Mohananey Phu Mon Htut Paloma Jeretič Samuel R. Bowman
This is my paper · ORCID
classification cs.CL
keywords BERTnegativepolarityitemsacceptabilityjudgmentsminimalpairsprobingclassifiersmaskedlanguagemodelinggrammaticalknowledgeevaluationmethodology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether BERT, a sentence representation model, knows the grammar of negative polarity items (NPIs)—words like 'any' and 'ever' that are only acceptable in negative or similar licensing environments. The authors build a generated dataset of 136,000 sentences that independently manipulate three licensing features (presence of a licensor, presence of an NPI, and whether the NPI falls inside the licensor's scope) across nine licensing environments, and then evaluate BERT with five different methods: Boolean acceptability classification, absolute minimal pairs, gradient minimal pairs, cloze tests, and feature probing. A sympathetic reader would care because the paper argues that the answer to 'does BERT know this grammar?' is not a single yes or no: BERT shows near-perfect sensitivity on gradient comparisons, but absolute judgments and probing reveal that its knowledge is uneven across features and environments. The conclusion is methodological: a single evaluation task can misrepresent a model's grammatical knowledge, so multiple complementary methods are needed.

What carries the argument

The load-bearing object is a generated dataset of 136,000 English sentences organized into nine NPI licensing environments (adverbs, conditionals, determiner negation, sentential negation, only, quantifiers, questions, simple questions, superlatives), each following a 2×2×2 paradigm over three Boolean features: licensor presence, NPI presence, and scope. Each paradigm yields minimal pairs isolating one feature at a time, with a reduced 2×2 paradigm for simple questions. Around this dataset, the paper constructs five evaluation methods: a Boolean acceptability classifier, an absolute minimal-pair test, a gradient minimal-pair test, an unsupervised cloze test using BERT's masked language modeling head, and probing classifiers that predict the three metadata features from frozen representations. The dataset is what makes the method comparison possible, and the divergence among the five methods is the evidence for the paper's main conclusion.

What would settle it

Replace the template labels with a fully human-validated set in which every sentence is rated by multiple native speakers, then rerun the five methods: if the discrepancies between gradient and absolute results disappear, those discrepancies were artifacts of label noise; if they persist, they reflect genuine properties of BERT.

Watch

Extended reading notes

Core claim

The central claim is that BERT has systematic knowledge of all the features needed to judge NPI sentences—licensor presence, NPI presence, and scope—but that this knowledge is not equal across features and does not behave like the Boolean acceptability contrast it supports. On gradient minimal pairs, where the model only has to rank the acceptable sentence above the unacceptable one, BERT is near perfect across almost all environments; the bag-of-words baseline also reaches ceiling on detecting licensor and NPI presence, apparently through word co-occurrence. Under the stricter absolute minimal-pair measure and in probing classifiers, BERT's performance drops, and scope detection—knowing whether the NPI is inside the licensor's syntactic scope—is the weakest ability. Additional fine-tuning on NLI or CCG data does not improve BERT's NPI knowledge, contrary to what prior work on structurally supervised LSTMs would suggest.

Load-bearing premise

The template-generated labels correctly instantiate real English NPI acceptability; if the labels are noisy or confounded, the measured differences between methods could reflect label artifacts rather than BERT's grammatical knowledge.

Editorial extensions

If this is right

  • A near-perfect score on gradient minimal pairs is not enough to conclude that a model has categorical grammatical knowledge, because absolute judgments reveal gaps.
  • The same model can look near-perfect under one method and substantially weaker under another, so claims about grammatical knowledge should report the evaluation method.
  • BERT's scope detection is the weakest of the three NPI features; tasks that isolate long-range structural dependencies will expose limitations that lexical cues hide.
  • Co-occurrence statistics can explain part of the apparent NPI knowledge: the bag-of-words baseline detects licensor and NPI presence at ceiling while failing on scope.
  • Intermediate fine-tuning on natural language inference or CCG data does not transfer to NPI licensing for BERT, so structural supervision gains found for LSTMs do not automatically carry over.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If method-dependence generalizes beyond NPIs, then 'a model knows grammar X' is best understood as a relation between the model and a family of probes, not a fixed internal fact (this is my editorial inference, not the paper's claim).
  • A natural extension is to run the same five-method battery on other non-local dependencies, such as reflexive licensing or filler-gap dependencies, to test whether the observed asymmetry between gradient and absolute measures is a general property of transformer language models.
  • The data validation results suggest a testable correction: building a fully human-validated subset of the generated sentences and rerunning all five methods would show whether the method discrepancies shrink when label noise is removed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares five methods for evaluating grammatical knowledge in sentence encoders, using English NPI licensing as a case study. The authors introduce a generated dataset of 136,000 sentences labeled for licensor presence, NPI presence, and scope, and evaluate BERT-large (with several fine-tuning configurations) and a GloVe bag-of-words baseline via Boolean acceptability classification, absolute and gradient minimal-pair tests, an unsupervised cloze test, and feature probing. The results show that BERT performs near ceiling on gradient minimal pairs but markedly worse on absolute minimal pairs and on scope detection in probing, with systematic variation across licensing environments. The paper concludes that different evaluation methods reveal different aspects of a model's grammatical knowledge and that BERT's knowledge of NPI licensing is real but unequal across features and environments.

Significance. If the empirical claims hold, the paper makes a valuable methodological contribution: it provides one of the first direct comparisons of five evaluation paradigms on a single linguistic phenomenon, and it ships the generated data, generation code, and a simple baseline. The unsupervised cloze control and the handcrafted Wilcox et al. (2019) test set are useful external checks. The main risk is that the central conclusion depends on the validity of the automatically generated acceptability labels, which the paper's own validation only partially supports, and on single-run point estimates without uncertainty quantification. With the validity and stability concerns addressed, the contribution would be solid and likely influential for future work on evaluating linguistic knowledge in pretrained encoders.

major comments (3)
  1. [§4, Table 4] The MTurk validation covers only 500 of 136,000 generated sentences, and the per-environment separation is weak exactly in the environments where BERT's performance is lowest: for Conditionals, 37.5% of sentences labeled unacceptable were rated acceptable and only 50.0% of intended-acceptable sentences were rated acceptable; for Simple Questions the corresponding numbers are 33.3% and 63.0%. Because these same labels are used for fine-tuning and evaluation in all five methods, the environment-level differences in the results (e.g., acceptability MCC 0.58 for SMP-Q under All-but-1 NPI in Figure 1) could partly reflect label noise rather than BERT's NPI knowledge. The authors should report per-environment inter-rater agreement and the exact Wilcoxon test procedure, and ideally verify the main environment-level claims on a validated subset or under alternative binarization thresholds.
  2. [§6, Fig. 2 and Fig. 4] All reported scores are single point estimates without confidence intervals or multiple runs. The paper's key claims are about relative differences—for example, near-ceiling gradient minimal-pair accuracy versus much lower absolute minimal-pair accuracy, and low scope-detection MCC in probing. It is unclear whether the environment-level differences (e.g., the drop in Simple Questions) are stable or within run-to-run noise. The authors should provide variance estimates via repeated training with different seeds or bootstrap resampling over test items.
  3. [§4, Data validation; §6, Acceptability Judgments] The conclusion that 'BERT has systematic knowledge of NPI licensing' is only partially supported by the external checks. The unsupervised cloze test is a useful control, but it applies only to length-matched minimal pairs with a single token difference, and the only naturalistic test, the Wilcox et al. (2019) handcrafted set, yields modest MCC values for BERT (0.55 under CoLA fine-tuning and lower under NPI fine-tuning, Figure 1). The paper should either temper the conclusion to acknowledge that the evidence is primarily on the template-generated distribution, or add a stronger validation of the labels in the environments that drive the main claims.
minor comments (5)
  1. [§2, Related Work] There is a typo in the paragraph on Marvin and Linzen: 'sentencew' should be 'sentences'.
  2. [§1, example (1)] The example 'Mary has n’teaten any cookies' is missing a space; it should read 'Mary hasn't eaten any cookies.'
  3. [§4, Table 4 caption] The definition of 'Diff' is not consistent with the displayed numbers: for Adverb, the difference between 61.67 and 8.33 is 53.34, not 70.00. The authors should correct the table or clarify how 'Diff' is computed.
  4. [§4, Data validation] The paper should state explicitly whether the reported agreement percentages (81.3%, 85.2%, 82.8%) are based on per-sentence majority votes or on individual participant ratings, and should describe the sampling procedure for the 500 validated sentences in more detail.
  5. [§6, Cloze Test] The cloze test uses only minimal pairs that are length-matched and differ in exactly one token; this restriction should be stated in the main text when interpreting the cloze results, since it limits the comparison with the other minimal-pair conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's five evaluation methods test BERT against externally constructed, human-validated data, and no prediction reduces to a fitted input or self-citation chain.

full rationale

The paper's central claim is that BERT has systematic but feature-dependent knowledge of NPI licensing, measured with five methods on a generated dataset. The dataset labels are produced by an explicit template grammar (Section 4) and are validated against human judgments (Table 4); they are inputs to the evaluation, not derived from BERT's outputs. None of the five methods fits a parameter and then reports that same parameter as a prediction: the acceptability classifiers are trained on one split and tested on held-out sentences (Section 5), the minimal-pair and cloze measures compare relative probabilities on unseen items (Section 6), and the probing classifiers are trained and evaluated on the meta-data features independently of the acceptability labels (Section 4). The unsupervised cloze test and probing on plain BERT (with no NPI fine-tuning) provide evidence that does not depend on the supervised training configurations. Self-citations (CoLA, jiant) are used as tools and prior datasets rather than as load-bearing justifications, and no uniqueness theorem or ansatz is imported to forbid alternatives. The MTurk validation limitation concerns label quality, which is a correctness risk, not a circularity, and the paper's conclusion does not reduce by construction to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on the template-based dataset and the assumption that the three Boolean features define NPI licensing. The paper partially validates the dataset with MTurk but the validation is small and imperfect. No free parameters are fitted to derive a result; the listed choices are experimental design decisions. No invented entities are introduced.

free parameters (2)
  • Likert binarization threshold = 3/4
    MTurk responses 1 to 6 are split at 4 to form Boolean acceptability labels; this boundary is arbitrary and affects validation agreement rates.
  • Fine-tuning hyperparameters = not reported
    BERT fine-tuning uses unspecified learning rate, epochs, batch size, and random seeds; results are point estimates.
assumptions (4)
  • domain assumption NPI licensing in English is adequately characterized by the Boolean features licensor presence, NPI presence, and scope.
    Section 4 defines the 2x2x2 paradigm; if this decomposition is wrong or incomplete, the probing and minimal pair tasks do not measure the proposed knowledge.
  • domain assumption Generated template sentences are grammatical or unacceptable as labeled for English speakers.
    Section 4 Data validation partially supports this with 82.8% agreement, but the assumption is load-bearing for all experiments.
  • domain assumption Off-the-shelf BERT-large, GloVe embeddings, and the jiant, AllenNLP, and HuggingFace implementations are used as-is and are reliable tools.
    Section 5 Experimental Settings; any implementation bug or version mismatch affects all results.
  • domain assumption Probing classifiers can recover latent linguistic features from frozen representations.
    Section 3 Feature Probing; a probe's accuracy reflects the information content of the representation, a standard but nontrivial assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs." pith.science (2026). https://pith.science/paper/YSYNXCZP

@misc{pith2026190902597,
  author       = {Pith},
  title        = {Pith review of: Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YSYNXCZP}},
  note         = {Machine review of arXiv:1909.02597}
}
read the original abstract

Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge. We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models. We use a single linguistic phenomenon, negative polarity item (NPI) licensing in English, as a case study for our experiments. NPIs like "any" are grammatical only if they appear in a licensing environment like negation ("Sue doesn't have any cats" vs. "Sue has any cats"). This phenomenon is challenging because of the variety of NPI licensing environments that exist. We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments. We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods. We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.

Figures

Figures reproduced from arXiv: 1909.02597 by the authors.

Figure 1
Figure 1. Results from the acceptability judgment experiment in MCC. The columns indicate evaluation tests, and [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Results from the minimal pair test. The top section shows the average accuracy for detecting the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Results of BERT MLM performance in the cloze test. The top section shows the average accuracy for detecting the presence of the NPI; the bottom sec￾tion show the accuracy for detecting the presence of the licensor. The columns represent different licensing environments Cloze Test The results ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Results of probing classification on NPI presence, licensor presence, and whether the NPI is in the scope [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Results from the acceptability judgment experiment in MCC. The columns indicate evaluation tests, and [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Results from minimal pair test for the NPI-presence contrast. The smaller diagrams of each sector show [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Results from minimal pair test for the licensor-presence contrast. The smaller diagrams of each sector [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Results from minimal pair test for the scope contrast. The smaller diagrams of each sector show [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Results of probing classification on NPI presence, licensor presence, and scope detection, shown in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 30 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04207

    arXiv 2016

  4. [4]

    Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

    Samuel R. Bowman, Ellie Pavlick, Edouard Grave, Benjamin Van Durme, Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, and Berlin Chen. 2018. http://arxiv.org/abs/1812.10860 Looking for elmo's friends: Sentence-level pretraining beyond language modeling . ...

    work page Pith review arXiv 2018

  5. [5]

    Gennaro Chierchia. 2013. Logic in grammar: P olarity, free choice, and intervention . Oxford U niversity P ress

    work page 2013

  6. [6]

    Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press

    work page 1965

  7. [7]

    Noam Chomsky. 1981. Lectures on Government and Binding. Number 9 in Studies in Generative Grammar. Foris Publications , Dordrecht, Holland ; Cinnaminson, [N.J.]

    work page 1981

  8. [8]

    Noam Chomsky. 1995. The Minimalist Program . Number 28 in Current Studies in Linguistics. The MIT Press , Cambridge, Mass

    work page 1995

Show all 45 references
  1. [9]

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo \" c Barrault, and Antoine Bordes. 2017. http://arxiv.org/abs/1705.02364 Supervised learning of universal sentence representations from natural language inference data . CoRR, abs/1705.02364

    2017 arXiv

  2. [10]

    Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \" c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.01070

    2018 arXiv

  3. [11]

    Ishita Dasgupta, Demi Guo, Andreas Stuhlm \"u ller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302

    2018 arXiv

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT : P re-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.04805

    2018 arXiv

  5. [13]

    Allison Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In P roceedings of the F irst W orkshop on E valuating V ector S pace R epresentations for NLP , pages 134--139. Association for Com...

    2016

  6. [14]

    Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th I nternational C onference on C omputational L inguistics , pages 1790--1801. Association for Computational Linguistics

    2018

  7. [15]

    Gilles Fauconnier. 1975. Polarity and the scale principle. Proceedings of the Chicago Linguistics Society, 11:188--199

    1975

  8. [16]

    Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), ...

    2018

  9. [17]

    Anastasia Giannakidou. 1994. The semantic licensing of NPI s and the M odern G reek subjunctive. Language and C ognition , 4:55--68

    1994

  10. [18]

    Anastasia Giannakidou. 1998. Polarity sensitivity as (non) veridical dependency, volume 23. John Benjamins Publishing

    1998

  11. [19]

    Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem H. Zuidema. 2018. http://arxiv.org/abs/1808.08079 Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information . CoRR, abs/1808.08079

    2018 arXiv

  12. [20]

    Julia Hockenmaier and Mark Steedman. 2007. CCG bank: A corpus of CCG derivations and dependency structures extracted from the P enn T reebank. Computational L inguistics , 33(3):355--396

    2007

  13. [21]

    Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? on the ability of lstms to understand negative polarity items. arXiv preprint arXiv:1808.10627

    2018 arXiv

  14. [22]

    Nirit Kadmon and Fred Landman. 1993. Any. Linguistics and P hilosophy , 16(4):353--422

    1993

  15. [23]

    Katharina Kann, Alex Warstadt, and Adina Williams. 2019. Verb argument structure alternations in word and sentence embeddings. In P roceedings of the Society for C omputation in L inguistics ( SCiL ) , pages 287--297

    2019

  16. [24]

    Manfred Krifka. 1995. The semantics and pragmatics of polarity items. Linguistic A nalysis , 25(3-4):209--257

    1995

  17. [25]

    William Ladusaw. 1979. Negative polarity items as inherent scope relations. Unpublished Ph.D. Dissertation, University of Texas at Austin

    1979

  18. [26]

    Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202--1241

    2017

  19. [27]

    Marcia Christine Linebarger. 1980. The grammar of negative polarity. Ph.D. thesis, Massachusetts Institute of Technology

    1980

  20. [28]

    Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521--535

    2016

  21. [29]

    Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. http://arxiv.org/abs/1901.11504 Multi-task deep neural networks for natural language understanding . CoRR, abs/1901.11504

    2019 arXiv

  22. [30]

    Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank

    1993

  23. [31]

    Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 C onference on E mpirical M ethods in N atural L anguage P rocessing , pages 1192--1202

    2018

  24. [32]

    Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442--451

    1975

  25. [33]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. http://www.aclweb.org/anthology/D14-1162 Glo V e: G lobal vectors for word representation . In Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543

    2014

  26. [34]

    Jason Phang, Thibault F\'evry, and Samuel R. Bowman. 2018. Sentence encoders on STILTs : Supplementary training on intermediate labeled-data tasks. In arXiv preprint

    2018

  27. [35]

    Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526--1534

    2016

  28. [36]

    Asa Cooper Stickland and Iain Murray. 2019. http://arxiv.org/abs/1902.02671 BERT and PALs : Projected attention layers for efficient adaptation in multi-task learning . CoRR, abs/1902.02671

    2019 arXiv

  29. [37]

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. https://openreview.net/forum?id=SJzSgnRcKX What do you learn from context? P robing for sentence structure in conte...

    2019

  30. [38]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks...

    2018

  31. [39]

    Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R

    Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Najoung Kim, Phu Mon Htut, Thibault F' e vry, Berlin Chen, Nikita Nangia, Haokun Liu, Anhad M...

    2019

  32. [40]

    Alex Warstadt and Samuel R Bowman. 2019. Grammatical analysis of pretrained sentence encoders with acceptability judgments. arXiv preprint arXiv:1901.03438

    2019 arXiv

  33. [41]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018. Neural network acceptability judgments. arXiv preprint 1805.12471

    2018 arXiv

  34. [42]

    Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballesteros, and Roger Levy. 2019. Structural supervision improves learning of non-local grammatical dependencies. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational L...

    2019

  35. [43]

    Frank Wilcoxon. 1945. http://www.jstor.org/stable/3001968 Individual comparisons by ranking methods . Biometrics Bulletin, 1(6):80--83

    1945

  36. [44]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo...

    2018

  37. [45]

    Frans Zwarts. 1998. Three types of polarity. In Plurality and Q uantification , pages 177--238. Springer

    1998

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.