Pith. sign in

REVIEW 4 major objections 6 minor 30 references

A MISMATCHED Benchmark for Scientific Natural Language Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces MISMATCHED, a human-annotated scientific NLI benchmark drawn from psychology, engineering, and public health, and argues that adding implicit-relation sentence pairs to training improves scientific NLI models.

desk verdict Useful OOD benchmark for scientific NLI; the implicit-relation claim is suggestive but only partly secured by the current pseudo-labeling setup. read the letter →

arxiv 2506.04603 v1 pith:K6NNO4SM submitted 2025-06-05 cs.CL

Firoz Shaik , Mobashir Sadat , Nikita Gautam , Doina Caragea , Cornelia Caragea This is my paper
classification cs.CL
keywords scientificnaturallanguageinferenceout-of-domainevaluationMISMATCHEDbenchmarkimplicitrelationsdistantsupervisionhumanannotationnon-CSdomains
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Scientific natural language inference has so far been tested only on computer-science text. This paper introduces MISMATCHED, a human-annotated evaluation benchmark of 2,700 sentence pairs from psychology, engineering, and public health, with no training data, so any model must generalize out of domain to score well. The paper establishes baselines showing the best small language model reaches 78.17% Macro F1 and the best open-source LLM 57.16%, leaving substantial room for improvement. It further argues that training on adjacent sentence pairs whose scientific-NLI relation is implicit, because no linking phrase is present, improves performance on scientific NLI, with SciBERT rising to 79.66% on MISMATCHED. The contribution is a harder, broader testbed and evidence that implicit relations are a usable training resource.

What carries the argument

The load-bearing mechanism is the four-way scientific NLI typology (entailment, reasoning, contrasting, neutral) combined with an implicit-relation extraction pipeline. Existing training sets are built by distant supervision: when the second sentence begins with a linking phrase such as 'However' or 'Therefore', the pair is labeled contrasting or reasoning and the phrase is removed; MISMATCHED applies the same recipe to non-CS abstracts, then keeps only pairs where human annotators agree with the automatic label. For the implicit-relation experiments, adjacent sentence pairs with no linking phrase are pseudo-labeled by a SciBERT model fine-tuned on the combined CS training data, filtered at 0.6 confidence, and then used either to continue fine-tuning SciBERT before explicit training or as few-shot exemplars for Phi-3. The pseudo-labeled implicit pairs are the mechanism claimed to expose models to more diverse discourse structure.

What would settle it

Have human annotators label a sample of the implicit sentence pairs used in training and measure agreement with the SciBERT pseudo-labels; agreement near chance would show that the reported implicit-relation gains are a self-training artifact rather than evidence about implicit scientific NLI.

Watch

Extended reading notes

Core claim

The paper's central claim is that existing scientific NLI datasets, SCINLI and MSCINLI, cover only computer science, so their high in-domain scores may overstate how well models understand scientific text. MISMATCHED supplies 2,700 human-annotated dev/test pairs (300 dev, 2,400 test), balanced across classes and across psychology, engineering, and public health, and constructed by distant supervision with linking phrases followed by human filtering with a Fleiss kappa of 0.72. On this out-of-domain testbed, the best fine-tuned SLM (SciBERT trained on the combined SCINLI+MSCINLI training sets) obtains 78.17% Macro F1, while the best open-source LLM baseline (Phi-3 in a few-shot setting) obtains 57.16% and the best proprietary LLM baseline (Gemini-1.5-Pro) reaches 62.95%. The paper also claims that implicit relations, defined as adjacent sentence pairs without linking phrases and pseudo-labeled by a SciBERT model at a confidence threshold of 0.6, improve results when added to training: SciBERT's Macro F1 on MISMATCHED rises from 78.17% to 79.66%, and Phi-3 improves on SciNLI when implicit MISMATCHED examples are used as few-shot exemplars.

Load-bearing premise

The implicit-relation result assumes that the model-generated labels for adjacent sentence pairs, kept only when the model is at least 60% confident, are genuine implicit scientific-NLI relations rather than just echoes of the model's own habits.

Editorial extensions

If this is right

  • MISMATCHED gives scientific NLI research a reusable out-of-domain testbed: any model trained on SCINLI or MSCINLI can now be checked for robustness beyond computer science, and current scores leave clear headroom.
  • Combining the SCINLI and MSCINLI training sets consistently improves out-of-domain performance for SLMs, suggesting that training diversity is a direct lever for OOD generalization.
  • Implicit relations, which are about twice as abundant as explicit ones in the extracted corpora, can be harvested cheaply and improve model performance, pointing to a new training signal beyond linking-phrase supervision.
  • Class-wise results identify reasoning as the hardest relation, with the confusion matrix showing reasoning frequently mistaken for entailment, so progress on reasoning should move the overall benchmark score.
  • The hypothesis-only baseline drops by roughly 22 points, indicating that MISMATCHED cannot be solved from stylistic artifacts in the hypothesis alone and that premise understanding is required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pseudo-labeling assumption holds, the same recipe could expand scientific NLI training to any unlabeled non-CS corpus, generating implicit-relation training data without additional human annotation.
  • The hypothesis-only drop suggests MISMATCHED is a useful robustness probe for detecting models that rely on surface patterns rather than genuine premise-hypothesis reasoning, a use the paper does not fully develop.
  • The gains from in-domain implicit relations indicate that domain adaptation to non-CS scientific text, not merely larger CS training sets, may be the most direct route to higher scores on MISMATCHED.
  • Because the implicit pairs are extracted from adjacent sentences, the approach connects scientific NLI to discourse coherence and could provide a testbed for whether models can recover missing discourse connectives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MISMATCHED, a human-annotated out-of-domain scientific NLI benchmark containing 2,700 sentence pairs from psychology, engineering, and public health, with a paper-level split into dev and test sets. The authors establish SLM and LLM baselines trained on existing scientific NLI datasets (SCINLI, MSCINLI, and their combination), report a best SLM Macro F1 of 78.17% and a best open-source LLM Macro F1 of 57.16%, and include analyses of hypothesis-only artifacts, class-wise performance, and few-shot scaling. The paper also claims that incorporating adjacent sentence pairs with implicit scientific NLI relations, where the relations are pseudo-labeled by a fine-tuned SCIBERT model at a confidence threshold of 0.6, improves performance on SCINLI and MISMATCHED.

Significance. The benchmark itself is a valuable and well-executed contribution: it fills a real gap by extending scientific NLI beyond computer science domains, uses human annotation for dev/test, prevents paper-level leakage, controls for hypothesis-only artifacts, and releases code and data. If the implicit-relation claim were properly validated, it would add a useful secondary contribution, but as presented that claim rests on unvalidated pseudo-labels and lacks the controls needed to distinguish learning of implicit discourse relations from domain adaptation or self-training effects. The benchmark contribution is stronger and more secure than the implicit-relation contribution.

major comments (4)
  1. [Section 5, 'Implicit Relation Annotation'] The pseudo-labels for implicit relations are produced by a SCIBERT model fine-tuned on MSciNLI+, with no human validation, no agreement statistics, and no error analysis for the pseudo-labeled set. The model subsequently evaluated in Table 9 is also a SCIBERT model, and the implicit training data are selected by that same model's confidence scores. The reported improvement on MISMATCHED (78.17 to 79.66 Macro F1) could therefore reflect the final model learning the pseudo-labeler's own biases rather than learning genuine implicit scientific NLI relations. To support the claim, the authors should validate a sample of the pseudo-labels with human annotators, report per-class agreement, and compare against a self-training control in which the same amount of target-domain data is labeled by a different mechanism or by an alternative model.
  2. [Section 5, 'Incorporating Implicit Relations'] For the MISMATCHED evaluation, the implicit training pairs are extracted from papers in the same three non-CS domains as the test set, whereas the explicit-only baseline is trained solely on CS-domain data. The observed gains may simply be an in-domain data effect rather than evidence that implicit relations are valuable. A control that adds explicit-relation pairs from the same target-domain papers, or otherwise holds the domain constant while varying only the implicit/explicit nature of the relation, is needed to support the paper's causal interpretation.
  3. [Table 9 and Section 5, 'Results'] The magnitudes of the reported improvements are modest and in several comparisons fall within one standard deviation of the baseline. For example, PHI-3 few-shot with MSciNLI implicit exemplars improves only 0.34 points on MISMATCHED (57.16 to 57.50), and the SCIBERT rows in Table 9 are reported without standard deviations despite Section 4.2 stating that all non-zero-shot experiments use three runs. No significance tests are reported. The conclusion that implicit relations 'can indeed improve' performance is stronger than the evidence supports.
  4. [Section 3.2, Phase 2: Human Annotation] Only sentence pairs for which the human gold label agrees with the automatically assigned distant-supervision label are retained in MISMATCHED; all disagreements are discarded. Appendix B.3 shows that this filtering is highly class-dependent, with only 68.3% agreement for NEUTRAL compared with over 93% for the other classes. This means the benchmark is not a random sample of scientific NLI relations and may systematically exclude ambiguous or hard cases, particularly in the NEUTRAL class. The authors should either release the discarded annotations, analyze the disagreement patterns, or justify the filtering as a deliberate quality choice with discussion of its effect on benchmark difficulty.
minor comments (6)
  1. [Table 9] The notation 'MS+' is not defined, and the SCIBERT rows do not report standard deviations; please make the table consistent with the experimental protocol described in Section 4.2.
  2. [Throughout] The combined dataset is referred to inconsistently as MSCINLI+, MS CINLI+, MSciNLI+, and MS+; please unify the notation.
  3. [Table 1] The strikethrough used to indicate deleted linking phrases is not visible in the plain-text rendering; an alternative visual cue or a note in the caption would improve readability.
  4. [Abstract and Section 3] The abstract says the benchmark contains 2,700 human-annotated sentence pairs, but the construction process annotated 3,253 pairs and discarded those without agreement; please clarify that 2,700 is the size after filtering and downsampling.
  5. [Section 4.2] The statement that the average performance gap is 'approximately 15%' should specify which top-performing LLM and which metric are being compared, since the gap differs across domains and settings.
  6. [Appendix G] The description of the pseudo-labeling setup says 'we utilize the SCIBERT model fine-tuned as our baseline' without specifying which training set; please state explicitly that the pseudo-labeler is fine-tuned on MSciNLI+, and clarify whether a separate SCIBERT is fine-tuned for SCINLI-derived pairs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MISMATCHED benchmark is human-annotated and independently evaluated, and the implicit-relation experiment, while relying on unvalidated pseudo-labels, is not a reduction by construction.

full rationale

The central benchmark contribution is self-contained. MISMATCHED dev/test labels are produced by human annotators (Fleiss kappa 0.72) after an iterative curation process, with paper-level splitting to prevent leakage and a hypothesis-only control to check for spurious artifacts. The distant-supervision method from Sadat and Caragea (2022, 2024) is used only to generate candidate pairs; the final benchmark keeps only pairs where the human gold label agrees with the automatic label, so the test labels do not reduce to the automatic rule. The baselines in Section 4 are standard fine-tuning or prompting evaluations against these held-out human labels. The only candidate for circularity is the implicit-relations experiment in Section 5: pseudo-labels are assigned by a SciBERT model fine-tuned on MSCINLI+, then a fresh SciBERT is trained on the selected pseudo-labeled pairs and further fine-tuned on explicit data. This is a self-training/semi-supervised procedure, and the absence of human validation of the pseudo-labels is a genuine validity limitation for the claim that implicit relations (rather than confident model predictions) drive the improvement. However, the reported gains in Table 9 are measured on independently human-annotated test sets (MISMATCHED and SCINLI), not on the pseudo-labeled training data itself, so the result is not forced by construction. No fitted parameter is renamed as a prediction, no load-bearing claim is justified solely by a self-citation, and no uniqueness or ansatz is imported from the authors' prior work to forbid alternatives. The reliance on SciNLI and MSCINLI is normal use of publicly available, published resources with human-annotated dev/test splits. Therefore, no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central benchmark does not introduce free parameters beyond dataset construction choices. The implicit-relation claim relies on a hand-picked confidence threshold and on unvalidated pseudo-labels, which are the main circularity risk.

free parameters (1)
  • confidence threshold for pseudo-label filtering = 0.6
    Used to filter implicit sentence pairs before training; chosen by hand, not derived from data.
assumptions (3)
  • domain assumption Linking phrases (e.g., 'However', 'Therefore', 'Specifically') reliably indicate the four scientific NLI relations.
    Used in Section 3.2 for distant supervision to automatically label sentence pairs; human agreement is high but not perfect, and NEUTRAL agreement is only 68.3%.
  • ad hoc to paper Pseudo-labels from a fine-tuned SCIBERT model with confidence at least 0.6 are accurate enough to train on as implicit relations.
    Section 5; no human validation of pseudo-labels, so this is an unverified assumption.
  • domain assumption Adjacent sentences without linking phrases are likely to have an implicit entailment, reasoning, or contrasting relation, while non-adjacent pairs are likely neutral.
    Used to construct the candidate set for implicit relations in Section 5; this heuristic is plausible but not directly validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A MISMATCHED Benchmark for Scientific Natural Language Inference." pith.science (2026). https://pith.science/paper/K6NNO4SM

@misc{pith2026250604603,
  author       = {Pith},
  title        = {Pith review of: A MISMATCHED Benchmark for Scientific Natural Language Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6NNO4SM}},
  note         = {Machine review of arXiv:2506.04603}
}
read the original abstract

Scientific Natural Language Inference (NLI) is the task of predicting the semantic relation between a pair of sentences extracted from research articles. Existing datasets for this task are derived from various computer science (CS) domains, whereas non-CS domains are completely ignored. In this paper, we introduce a novel evaluation benchmark for scientific NLI, called MISMATCHED. The new MISMATCHED benchmark covers three non-CS domains-PSYCHOLOGY, ENGINEERING, and PUBLIC HEALTH, and contains 2,700 human annotated sentence pairs. We establish strong baselines on MISMATCHED using both Pre-trained Small Language Models (SLMs) and Large Language Models (LLMs). Our best performing baseline shows a Macro F1 of only 78.17% illustrating the substantial headroom for future improvements. In addition to introducing the MISMATCHED benchmark, we show that incorporating sentence pairs having an implicit scientific NLI relation between them in model training improves their performance on scientific NLI. We make our dataset and code publicly available on GitHub.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 6 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. https://doi.org/10.18653/v1/D19-1371 S ci BERT : A pretrained language model for scientific text . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615--3620, Hong Kong, China...

    doi:10.18653/v1/d19-1371 2019

  5. [5]

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1

    2009

  6. [6]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/v1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642, Lisbon, Portugal. Association for Computational Linguistics

    doi:10.18653/v1/d15-1075 2015

  7. [7]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177--190, Berlin, Heidelberg. Springer Berlin Heidelberg

    work page 2006

  8. [8]

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021. http://arxiv.org/abs/2110.02861 8-bit Optimizers via Block-wise Quantization . arXiv preprint arXiv:2110.02861. Published as a conference paper at ICLR 2022

    arXiv 2021

Show all 30 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  2. [10]

    Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  3. [11]

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1--9

    2007

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pages 785--794

    2006

  6. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  7. [15]

    Ma \"e l Jullien, Marco Valentino, and Andr \'e Freitas. 2024. Semeval-2024 task 2: Safe biomedical natural language inference for clinical trials. arXiv preprint arXiv:2404.04963

    2024 arXiv

  8. [16]

    Ma \"e l Jullien, Marco Valentino, Hannah Frost, Paul O’Regan, D \'o nal Landers, and Andr \'e Freitas. 2023. Nli4ct: Multi-evidence natural language inference for clinical trial reports. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

    2023

  9. [17]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  10. [18]

    Dan Klein and Christopher D. Manning. 2003. https://doi.org/10.3115/1075096.1075150 Accurate unlexicalized parsing . In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423--430, Sapporo, Japan. Association for Computational Linguistics

    2003

  11. [19]

    Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S Gerber, and Laura E Barnes. 2017. Hdltex: Hierarchical deep learning for text classification. In 2017 16th IEEE international conference on machine learning and applications (ICMLA), pages 36...

    2017

  12. [20]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  13. [21]

    Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. http://www.lrec-conf.org/proceedings/lrec2014/pdf/363_Paper.pdf A SICK cure for the evaluation of compositional distributional semantic models . In Proceedings of t...

    2014

  14. [22]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    2020 doi

  15. [23]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al

    OpenAI , Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o System Card . arXiv preprint arXiv:2410.21276

    2024 arXiv

  16. [24]

    Alexey Romanov and Chaitanya Shivade. 2018. https://doi.org/10.18653/v1/D18-1187 Lessons from natural language inference in the clinical domain . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1586--1596, Brussels, Belg...

    2018 doi

  17. [25]

    Mobashir Sadat and Cornelia Caragea. 2022. https://doi.org/10.18653/v1/2022.acl-long.511 S ci NLI : A corpus for natural language inference on scientific text . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2022 doi

  18. [26]

    Mobashir Sadat and Cornelia Caragea. 2024. https://arxiv.org/abs/2404.08066 M sci NLI : A diverse benchmark for scientific natural language inference . In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Mexico City, Mexico...

    2024 arXiv

  19. [27]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  20. [28]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks ...

    2018

  21. [29]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  22. [30]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.