Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that pretraining BERT on a 16.3-million-question multiple-choice dataset built by aligning ConceptNet triples with Wikipedia sentences and masking one concept improves commonsense benchmarks substantially while preserving…

desk verdict A clean empirical paper with a genuinely useful AMS data-construction pipeline, but the headline CSQA gain is confounded by the shared ConceptNet source and the missing control the authors themselves promise in footnote 4. read the letter →

arxiv 1908.06725 v5 pith:HV6QCD6J submitted 2019-08-19 cs.CL

Zhi-Xiu Ye , Qian Chen , Wen Wang , Zhen-Hua Ling This is my paper · ORCID
classification cs.CL
keywords commonsensereasoningpretrainedlanguagemodelsBERTmulti-choicequestionansweringConceptNetdistantsupervisionQAWinogradSchemaChallenge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to push commonsense knowledge into pretrained language models without losing their general language abilities. The method, called align-mask-select (AMS), builds a 16.3-million-question multiple-choice dataset by matching ConceptNet triples to Wikipedia sentences, masking one concept, and choosing distractor concepts that share the same relation. Pretraining BERT on this dataset before fine-tuning raises accuracy to 62.2% on CommonsenseQA, 5.5 points above BERT-large and 4.0 points above the prior best, and to 75.5% on the Winograd Schema Challenge, while staying within about one point of BERT on GLUE. The core claim is that commonsense can be injected cheaply and automatically, with no extra labels or architecture changes.

What carries the argument

The machinery is the AMS (align-mask-select) data construction pipeline. Align takes a ConceptNet triple (concept1, relation, concept2) and finds a Wikipedia sentence containing both concepts as exact string matches; mask replaces one concept with the token [QW] to form the question; select pulls four distractors from ConceptNet by querying (concept1, relation, *) or (*, relation, concept2), so every candidate shares the same relation and one concept with the correct answer. This yields 16,324,846 questions. The pretraining objective then scores each candidate by a softmax over the [CLS] representation of the concatenated question-candidate sequence, teaching the model whole-concept selection rather than word-level prediction.

What would settle it

Take a random sample of about 1,000 sentences from the DAMS dataset, show each sentence with the masked concept replaced by a blank and the correct concept plus distractors, and ask human annotators whether the correct concept is entailed by the sentence alone; if a large fraction of items are judged unanswerable or the correct answer is not entailed, the distant-supervision premise fails. A cheaper computational check is to train BERT-CS on the same questions with the correct answer randomly permuted among candidates and measure how much of the CSQA gain survives.

Watch

Extended reading notes

Core claim

The central claim is that a pure multi-choice question answering pretraining task over distantly supervised natural-language questions transfers commonsense reasoning to BERT. To create the task, the AMS method aligns each ConceptNet relation triple to sentences containing both concepts as exact strings, masks one concept, and selects four distractors that share the same unmasked concept and relation, so the wrong answers are semantically close rather than random. By replacing the masked-language-model and next-sentence objectives with this QA objective, the model learns to choose the concept that the sentence's commonsense relation actually licenses. On CommonsenseQA, BERT-CS-large reaches 62.2% test accuracy, and on WSC it reaches 75.5%, outscoring the previous best BERT-based system by 3.3 points; on eight GLUE tasks it stays comparable to the original BERT.

Load-bearing premise

The load-bearing assumption is that a Wikipedia sentence containing both concepts of a ConceptNet triple actually expresses that triple's relation whenever the two concepts appear as exact strings; if that distant-supervision link is frequently wrong, many of the 16.3 million training questions are unanswerable or mislabeled.

Editorial extensions

If this is right

  • Pretraining on distantly supervised natural-language QA can substitute for human-labeled explanation data: BERT-CS-large beats CoS-E, which uses collected explanations, by 4.0 points on CommonsenseQA.
  • The approach transfers across commonsense benchmarks: the same pretrained model improves both CommonsenseQA and Winograd Schema Challenge, indicating the learned signal is not tied to one dataset's format.
  • The semantic closeness of distractors matters: compared with randomly chosen distractors, same-relation distractors improve CSQA development accuracy, so harder choice sets force deeper semantic discrimination.
  • MCQA alone outperforms MCQA combined with masked language modeling; adding MLM's 15% token masking hurts, likely because it moves inputs away from natural sentences.
  • Commonsense injection does not degrade general language representation: on eight GLUE tasks, BERT-CS-large matches BERT-large and BERT-CS-base slightly exceeds BERT-base.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the same AMS pipeline were run on a larger or more recent knowledge graph and corpus, the WSC 'non-associative' subset would be the most informative number to watch, since that subset isolates genuine commonsense from statistical antecedent cues.
  • Editorial inference: the exact-string align step suggests a concrete stress test: replace aligned sentences with paraphrases or use entity linking, and see whether gains persist; if they vanish, the model may be relying on lexical overlap rather than relation semantics.
  • Editorial inference: the method should apply to other relation-rich pretraining signals, such as temporal, spatial, or causal text, since the objective only needs triples plus a corpus rather than task-specific supervision.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a three-step AMS (align, mask, select) procedure to automatically construct a multi-choice question answering corpus, DAMS, from ConceptNet triples aligned with English Wikipedia sentences. A masked concept is treated as the correct answer, and distractors are selected from ConceptNet triples sharing the same relation and the unmasked concept. The authors initialize BERT-base and BERT-large from released checkpoints, continue pre-training on DAMS with a multi-choice QA objective to obtain BERT CSbase and BERT CSlarge, and then fine-tune on CSQA, WSC/WNLI, and GLUE. They report that BERT CSlarge reaches 62.2% on CSQA (5.5 points above BERT-large and 4.0 above CoS-E), 75.5% on WSC (3.3 above previous state of the art), and roughly unchanged performance on eight GLUE tasks. They also perform ablations over data construction choices and pre-training tasks on BERT-base.

Significance. If the results hold, the paper contributes a simple, knowledge-source-agnostic pre-training recipe for injecting structured commonsense knowledge into language representations without task-specific architectures, along with a large automatically generated MCQA corpus. The WSC result is a genuinely external check that the benefit transfers beyond CSQA, and the ablation study comparing triples versus natural-language sentences, random versus relation-filtered distractors, and MCQA versus MLM is informative. The method is easy to reproduce in principle and likely to be useful to the community. However, the headline CSQA improvement is currently confounded by the overlap between the pre-training source and the evaluation benchmark, and the central causal claim that commonsense knowledge is incorporated into the representations is not yet established.

major comments (3)
  1. [Section 4.1, Table 4, footnote 4] The main CSQA claim is not isolated from a simple data-augmentation effect. In Table 4, BERT CSlarge is compared with BERT-large after BERT CSlarge has been pre-trained on 16.3 million ConceptNet-derived MCQA examples, and CSQA itself is a ConceptNet-derived benchmark. Footnote 4 explicitly acknowledges that DAMS 'may be considered as an augmented data for CSQA' and promises a control experiment: fine-tuning the original BERT on CSQA-train plus DAMS. That control is not reported anywhere in the paper. Until it is run, the 5.5-point CSQA gain is equally consistent with the model having seen a large volume of in-domain, ConceptNet-style cloze data in the same multi-choice format, rather than with commonsense knowledge being transferred into the representation. Please run the promised control with matched fine-tuning budgets and report whether the CSQA conclusion changes.
  2. [Section 3.2, Table 2] The distant-supervision validity of the align step is not quantified. The method assumes that if a Wikipedia sentence contains exact string matches of concept1 and concept2 from a ConceptNet triple, then masking one concept produces a question whose correct answer is the other concept. The paper cites Riedel et al. (2010) to acknowledge noise in distant supervision, but it provides no estimate of how often the aligned sentences actually express the intended relation in DAMS. If a large fraction of the 16.3 million labels are false positives from homonyms, coincidental co-occurrence, or independent mentions, then the pre-training could teach shallow dataset-specific cues rather than general commonsense. Please provide a human-annotated or automatically estimated precision sample of the alignments, and ideally analyze whether filtering by alignment confidence changes the CSQA or WSC results.
  3. [Section 5.2, Figure 1] The reported performance-curve procedure is not fully specified. The text says that 'we remove the results that are significantly lower than the mean' without defining the criterion, the window over which the mean is computed, or how many seeds were removed. This makes the standard-deviation bars in Figure 1 non-reproducible and potentially optimistic, and it weakens the comparison between BERT CSbase and BERT CSlarge convergence behavior. Please specify the exact outlier-removal rule and report all seed results, or use a standard robust summary such as the median with interquartile range.
minor comments (5)
  1. [Section 4.2, Table 5] Please clarify whether the 'BERTlarge + MCQA' row is a new baseline computed by the authors with the same hyperparameter search as BERT CSlarge, or an external result; the text describes the setup for the authors' models but does not explicitly state the provenance of this row.
  2. [Section 4.2] The paper uses the WSCR test partition as a development set for selecting alpha and beta. Please state explicitly whether the 273-instance WSC test set is disjoint from WSCR; if there is any overlap, the reported WSC numbers may be optimistic for hyperparameter selection.
  3. [Section 4.3, Table 6] The claim that BERT CSbase is 'slightly better' than BERT base on GLUE is based on point estimates without variance or significance testing; please report per-seed standard deviations or another measure of variability.
  4. [Section 5.1, Table 7] The ablation results are also single point estimates; given that fine-tuning uses five seeds with the best development model selected, please report the variance or at least the seed-level values for the key comparisons in Table 7.
  5. [Section 1 and throughout] There are minor typographical and formatting issues: 'commmonsense' in the introduction, and the dataset symbol appearing as 'DAM S' in several places due to subscript spacing; these should be cleaned up.

Circularity Check

1 steps flagged · score 4.0 of 10

CSQA result confounded by pretraining-data overlap; the control promised in footnote 4 is absent, so the headline gain is not independent evidence for commonsense pre-training.

  1. fitted input called prediction [Section 4.2 (Winograd Schema Challenge), including footnote 4]
    "Since the CSQA dataset is created using concepts from ConceptNet and DAMS is created using ConceptNet, DAMS may be considered as an augmented data for CSQA. However, these WSC results demonstrate that the proposed approach is general for improving commonsense reasoning capabilities of the pre-trained models."

    The claimed improvement on CSQA is presented as evidence that the pre-training step incorporated commonsense knowledge, but DAMS and CSQA are both generated from ConceptNet and both use the same multi-choice QA format with five candidates. The paper itself acknowledges that DAMS 'may be considered as an augmented data for CSQA' and promises, in footnote 4, to compare BERT CS against fine-tuning the original BERT on CSQA-train plus DAMS. That control is never reported. Without it, the 5.5-point CSQA gain can be explained by exposing the model to ConceptNet-derived in-domain cloze data rather than by any general commonsense transfer, so this particular evaluation is partially forced by construction.

full rationale

The central derivation is a pre-training corpus (DAMS) built from ConceptNet and Wikipedia, followed by fine-tuning on CSQA, WSC, and GLUE. The only substantial circularity concern is the CSQA evaluation: DAMS and CSQA share the same knowledge graph source and the same multi-choice task format, and the paper explicitly concedes that DAMS may be viewed as augmented data for CSQA while deferring the decisive control experiment to future work. Because that control is absent, the headline CSQA number cannot cleanly establish that commonsense knowledge was incorporated. This is not a full derivation-level circularity: WSC is an external benchmark, GLUE is external, and the WSC gain of 3.3 points provides independent evidence that something general was learned. No equations are shown to be equivalent by construction, and no load-bearing self-citation chain appears in the paper. The score of 4 reflects one partially forced 'prediction' (CSQA) with independent external results keeping the overall circularity moderate rather than high.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The empirical claims rest mainly on two external assumptions: ConceptNet is a trustworthy commonsense source, and exact string matching in Wikipedia is enough to turn a ConceptNet triple into a reliably labeled question. The paper acknowledges distant-supervision noise in general but never measures it for DAMS. Hyperparameters tuned on development sets are listed as free parameters; no invented entities are introduced.

free parameters (3)
  • WSC margin loss hyperparameters alpha and beta = Grid searched: alpha in {2.5, 5, 10, 20}, beta in {0.05, 0.1, 0.2, 0.4}; final values not reported
    Selected by optimizing accuracy on the WSCR test partition used as the development set for WSC, Section 4.2.
  • GLUE fine-tuning learning rate = 1e-5, 2e-5, or 3e-5, chosen per task by development set performance
    Follows Devlin et al. (2019) and is tuned on the development set, Section 4.3.
  • Number of pretraining epochs on DAMS = 1 epoch; 2 epochs gave worse development accuracy
    Chosen from observed performance in Section 5.2.
assumptions (5)
  • domain assumption ConceptNet triples are reliable commonsense facts suitable as training supervision for language models.
    Used as the sole knowledge source for DAMS in Section 3.1. If ConceptNet is biased or noisy, the pretraining data inherits that.
  • domain assumption Any sentence that mentions both concepts of a ConceptNet triple expresses the triple's relation (distant supervision).
    Section 3.2 align step matches the two concepts exactly and treats the masked concept as the correct answer; the paper cites Mintz et al. (2009) but does not validate the assumption on DAMS.
  • domain assumption Exact string matching in Wikipedia aligns concepts with their intended senses.
    No entity linking is used, so homonyms can produce false alignments. The paper says 'in future work, we will explore entity linking' in Section 3.2.
  • domain assumption Fine-tuning a model pretrained on multi-choice QA transfers to WSC and GLUE tasks.
    Underlying transfer learning assumption used throughout Section 4; standard in NLP pretraining.
  • domain assumption WSCR training data is an appropriate supervision source for WSC pronoun disambiguation.
    Used in Section 4.2; prior work by Rahman and Ng (2012) and Kocijan et al. (2019) established this setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models." pith.science (2026). https://pith.science/paper/HV6QCD6J

@misc{pith2026190806725,
  author       = {Pith},
  title        = {Pith review of: Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HV6QCD6J}},
  note         = {Machine review of arXiv:1908.06725}
}
read the original abstract

The state-of-the-art pre-trained language representation models, such as Bidirectional Encoder Representations from Transformers (BERT), rarely incorporate commonsense knowledge or other knowledge explicitly. We propose a pre-training approach for incorporating commonsense knowledge into language representation models. We construct a commonsense-related multi-choice question answering dataset for pre-training a neural language representation model. The dataset is created automatically by our proposed "align, mask, and select" (AMS) method. We also investigate different pre-training tasks. Experimental results demonstrate that pre-training models using the proposed approach followed by fine-tuning achieve significant improvements over previous state-of-the-art models on two commonsense-related benchmarks, including CommonsenseQA and Winograd Schema Challenge. We also observe that fine-tuned models after the proposed pre-training approach maintain comparable performance on other NLP tasks, such as sentence classification and natural language inference tasks, compared to the original BERT models. These results verify that the proposed approach, while significantly improving commonsense-related NLP tasks, does not degrade the general language representation capabilities.

Figures

Figures reproduced from arXiv: 1908.06725 by the authors.

Figure 1
Figure 1. BERT CSbase and BERT CSlarge accuracy on the CSQA development set against the number of pre-training steps. 5.2 Performance Curve We investigate the performance from BERT CS on the CSQA development set against the pre-training steps. For every 10,000 training steps, we save the model as the initial model for fine-tuning. For each model, we run experiments for 10 times with the same pre-trained checkpoint but use dif… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 5 canonical work pages

  1. [1]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  2. [2]

    Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2018. A knowledge hunting framework for common sense reasoning. arXiv preprint arXiv:1810.01375

    arXiv 2018

  3. [3]

    Jeremy Howard and Sebastian Ruder. 2018. https://doi.org/10.18653/v1/P18-1031 Universal language model fine-tuning for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328--339, Melbourne, Australia. Association for Computational Linguistics

    doi:10.18653/v1/p18-1031 2018

  4. [4]

    Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv:1905.06290

    arXiv 2019

  5. [5]

    Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning

    2012

  6. [6]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa : A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    arXiv 2019

  7. [7]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111--3119

    2013

  8. [8]

    Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. https://www.aclweb.org/anthology/P09-1113 Distant supervision for relation extraction without labeled data . In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP , pages 1003--1011, ...

    2009

Show all 31 references
  1. [9]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G love: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doha,...

    2014 doi

  2. [10]

    Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. https://doi.org/10.18653/v1/P17-1161 Semi-supervised sequence tagging with bidirectional language models . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vo...

    2017 doi

  3. [11]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the Assoc...

    2018 doi

  4. [12]

    Peters, Mark Neumann, Robert L Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A

    Matthew E. Peters, Mark Neumann, Robert L Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In EMNLP, pages 43--54

    2019

  5. [13]

    Miller, and Sebastian Riedel

    Fabio Petroni, Tim Rockt?schel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In EMNLP, pages 2463--2473

    2019

  6. [14]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf

    2018

  7. [15]

    Altaf Rahman and Vincent Ng. 2012. https://www.aclweb.org/anthology/D12-1071 Resolving complex cases of definite pronouns: The W inograd schema challenge . In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural...

    2012

  8. [16]

    Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361

    2019 arXiv

  9. [17]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  10. [18]

    Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 148--163. Springer

    2010

  11. [19]

    Yu-Ping Ruan, Xiaodan Zhu, Zhen-Hua Ling, Zhan Shi, Quan Liu, and Si Wei. 2019. Exploring unsupervised pretraining and sentence structure modelling for winograd schema challenge. arXiv preprint arXiv:1904.09705

    2019 arXiv

  12. [20]

    Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence

    2017

  13. [21]

    Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019 a . Probing prior knowledge needed in challenging chinese machine reading comprehension. arXiv preprint arXiv:1904.09679

    2019 arXiv

  14. [22]

    Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019 b . ERNIE : E nhanced R epresentation through K nowledge I ntegration. arXiv preprint arXiv:1904.09223

    2019 arXiv

  15. [23]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  16. [24]

    Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2019. https://doi.org/10.18653/v1/D19-1335 How reasonable are common-sense reasoning tasks: A case-study on the W inograd schema challenge and SWAG . In Proceedings of the 2019 Conference on...

    2019 doi

  17. [25]

    Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847

    2018 arXiv

  18. [26]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  19. [27]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144

    2016 arXiv

  20. [28]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet : G eneralized A utoregressive P retraining for L anguage U nderstanding. arXiv preprint arXiv:1906.08237

    2019 arXiv

  21. [29]

    Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE : E nhanced L anguage R epresentation with I nformative E ntities. arXiv preprint arXiv:1905.07129

    2019 arXiv

  22. [30]

    Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2018. Improving question answering by commonsense-based pre-training. arXiv preprint arXiv:1809.03568

    2018 arXiv

  23. [31]

    Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2019. Evaluating commonsense in pre-trained language models. arXiv preprint arXiv:1911.11931

    2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.