Pith. sign in

REVIEW 4 major objections 8 minor 6 cited by

Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

T0 review · 4 major / 8 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces Cosmos QA, a 35,588-question benchmark asserting that machines (68.4%) still lag humans (94.0%) when reading comprehension requires commonsense inference beyond the text.

desk verdict Cosmos QA is a genuinely useful benchmark that deserves peer review; the human-ceiling number is softer than it looks, but the dataset carries the paper. read the letter →

arxiv 1909.00277 v2 pith:GC72YH52 submitted 2019-08-31 cs.CL cs.AI

Lifu Huang , Ronan Le Bras , Chandra Bhagavatula , Yejin Choi This is my paper · ORCID
classification cs.CLcs.AI
keywords commonsensereasoningreadingcomprehensionmultiple-choicequestionansweringcontextualinferencebenchmarkdatasetBERTcounterfactualcrowdsourcing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that contextual commonsense reasoning is a distinct, measurable component of reading comprehension, and that it can be benchmarked at scale. To do so it introduces Cosmos QA, a dataset of 35,588 multiple-choice problems built from personal blog narratives, where the correct answer is not a text span but an inference about causes, effects, motives, or counterfactuals. The paper reports that the strongest model it tests, a BERT variant with multiway attention, reaches 68.4 percent accuracy while human annotators reach 94.0 percent on a 200-question sample, leaving a 25.6-point gap. If the benchmark is valid, it gives the field a shared yardstick for progress on reading between the lines.

What carries the argument

The load-bearing object is the dataset itself: 35,588 crowdsourced question sets built from a large blog corpus, with a validation filter that keeps a question only if at least two of three workers pick the intended answer and all judge it satisfactory. On top of it, the paper's best model is BERT fine-tuned for the answer-selection task with multiway bidirectional attention, which computes separate attentive representations of the paragraph conditioned on the question, on the answer, and on both, then fuses them for classification. The constructor also creates 'None of the above' questions from items that workers could answer without the context, and uses three BERT next-sentence-prediction models to replace the least challenging negative answer with 'None of the above' in 70% of training questions.

What would settle it

Take a random sample of Cosmos QA test questions and re-annotate them with unanimous agreement among five or more annotators, then recompute both the human ceiling and the best model's accuracy on that stricter subset. If the stricter subset shrinks the machine-human gap substantially, the reported 25.6-point gap is inflated by annotation noise; if the gap persists on unanimous questions, the benchmark's difficulty claim is supported.

Watch

Extended reading notes

Core claim

Cosmos QA is a large-scale multiple-choice machine reading comprehension benchmark in which roughly 93.8% of questions require commonsense inference rather than literal text matching, and more than 83% of correct answers are not stated in the context paragraph. The contexts are 21,886 everyday narratives from personal weblogs, and questions target causes, effects, facts about entities, and hypotheticals and counterfactuals. The paper's central empirical claim is that contemporary models, including a fine-tuned BERT with multiway attention, solve this task only at 68.4% accuracy, well below the 94.0% human estimate, so there is a substantial machine-human gap in contextual commonsense reasoning.

Load-bearing premise

The whole benchmark rests on the assumption that the crowdsourced agreement filter, which keeps a question when at least two of three workers pick the intended answer and find the item satisfactory, actually isolates questions that require contextual commonsense reasoning rather than surface cues or general world knowledge.

Editorial extensions

If this is right

  • If Cosmos QA measures what it claims, then any future model that closes the 25.6-point gap will have demonstrated a real gain in contextual commonsense reasoning.
  • The finding that multiway attention improves over plain BERT fine-tuning suggests that explicitly modeling paragraph-question-answer interactions helps on inference-heavy reading comprehension, not just literal question answering.
  • The reported 83% non-literal answer rate means span-extraction and lexical-overlap models should be expected to fail on this task, making it a stress test for entailment-style reasoning.
  • Because the dataset supports generative evaluation, it can be used to assess open-ended commonsense generation, not only multiple-choice selection.
  • Knowledge-transfer results indicate that fine-tuning on a document-level reading-comprehension dataset and on a grounded commonsense-inference dataset both transfer to Cosmos QA, so it can serve as a downstream evaluation for pretraining objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit: if the two-of-three validation filter admits questions answerable by surface cues, then the 25.6-point gap may overstate the difficulty of true commonsense inference; a stricter unanimous-agreement subset would be a sharper test.
  • The 'None of the above' mechanism is likely to interact with model calibration: a model with no explicit rejection head may under-predict the unanswerable option, and this could be tested by measuring accuracy on the 6-8% unanswerable split separately.
  • A natural extension would be to build parallel versions of Cosmos QA in other languages or with controlled context perturbations, such as swapped entities, to separate world-knowledge inference from narrative-coherence inference.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces Cosmos QA, a multiple-choice reading comprehension dataset of 35,588 problems built from 21,886 personal-narrative blog contexts, and argues that the dataset requires contextual commonsense reasoning beyond literal text matching. The construction pipeline uses AMT workers to author questions and answers, three workers to validate each question set, a BERT-based heuristic to create 'None of the above' unanswerable instances, and a test set drawn from questions authored by a selected group of high-quality workers. The empirical section evaluates several neural baselines and a new BERT-with-multiway-attention model, reporting 68.4% test accuracy versus a claimed 94.0% human performance, together with ablations, knowledge-transfer experiments, error analysis, and a generative evaluation with GPT-2.

Significance. Cosmos QA is a substantial and well-documented dataset contribution, and the release of the data, code, and leaderboard is a concrete strength. The three-stage validation, the public leaderboard, the detailed error analysis, and the transparent ablations (including the question-removal ablation in Table 4) are valuable for the community. If the 94.0% human ceiling is reliable, the 25.6-point machine-human gap provides a meaningful target for commonsense reading comprehension research. The main caveat is that the headline human number rests on a small sample and on majority voting over three annotators, so the precise size of the gap is less certain than the paper suggests.

major comments (4)
  1. [Section 4.1, Table 3] The human performance estimate of 94.0% is based on 200 randomly sampled test-set questions with majority vote over three AMT workers, and no confidence interval or inter-annotator agreement is reported. For a binomial proportion near 0.94 with n=200, the standard error is about 1.7 points, so the true human accuracy could be several points lower; if it were, say, 90%, the headline 25.6-point gap would shrink to about 21.6 points. I ask the authors to report a substantially larger human evaluation (or at least a confidence interval and agreement statistics on the existing 200 questions), and to state explicitly whether these 200 questions are a random sample of the Section 2.5 test set.
  2. [Section 2.3] The validation rule accepts a question set when at least two of three workers select the intended answer and all workers rate the paragraph, question, and answers as satisfactory. This means a question with one dissenting worker is retained. The paper does not report the distribution of agreement levels (2/3 versus 3/3) for accepted questions, either in validation or in the human-performance sample. If a large fraction of accepted questions are only 2/3-agreed, the gold labels may be more ambiguous than the 94.0% human ceiling implies, and the benchmark's reliability as ground truth is weakened. Please report the agreement distribution and consider reporting human accuracy on the subset with 3/3 agreement.
  3. [Section 2.6, Table 2] The statement that 93.8% of Cosmos QA requires contextual commonsense reasoning is based on a manual categorization of 500 answerable questions, but the paper does not describe how many annotators performed this categorization or provide inter-annotator agreement. Since this statistic is central to the dataset's claimed novelty over MCScript and other reading comprehension datasets, the categorization protocol should be documented and the reliability of the 93.8% figure should be quantified.
  4. [Section 5.1, Table 4] The question-ablation result shows that BERT-FT with only paragraph and answer (A|P) reaches 64.5% test accuracy, only 2.6 points below the full (A|P,Q) model. This raises the concern that a sizable fraction of Cosmos QA questions are predictable from the paragraph and answer set alone, which bears directly on the benchmark's validity as a reading comprehension task. The paper discusses this as a consequence of limited question-type diversity, but it does not quantify how many instances are answerable without the question. I request an analysis that estimates this fraction, for example by examining the questions that the (A|P) model answers correctly, and a discussion of the implications for the 'contextual' claim.
minor comments (8)
  1. [Abstract, Table 1] The abstract says 35,600 problems, while Table 1 totals 35,588; please make the numbers consistent.
  2. [Table 7] Table 7 lists the Cosmos QA size as 31.8K, which differs from the 35,588 total in Table 1; please clarify which set this count refers to.
  3. [Section 2.4] There is a typo in 'at least on worker' which should be 'at least one worker'.
  4. [Table 3 caption] The caption contains the typo 'approachs', which should be 'approaches'.
  5. [Figure 2] The figure shows the placeholder text 'Chart Title' in the displayed labels; this should be removed before publication.
  6. [Section 5.4] The text contains the typo 'None of the aboce', which should be 'None of the above'.
  7. [Section 3.1] The classification loss equation has an unbalanced parenthesis: the denominator should be written as \(\sum_{j=1}^4 \exp(W_f^\top F_j)\), with the closing parenthesis matching the opening one after 'log'.
  8. [Section 4.2] The claim that more than 83% of correct answers are not stated in the paragraphs should be accompanied by a description of how this percentage was computed (manual sample or automatic span matching), since it is used to motivate the task design.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark's labels and human ceiling are independently sourced, and BERT's auxiliary roles in construction do not define the answers.

full rationale

Cosmos QA is primarily an externally sourced artifact: context comes from web blogs, questions and answers are authored by AMT workers, and gold labels are validated by independent worker agreement (Sections 2.2-2.3). Machine accuracies in Table 3 are measured on held-out test questions (Section 2.5), not predicted from the construction procedure. BERT appears in two auxiliary roles: paragraph segmentation (Appendix A) and selecting which negative answer to replace with 'None of the above' (Section 2.4). Neither role determines the correct answer: correct answers remain human-authored, and the BERT-based selection only modifies distractor sets. The human performance estimate of 94.0% (Section 4.1) is an empirical measurement on 200 sampled test questions, not an identity derived from the validation filter. Self-citations to ATOMIC, SWAG, and SocialIQa are contextual comparisons and are not load-bearing for the benchmark's validity. No equation in the paper reduces a claimed result to its own inputs, and no fitted parameter is renamed as a prediction. Therefore the paper exhibits no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central contribution is a benchmark, not a theory, so there are no derived free parameters in the physics sense. The listed quantities are design choices that the benchmark's difficulty and validity depend on. The axioms are assumptions about crowdsourcing reliability and the representativeness of the human estimate.

free parameters (5)
  • Validation agreement threshold = 2 of 3 workers
    A question set is accepted if at least two of three AMT workers pick the intended answer (Section 2.3). This modest threshold sets the quality bar for the whole dataset.
  • Human evaluation sample size = 200 question sets
    Human accuracy of 94.0% is computed on 200 randomly sampled test questions with 3-worker majority vote (Section 4.1). No confidence interval is reported.
  • False-negative creation rate = 70% of good question sets
    70% of the 33,219 good question sets get their least challenging negative answer replaced with 'None of the above' (Section 2.4), changing the distractor distribution for training.
  • Paragraph length bounds = 30 to 150 words
    Contexts are segmented so each paragraph contains 30-150 words (Appendix A), which shapes the difficulty and required inference distance.
  • Least-challenging negative selection with BERT = A' = argmin sum of BERT(A|delta) over subsets
    The distractor replaced by 'None of the above' is chosen by a fine-tuned BERT next-sentence predictor (Section 2.4). This couples benchmark construction to a model that is also evaluated.
assumptions (4)
  • domain assumption AMT workers' consensus labels are reliable ground truth for commonsense reasoning
    The central claim that Cosmos QA tests contextual commonsense reasoning assumes that the crowdsourced validation (Section 2.3) yields correct answers and filters out questions that are nonsensical or answerable without context.
  • domain assumption The four question categories (causes, effects, facts, counterfactuals) cover the space of contextual commonsense reasoning
    The paper claims these categories cover all 9 social commonsense types of ATOMIC (Section 2.2), which underpins the '93.8% requires commonsense' estimate.
  • domain assumption Majority vote of 3 AMT workers approximates human performance
    Human 94% is estimated from 200 questions with 3 workers each; this assumes the majority-vote label is correct and the sample is representative of the test set (Section 4.1).
  • ad hoc to paper BERT-large uncased provides a valid tool for dataset preprocessing
    BERT is used for paragraph segmentation (Appendix A) and for choosing which negative answer to replace (Section 2.4), even though the resulting benchmark is then used to evaluate BERT-based models. This does not break circularity because labels are human, but it is an ad hoc design choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning." pith.science (2026). https://pith.science/paper/GC72YH52

@misc{pith2026190900277,
  author       = {Pith},
  title        = {Pith review of: Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GC72YH52}},
  note         = {Machine review of arXiv:1909.00277}
}
read the original abstract

Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos QA, a large-scale dataset of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's everyday narratives, asking such questions as "what might be the possible reason of ...?", or "what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on Cosmos QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.github.io/cosmos.

Figures

Figures reproduced from arXiv: 1909.00277 by the authors.

Figure 1
Figure 1. , we can understand that the writer is not a child, yet needs someone to dress him or her every ∗*The work has been done during the author’s internship in AI2. P2: A woman had topped herself by jumping off the roof of the hospital she had just recently been admitted to. She was there because the first or perhaps latest suicide attempt was unsuccessful. She put her clothes on, folded the hospital gown and made the be… view at source ↗
Figure 2
Figure 2. Distribution of trigram prefixes of questions in C [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of each type of commonsense reasoning in C [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Prediction comparison between our approach [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Examples errors of our approach. (✓ indi￾cates correct answers and ✗ shows prediction errors.) tion between paragraph, question and each candi￾date answer. With sequentially fine-tuning, SWAG provides better performance, which indicates that with fine-tuning on SWAG, B…
Figure 7
Figure 7. Figure 7: Examples of human authored correct an￾swers, and automatically generated answers by pre￾trained GPT2 and GPT2-FT. (✓indicates the answer is correct while ✗shows that the answer is incorrect.) 6 Related Work There have been many exciting new datasets developed for readi…
Figure 8
Figure 8. Figure 8: Performance on COSMOS with various amount of training data E Details for Generative Evaluation For generative evaluation, we base on the OpenAI pre-trained GPT2 transformer language model,6 which has 117M parameters, and fine-tune it with all Paragraph, Question, Corre…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions

    cs.LG 2025-02 conditional novelty 8.0 of 10

    Weak-to-strong performance is governed by the overlap between the weak model's unlearnable error space and the strong model's principal-representation space, quantified by ||P_s(I-P_w)||.

  2. Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss

    cs.LG 2025-01 conditional novelty 6.0 of 10

    For convex and approximately convex model classes, the loss gain in weak-to-strong learning is at least the KL misfit between strong and weak models, plus an error term that vanishes as k grows.

  3. Debate Helps Weak-to-Strong Generalization

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Debate transcripts from two strong models, used as context when training an ensemble of weak models, improve weak-to-strong generalization on four NLP classification benchmarks.

  4. Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models

    cs.LG 2025-02 reject novelty 5.0 of 10

    CAP tunes conformal thresholds with RL to switch between single answers, sets, and abstention, but its test-set-fitting undermines the claimed statistical guarantees.

  5. Options-Aware Dense Retrieval for Multiple-Choice query Answering

    cs.IR 2025-01 conditional novelty 5.0 of 10

    Fine-tuning a sentence transformer so query-plus-options embeddings mimic oracle query-plus-answer embeddings improves evidence retrieval and multiple-choice accuracy on QuALITY.

  6. Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning

    cs.RO 2025-08 conditional novelty 4.0 of 10

    The thesis demonstrates that combining implicit 3D scene representations with LLM-based reasoning, using text as an interface, yields strong performance on robotic perception and spatial language tasks.

Reference graph

Works this paper leans on

49 extracted references · 29 canonical work pages · cited by 6 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  4. [4]

    Kevin Burton, Akshay Java, Ian Soboroff, et al. 2009. The icwsm 2009 spinn3r dataset. In Proceedings of ICWSM 2009

    work page 2009

  5. [5]

    Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In Proceedings of ACL 2017, pages 616--622

    work page 2017

  6. [6]

    Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of ACL 2016, pages 2358--2367

    work page 2016

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  8. [8]

    Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of ACL 2017, pages 1832--1846

    work page 2017

Show all 49 references
  1. [9]

    Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179

    2017 arXiv

  2. [10]

    Andrew Gordon and Reid Swanson. 2009. Identifying personal stories in millions of weblog entries. In Third International Conference on Weblogs and Social Media, Data Challenge Workshop

    2009

  3. [11]

    Andrew S Gordon. 2016. Commonsense interpretation of triangle behavior. In Proceedings of AAAI 2016

    2016

  4. [12]

    Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of NAACL 2018, pages 107--112

    2018

  5. [13]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of NIPS 2015, pages 1693--1701

    2015

  6. [14]

    Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children's books with explicit memory representations. arXiv preprint arXiv:1511.02301

    2015 arXiv

  7. [15]

    Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of ACL 2018, pages 328--339

    2018

  8. [16]

    Rudolf Kadlec, Martin Schmid, Ond r ej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Proceedings of ACL 2016, pages 908--918

    2016

  9. [17]

    Tom \'a s Ko c isk \`y , Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G \'a abor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Proceedings of TACL 2018, pages 317--328

    2018

  10. [18]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of EMNLP 2017, pages 785--794

    2017

  11. [19]

    Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning

    2012

  12. [20]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out

    2004

  13. [21]

    Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In Proceedings of ACL 2017, pages 510--517

    2017

  14. [22]

    Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. arXiv preprint arXiv:1805.06975

    2018 arXiv

  15. [23]

    Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46--51

    2017

  16. [24]

    Peter Norvig. 1987. http://www2.eecs.berkeley.edu/Pubs/TechRpts/1987/5995.html A Unified Theory of Inference for Text Understanding . Ph.D. thesis, EECS Department, University of California, Berkeley

    1987

  17. [25]

    Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of LREC 2018

    2018

  18. [26]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311--318

    2002

  19. [27]

    Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI

    2018

  20. [28]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, page 8

    2019

  21. [29]

    Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of EMNLP 2012, pages 777--789

    2012

  22. [30]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of ACL 2018, pages 784--789

    2018

  23. [31]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP 2016, pages 2383--2392

    2016

  24. [32]

    Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018. Modeling naive psychology of characters in simple commonsense stories. In Proceedings of ACL 2018, pages 2289--2299

    2018

  25. [33]

    Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042

    2018 arXiv

  26. [34]

    Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of EMNLP 2013, pages 193--203

    2013

  27. [35]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series

    2011

  28. [36]

    Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2018. Atomic: An atlas of machine commonsense for if-then reasoning. arXiv preprint arXiv:1811.00146

    2018 arXiv

  29. [37]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728

    2019 arXiv

  30. [38]

    Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603

    2016 arXiv

  31. [39]

    Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge dataset and models for dialogue-based reading comprehension. arXiv preprint arXiv:1902.00164

    2019 arXiv

  32. [40]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    2018 arXiv

  33. [41]

    Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191--200

    2017

  34. [42]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of CVPR 2015, pages 4566--4575

    2015

  35. [43]

    Liang Wang, Meng Sun, Wei Zhao, Kewei Shen, and Jingming Liu. 2018 a . Yuanfudao at semeval-2018 task 11: Three-way attention and relational knowledge for commonsense machine comprehension. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 758--762

    2018

  36. [44]

    Shuohang Wang, Mo Yu, Jing Jiang, and Shiyu Chang. 2018 b . A co-matching model for multi-choice reading comprehension. In Proceedings of ACL 2018, pages 746--751

    2018

  37. [45]

    Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of EMNLP 2018, pages 93--104

    2018

  38. [46]

    Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885

    2018 arXiv

  39. [47]

    Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019 a . Dual co-matching network for multi-choice reading comprehension. arXiv preprint arXiv:1901.09381

    2019 arXiv

  40. [48]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 b . Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  41. [49]

    Haichao Zhu, Furu Wei, Bing Qin, and Ting Liu. 2018. Hierarchical attention flow for multiple-choice reading comprehension. In Proceedings of AAAI 2018

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.