{"id":"cf602743-ff70-456c-8cc9-ec8b25f9a4ee","arxiv_id":"1909.00277","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Cosmos QA is a new multiple-choice reading comprehension benchmark built from personal blogs, where correct answers require commonsense inference beyond the literal text and machines trail humans by about 25 points.","lead":"This paper introduces Cosmos QA, a dataset of 35,588 multiple-choice questions that test whether AI systems can read everyday stories and infer causes, effects, and unstated facts. The best model reaches 68.4 percent accuracy versus 94 percent for humans, so the benchmark exposes a clear gap in commonsense reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The machine-vs-human gap may be overstated because the 94% human figure comes from only 200 test questions with a 3-worker majority vote and no error bar; the most load-bearing uncertainty is whether the official test set itself is answerable at that high rate by independent workers.","rationale":"The reader's verdict is CONDITIONAL with the weakest assumption being annotation reliability and the small 200-question human sample. My stress-test agrees that this is the load-bearing concern: the headline gap of 25.6 points is the central quantitative claim, and it depends entirely on the 94.0% human number. The validation filter (2-of-3 agreement) guarantees at least a weak majority endorsement for the gold labels, but it does not guarantee that the intended answer is clearly the best answer, nor does it bound how often a different answer would be selected by an independent majority. The human performance estimate is not a measurement of the gold-label agreement rate on the test set; it is a separate small-sample measurement. The test set is additionally drawn from 'excellent workers' (Section 2.5), so the gold labels may be systematically idiosyncratic relative to the general population. I am not raising this as a fatal flaw: the resource is substantial and the model-vs-model comparisons are meaningful. But the benchmark's claim of a large headroom is precisely what would be invalidated if independent human accuracy were lower. The check I propose is cheap and standard: a larger, independent human evaluation with more annotators and confidence intervals. This is the same concern the reader identified, and the paper does not report inter-annotator agreement or a confidence interval for the 94% figure, so the conditional verdict is appropriate. If the check confirms ~94% human accuracy, the benchmark claim would be strengthened and the verdict could move to ACCEPT; if not, the paper's central argument about headroom would need to be revised.","tokens_in":17894,"tokens_out":1695,"duration_ms":13956,"concrete_test":"Recruit a fresh pool of annotators (not the original excellent workers) to answer a larger random sample of the official test set, e.g., 500-1000 questions, with 5-10 judgments per question, and compute human accuracy with a majority vote and a confidence interval. Optionally also compute agreement between the fresh annotators and the original gold labels. If the fresh human accuracy is 90% or lower with non-overlapping confidence intervals, the reported 94.0% ceiling and the 25.6% gap are overstated; if it is 93% or above, the concern is resolved. This directly tests whether the intended answers are the most-plausible answers for independent readers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim of a 25.6-point machine-human gap rests on a human performance estimate of 94.0% from 200 randomly sampled test questions, evaluated by a majority vote of just 3 AMT workers (Section 4.1). This is a small sample with no confidence interval; with 200 questions, the standard error of a 94% accuracy is roughly 1.7 points, so the gap estimate has meaningful uncertainty. More importantly, the same weak-agreement filter used in validation (at least 2 of 3 workers picking the intended answer, Section 2.3) is used for the human ceiling. If the intended answers are not the most-plausible answers for a sizable minority of independent annotators, the true human accuracy on the official test set could be several points lower, shrinking the headroom. This concern is about the reliability of the benchmark's human ceiling, not about the models' 68.4%, which is measured on a much larger set. The test set's construction from a self-selected group of 'excellent workers' (Section 2.5) also means the test set contains their authored questions; the human estimate is not confirmed on that exact test set with an independent, larger sample.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Cosmos QA, a multiple-choice reading comprehension dataset of 35,588 problems built from 21,886 personal-narrative blog contexts, and argues that the dataset requires contextual commonsense reasoning beyond literal text matching. The construction pipeline uses AMT workers to author questions and answers, three workers to validate each question set, a BERT-based heuristic to create 'None of the above' unanswerable instances, and a test set drawn from questions authored by a selected group of high-quality workers. The empirical section evaluates several neural baselines and a new BERT-with-multiway-attention model, reporting 68.4% test accuracy versus a claimed 94.0% human performance, together with ablations, knowledge-transfer experiments, error analysis, and a generative evaluation with GPT-2.","tokens_in":18336,"tokens_out":7005,"duration_ms":61895,"significance":"Cosmos QA is a substantial and well-documented dataset contribution, and the release of the data, code, and leaderboard is a concrete strength. The three-stage validation, the public leaderboard, the detailed error analysis, and the transparent ablations (including the question-removal ablation in Table 4) are valuable for the community. If the 94.0% human ceiling is reliable, the 25.6-point machine-human gap provides a meaningful target for commonsense reading comprehension research. The main caveat is that the headline human number rests on a small sample and on majority voting over three annotators, so the precise size of the gap is less certain than the paper suggests.","major_comments":[{"comment":"The human performance estimate of 94.0% is based on 200 randomly sampled test-set questions with majority vote over three AMT workers, and no confidence interval or inter-annotator agreement is reported. For a binomial proportion near 0.94 with n=200, the standard error is about 1.7 points, so the true human accuracy could be several points lower; if it were, say, 90%, the headline 25.6-point gap would shrink to about 21.6 points. I ask the authors to report a substantially larger human evaluation (or at least a confidence interval and agreement statistics on the existing 200 questions), and to state explicitly whether these 200 questions are a random sample of the Section 2.5 test set.","section":"Section 4.1, Table 3"},{"comment":"The validation rule accepts a question set when at least two of three workers select the intended answer and all workers rate the paragraph, question, and answers as satisfactory. This means a question with one dissenting worker is retained. The paper does not report the distribution of agreement levels (2/3 versus 3/3) for accepted questions, either in validation or in the human-performance sample. If a large fraction of accepted questions are only 2/3-agreed, the gold labels may be more ambiguous than the 94.0% human ceiling implies, and the benchmark's reliability as ground truth is weakened. Please report the agreement distribution and consider reporting human accuracy on the subset with 3/3 agreement.","section":"Section 2.3"},{"comment":"The statement that 93.8% of Cosmos QA requires contextual commonsense reasoning is based on a manual categorization of 500 answerable questions, but the paper does not describe how many annotators performed this categorization or provide inter-annotator agreement. Since this statistic is central to the dataset's claimed novelty over MCScript and other reading comprehension datasets, the categorization protocol should be documented and the reliability of the 93.8% figure should be quantified.","section":"Section 2.6, Table 2"},{"comment":"The question-ablation result shows that BERT-FT with only paragraph and answer (A|P) reaches 64.5% test accuracy, only 2.6 points below the full (A|P,Q) model. This raises the concern that a sizable fraction of Cosmos QA questions are predictable from the paragraph and answer set alone, which bears directly on the benchmark's validity as a reading comprehension task. The paper discusses this as a consequence of limited question-type diversity, but it does not quantify how many instances are answerable without the question. I request an analysis that estimates this fraction, for example by examining the questions that the (A|P) model answers correctly, and a discussion of the implications for the 'contextual' claim.","section":"Section 5.1, Table 4"}],"minor_comments":[{"comment":"The abstract says 35,600 problems, while Table 1 totals 35,588; please make the numbers consistent.","section":"Abstract, Table 1"},{"comment":"Table 7 lists the Cosmos QA size as 31.8K, which differs from the 35,588 total in Table 1; please clarify which set this count refers to.","section":"Table 7"},{"comment":"There is a typo in 'at least on worker' which should be 'at least one worker'.","section":"Section 2.4"},{"comment":"The caption contains the typo 'approachs', which should be 'approaches'.","section":"Table 3 caption"},{"comment":"The figure shows the placeholder text 'Chart Title' in the displayed labels; this should be removed before publication.","section":"Figure 2"},{"comment":"The text contains the typo 'None of the aboce', which should be 'None of the above'.","section":"Section 5.4"},{"comment":"The classification loss equation has an unbalanced parenthesis: the denominator should be written as \\(\\sum_{j=1}^4 \\exp(W_f^\\top F_j)\\), with the closing parenthesis matching the opening one after 'log'.","section":"Section 3.1"},{"comment":"The claim that more than 83% of correct answers are not stated in the paragraphs should be accompanied by a description of how this percentage was computed (manual sample or automatic span matching), since it is used to motivate the task design.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong dataset contribution and the concerns above are addressable with additional experiments and analysis rather than fundamental flaws. I would support acceptance once the human-ceiling estimate is reinforced and the agreement statistics are reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a dataset paper, and the dataset is the contribution. Cosmos QA gives the MRC community a large, non-extractive, multiple-choice benchmark that actually forces reading between the lines: 35,588 problems over personal blog narratives, with the vast majority of answers requiring causal or social commonsense rather than text matching. That design choice, plus the unanswerable 'None of the above' option and a generative evaluation setup, makes it a resource people keep using. The model work is fine but incremental: BERT with multiway attention beats plain BERT-FT by about a point, which is not the story. The story is the resource itself and the documented three-stage validation pipeline, which is careful and reproducible.\n\nThe soft spots are real but do not sink the paper. The 94% human ceiling comes from 200 test questions, judged by majority vote of three AMT workers, with no confidence interval. That is a small sample; the standard error alone is roughly 1.7 points, and the same 2-of-3 agreement filter used in validation is baked into the human estimate. The test set is also drawn from a self-selected group of 'excellent workers', so the human number may not generalize to the full test distribution. The stress-test note is right to flag this, but I would not call it a load-bearing flaw: the gap between even 90% and 68% is still large, and the dataset is useful even if the headroom is a few points smaller than advertised. The reliance on BERT to create false negatives and to segment paragraphs is a mild circularity concern, but answers are human-authored and human-validated, so 'the model is evaluating itself' would be an overstatement.\n\nThe citation pattern is honest; the comparisons to ReCoRD, CommonsenseQA, and SWAG are accurate, and the limitations are disclosed. This is a serious piece of benchmark engineering written by people who know what they are doing.\n\nWho should read it? Anyone building or testing MRC or commonsense models. It is worth a serious referee; I would accept it with minor revisions and ask for a confidence interval on the human estimate and a clearer statement about the excellent-worker test-set selection.","headline":"Cosmos QA is a genuinely useful benchmark that deserves peer review; the human-ceiling number is softer than it looks, but the dataset carries the paper.","tokens_in":18696,"tokens_out":1374,"would_cite":true,"duration_ms":15482,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Cosmos QA, a 35,588-question benchmark asserting that machines (68.4%) still lag humans (94.0%) when reading comprehension requires commonsense inference beyond the text.","keywords":["commonsense reasoning","reading comprehension","multiple-choice question answering","contextual inference","benchmark dataset","BERT","counterfactual reasoning","crowdsourcing"],"falsifier":"Take a random sample of Cosmos QA test questions and re-annotate them with unanimous agreement among five or more annotators, then recompute both the human ceiling and the best model's accuracy on that stricter subset. If the stricter subset shrinks the machine-human gap substantially, the reported 25.6-point gap is inflated by annotation noise; if the gap persists on unanimous questions, the benchmark's difficulty claim is supported.","tokens_in":17731,"feed_emoji":"🧠","tokens_out":7483,"duration_ms":62109,"temperature":0.7,"pith_summary":"The paper sets out to establish that contextual commonsense reasoning is a distinct, measurable component of reading comprehension, and that it can be benchmarked at scale. To do so it introduces Cosmos QA, a dataset of 35,588 multiple-choice problems built from personal blog narratives, where the correct answer is not a text span but an inference about causes, effects, motives, or counterfactuals. The paper reports that the strongest model it tests, a BERT variant with multiway attention, reaches 68.4 percent accuracy while human annotators reach 94.0 percent on a 200-question sample, leaving a 25.6-point gap. If the benchmark is valid, it gives the field a shared yardstick for progress on reading between the lines.","feed_headline":"Machines trail humans by 25.6 points on commonsense reading","feed_subtitle":"Cosmos QA forces models to infer causes, effects, and counterfactuals beyond the text.","key_machinery":"The load-bearing object is the dataset itself: 35,588 crowdsourced question sets built from a large blog corpus, with a validation filter that keeps a question only if at least two of three workers pick the intended answer and all judge it satisfactory. On top of it, the paper's best model is BERT fine-tuned for the answer-selection task with multiway bidirectional attention, which computes separate attentive representations of the paragraph conditioned on the question, on the answer, and on both, then fuses them for classification. The constructor also creates 'None of the above' questions from items that workers could answer without the context, and uses three BERT next-sentence-prediction models to replace the least challenging negative answer with 'None of the above' in 70% of training questions.","core_discovery":"Cosmos QA is a large-scale multiple-choice machine reading comprehension benchmark in which roughly 93.8% of questions require commonsense inference rather than literal text matching, and more than 83% of correct answers are not stated in the context paragraph. The contexts are 21,886 everyday narratives from personal weblogs, and questions target causes, effects, facts about entities, and hypotheticals and counterfactuals. The paper's central empirical claim is that contemporary models, including a fine-tuned BERT with multiway attention, solve this task only at 68.4% accuracy, well below the 94.0% human estimate, so there is a substantial machine-human gap in contextual commonsense reasoning.","pith_inferences":["One consequence the paper leaves implicit: if the two-of-three validation filter admits questions answerable by surface cues, then the 25.6-point gap may overstate the difficulty of true commonsense inference; a stricter unanimous-agreement subset would be a sharper test.","The 'None of the above' mechanism is likely to interact with model calibration: a model with no explicit rejection head may under-predict the unanswerable option, and this could be tested by measuring accuracy on the 6-8% unanswerable split separately.","A natural extension would be to build parallel versions of Cosmos QA in other languages or with controlled context perturbations, such as swapped entities, to separate world-knowledge inference from narrative-coherence inference."],"forward_implications":["If Cosmos QA measures what it claims, then any future model that closes the 25.6-point gap will have demonstrated a real gain in contextual commonsense reasoning.","The finding that multiway attention improves over plain BERT fine-tuning suggests that explicitly modeling paragraph-question-answer interactions helps on inference-heavy reading comprehension, not just literal question answering.","The reported 83% non-literal answer rate means span-extraction and lexical-overlap models should be expected to fail on this task, making it a stress test for entailment-style reasoning.","Because the dataset supports generative evaluation, it can be used to assess open-ended commonsense generation, not only multiple-choice selection.","Knowledge-transfer results indicate that fine-tuning on a document-level reading-comprehension dataset and on a grounded commonsense-inference dataset both transfer to Cosmos QA, so it can serve as a downstream evaluation for pretraining objectives."],"supporting_citations":[{"why":"Supplies the corpus of personal narratives from which the 21,886 context paragraphs are drawn.","marker":"Gordon and Swanson, 2009"},{"why":"Provides the Spinn3r blog dataset that the narrative corpus is extracted from.","marker":"Burton et al., 2009"},{"why":"Defines SQuAD, the literal span-extraction benchmark that Cosmos QA contrasts against.","marker":"Rajpurkar et al., 2016"},{"why":"Provides the pre-trained BERT model that all strong baselines and the proposed architecture fine-tune.","marker":"Devlin et al., 2018"},{"why":"Introduces three-way attention for commonsense machine comprehension, the direct precursor of the multiway attention used here.","marker":"Wang et al., 2018a"},{"why":"Supplies RACE, the large-scale reading comprehension dataset used in knowledge-transfer experiments.","marker":"Lai et al., 2017"},{"why":"Supplies SWAG, the grounded commonsense inference dataset used in the same transfer experiments.","marker":"Zellers et al., 2018"},{"why":"Introduces SQuAD 2.0 with unanswerable questions, the pattern the paper adapts for 'None of the above' items.","marker":"Rajpurkar et al., 2018"},{"why":"Provides GPT-2, the generator used for the paper's generative evaluation of Cosmos QA.","marker":"Radford et al., 2019"},{"why":"Provides MCScript, whose 27.4% commonsense-inference rate is the comparison point for the 93.8% figure.","marker":"Ostermann et al., 2018"}],"fun_headline_variants":["Machines struggle to read between lines in Cosmos QA test","Commonsense reading gap: machines 68%, humans 94%","New benchmark reveals 25-point AI gap in commonsense reading","Cosmos QA: AI falls short on inference-based reading","Reading between lines: AI trails humans by 25 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark rests on the assumption that the crowdsourced agreement filter, which keeps a question when at least two of three workers pick the intended answer and find the item satisfactory, actually isolates questions that require contextual commonsense reasoning rather than surface cues or general world knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Machines struggle to read between lines in Cosmos QA test","Commonsense reading gap: machines 68%, humans 94%","New benchmark reveals 25-point AI gap in commonsense reading","Cosmos QA: AI falls short on inference-based reading","Reading between lines: AI trails humans by 25 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2692,"prompt_tokens":882,"completion_tokens":1810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":1724}},"tokens_in":498,"tokens_out":1810,"duration_ms":11269,"temperature":1.0,"reasoning_tokens":1724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:56:23.069807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Cosmos QA test questions and re-annotate them with unanimous agreement among five or more annotators, then recompute both the human ceiling and the best model's accuracy on that stricter subset. If the stricter subset shrinks the machine-human gap substantially, the reported 25.6-point gap is inflated by annotation noise; if the gap persists on unanimous questions, the benchmark's difficulty claim is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the corpus of personal narratives from which the 21,886 context paragraphs are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Spinn3r blog dataset that the narrative corpus is extracted from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SQuAD, the literal span-extraction benchmark that Cosmos QA contrasts against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies RACE, the large-scale reading comprehension dataset used in knowledge-transfer experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SWAG, the grounded commonsense inference dataset used in the same transfer experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces SQuAD 2.0 with unanswerable questions, the pattern the paper adapts for 'None of the above' items."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides GPT-2, the generator used for the paper's generative evaluation of Cosmos QA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MCScript, whose 27.4% commonsense-inference rate is the comparison point for the 93.8% figure."}],"review_version":1}