{"id":"93f5f22f-4d9c-47f7-b17c-c3f19c5a43eb","arxiv_id":"1908.06725","paper_version":5,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Pre-training BERT on automatically generated multiple-choice questions from ConceptNet and Wikipedia improves commonsense benchmarks and leaves GLUE performance essentially unchanged.","lead":"The authors build a 16.3 million question commonsense quiz by pairing ConceptNet facts with Wikipedia sentences, masking a concept, and selecting confusable distractors. Pre-training BERT on this quiz lifts CommonsenseQA and Winograd Schema Challenge scores without hurting performance on general language tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: footnote 4 promises to compare BERT CS with fine-tuning original BERT on CSQA-train + DAMS, but that experiment is absent; without it, the 5.5-point CSQA gain could reflect extra ConceptNet-derived training data rather than commonsense pre-training.","rationale":"The paper's central claim is that an AMS-constructed commonsense MCQA dataset, used as a pre-training task, improves commonsense reasoning without degrading general representation. The strongest evidence is the CSQA gain (62.2 vs 56.7), the WSC gain (75.5 vs 72.2), and GLUE parity. The WSC result is genuinely outside ConceptNet, and the ablations show natural-language MCQA beats triple input and random distractors; I credit those. However, the most load-bearing condition for the CSQA half of the claim is that the gain comes from the pre-training procedure rather than from adding ConceptNet-derived training data in the same format as the target task. The authors explicitly flag this confound in footnote 4 and promise a control that is absent. Because CSQA is itself built from ConceptNet, DAMS and CSQA share vocabulary, relation types, and distractor style; extra exposure to that distribution during pre-training could plausibly account for all or most of the 5.5 points. The WSC improvement makes the overall method interesting, but it does not rescue the CSQA-specific causal claim, and the paper's headline emphasizes CSQA. Distant supervision validity is a related but less decisive issue: even a noisy DAMS would still be a large in-domain cloze corpus, and the promised control would reveal whether the noise matters. I therefore agree with the reader's CONDITIONAL verdict; the missing control is the same concern reflected in the reader's rationale, though not the reader's stated weakest assumption.","tokens_in":11553,"tokens_out":6156,"duration_ms":67723,"concrete_test":"Run the control promised in footnote 4: take the released BERTlarge checkpoint and fine-tune it on CSQA-train augmented with a matched subsample of DAMS (same five-choice format; match the number of optimizer steps or total token exposure to the BERT CSlarge pipeline, including the 1-epoch pre-training and 2-epoch CSQA fine-tuning), then evaluate on the CSQA test set. If this model reaches or exceeds 62.2%, the gain is attributable to in-domain training data and the proposed pre-training stage is not the cause; if it remains near the 56.7% baseline, the AMS pre-training is doing real work. Report accuracy over the same five random seeds for a fair comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The CSQA result does not isolate the proposed pre-training mechanism. Both DAMS and CSQA are generated from ConceptNet, and the paper's own footnote 4 acknowledges that DAMS 'may be considered as an augmented data for CSQA' while promising a control: fine-tuning the original BERT on CSQA-train + DAMS. That comparison is not reported. In the current design, the only difference between BERT CSlarge and the BERTlarge baseline is pre-training on 16.3M MCQA examples drawn from the same knowledge graph and in the same multi-choice format as CSQA. The 5.5-point gain could therefore come from (a) exposure to a large volume of ConceptNet-derived, in-domain cloze data, (b) the particular pre-training schedule, or (c) genuine transfer of commonsense knowledge into the representation. Only the promised control separates these explanations. The WSC result is a useful external check, and I credit it, but the headline CSQA number is the largest claimed gain and is confounded. This is not an internal inconsistency, but it is a missing experimental condition that the authors themselves flagged; until it is run, 'incorporating commonsense knowledge' is not established as the cause of the improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-step AMS (align, mask, select) procedure to automatically construct a multi-choice question answering corpus, DAMS, from ConceptNet triples aligned with English Wikipedia sentences. A masked concept is treated as the correct answer, and distractors are selected from ConceptNet triples sharing the same relation and the unmasked concept. The authors initialize BERT-base and BERT-large from released checkpoints, continue pre-training on DAMS with a multi-choice QA objective to obtain BERT CSbase and BERT CSlarge, and then fine-tune on CSQA, WSC/WNLI, and GLUE. They report that BERT CSlarge reaches 62.2% on CSQA (5.5 points above BERT-large and 4.0 above CoS-E), 75.5% on WSC (3.3 above previous state of the art), and roughly unchanged performance on eight GLUE tasks. They also perform ablations over data construction choices and pre-training tasks on BERT-base.","tokens_in":11799,"tokens_out":8301,"duration_ms":87319,"significance":"If the results hold, the paper contributes a simple, knowledge-source-agnostic pre-training recipe for injecting structured commonsense knowledge into language representations without task-specific architectures, along with a large automatically generated MCQA corpus. The WSC result is a genuinely external check that the benefit transfers beyond CSQA, and the ablation study comparing triples versus natural-language sentences, random versus relation-filtered distractors, and MCQA versus MLM is informative. The method is easy to reproduce in principle and likely to be useful to the community. However, the headline CSQA improvement is currently confounded by the overlap between the pre-training source and the evaluation benchmark, and the central causal claim that commonsense knowledge is incorporated into the representations is not yet established.","major_comments":[{"comment":"The main CSQA claim is not isolated from a simple data-augmentation effect. In Table 4, BERT CSlarge is compared with BERT-large after BERT CSlarge has been pre-trained on 16.3 million ConceptNet-derived MCQA examples, and CSQA itself is a ConceptNet-derived benchmark. Footnote 4 explicitly acknowledges that DAMS 'may be considered as an augmented data for CSQA' and promises a control experiment: fine-tuning the original BERT on CSQA-train plus DAMS. That control is not reported anywhere in the paper. Until it is run, the 5.5-point CSQA gain is equally consistent with the model having seen a large volume of in-domain, ConceptNet-style cloze data in the same multi-choice format, rather than with commonsense knowledge being transferred into the representation. Please run the promised control with matched fine-tuning budgets and report whether the CSQA conclusion changes.","section":"Section 4.1, Table 4, footnote 4"},{"comment":"The distant-supervision validity of the align step is not quantified. The method assumes that if a Wikipedia sentence contains exact string matches of concept1 and concept2 from a ConceptNet triple, then masking one concept produces a question whose correct answer is the other concept. The paper cites Riedel et al. (2010) to acknowledge noise in distant supervision, but it provides no estimate of how often the aligned sentences actually express the intended relation in DAMS. If a large fraction of the 16.3 million labels are false positives from homonyms, coincidental co-occurrence, or independent mentions, then the pre-training could teach shallow dataset-specific cues rather than general commonsense. Please provide a human-annotated or automatically estimated precision sample of the alignments, and ideally analyze whether filtering by alignment confidence changes the CSQA or WSC results.","section":"Section 3.2, Table 2"},{"comment":"The reported performance-curve procedure is not fully specified. The text says that 'we remove the results that are significantly lower than the mean' without defining the criterion, the window over which the mean is computed, or how many seeds were removed. This makes the standard-deviation bars in Figure 1 non-reproducible and potentially optimistic, and it weakens the comparison between BERT CSbase and BERT CSlarge convergence behavior. Please specify the exact outlier-removal rule and report all seed results, or use a standard robust summary such as the median with interquartile range.","section":"Section 5.2, Figure 1"}],"minor_comments":[{"comment":"Please clarify whether the 'BERTlarge + MCQA' row is a new baseline computed by the authors with the same hyperparameter search as BERT CSlarge, or an external result; the text describes the setup for the authors' models but does not explicitly state the provenance of this row.","section":"Section 4.2, Table 5"},{"comment":"The paper uses the WSCR test partition as a development set for selecting alpha and beta. Please state explicitly whether the 273-instance WSC test set is disjoint from WSCR; if there is any overlap, the reported WSC numbers may be optimistic for hyperparameter selection.","section":"Section 4.2"},{"comment":"The claim that BERT CSbase is 'slightly better' than BERT base on GLUE is based on point estimates without variance or significance testing; please report per-seed standard deviations or another measure of variability.","section":"Section 4.3, Table 6"},{"comment":"The ablation results are also single point estimates; given that fine-tuning uses five seeds with the best development model selected, please report the variance or at least the seed-level values for the key comparisons in Table 7.","section":"Section 5.1, Table 7"},{"comment":"There are minor typographical and formatting issues: 'commmonsense' in the introduction, and the dataset symbol appearing as 'DAM S' in several places due to subscript spacing; these should be cleaned up.","section":"Section 1 and throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and the WSC result makes the central idea credible, so I do not see grounds for rejection. The decisive issue is the missing control promised in footnote 4: before the CSQA claim can be attributed to commonsense knowledge incorporation rather than in-domain data augmentation, that experiment must be run. I would also encourage the authors to release DAMS and the pre-trained checkpoints, since the paper does not currently state whether these resources will be available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the thing to know: this is a solid empirical method paper with a genuinely useful data-construction trick, but the headline CSQA number does not isolate what the authors claim. Both DAMS and CSQA derive from ConceptNet, and the paper's own footnote 4 admits DAMS \"may be considered as an augmented data for CSQA\" while promising the control of fine-tuning original BERT on CSQA-train plus DAMS. That control never appears. Without it, the 5.5-point gain on CSQA could just reflect more in-domain training data, not commonsense knowledge being incorporated into representations.\n\nWhat's actually new: the AMS procedure. Aligning ConceptNet triples to Wikipedia sentences by exact string match, masking one concept, and selecting distractors that share the same relation and unmasked concept is a clean, clearly specified pipeline that produces 16.3 million MCQA samples. This goes beyond Sun et al. (2019a), which pretrained directly on triple strings. The ablations are informative: natural-language MCQA beats triple pretraining, relation-matched distractors help, and MLM hurts when combined with MCQA. The WSC result (75.5 vs. 72.2 for the previous best) is on a benchmark not built from ConceptNet, which is a genuinely useful external check and supports the claim that something general is happening. GLUE results show no meaningful regression. The paper is honestly written and the method sections are easy to follow.\n\nSoft spots, in proportion: the missing control is the main one, and it is a load-bearing omission rather than a minor quibble. There are also no error bars or significance tests on the headline CSQA and WSC test numbers; selecting the best of five seeds on the dev set can inflate results. The distant supervision noise from exact string matching is acknowledged but never quantified, so we don't know how many of the 16.3M labels are actually answerable. No code or data release, which limits reproducibility in a line of work where dataset construction is the contribution.\n\nThis is not a takedown. The method is plausible, the ablations are thoughtful, and the WSC result suggests the approach has real value. But as written, the causal claim that pretraining on DAMS injects commonsense knowledge is not established; the control experiment could settle it in an afternoon. The paper deserves a serious referee, but the referee should require that control, plus variance reporting and data release, before acceptance.\n\nBottom line: worth engaging with, worth citing for the method, and worth sending to peer review — but with the expectation of heavy revision.","headline":"A clean empirical paper with a genuinely useful AMS data-construction pipeline, but the headline CSQA gain is confounded by the shared ConceptNet source and the missing control the authors themselves promise in footnote 4.","tokens_in":12364,"tokens_out":1470,"would_cite":true,"duration_ms":16809,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pretraining BERT on a 16.3-million-question multiple-choice dataset built by aligning ConceptNet triples with Wikipedia sentences and masking one concept improves commonsense benchmarks substantially while preserving…","keywords":["commonsense reasoning","pretrained language models","BERT","multi-choice question answering","ConceptNet","distant supervision","CommonsenseQA","Winograd Schema Challenge"],"falsifier":"Take a random sample of about 1,000 sentences from the DAMS dataset, show each sentence with the masked concept replaced by a blank and the correct concept plus distractors, and ask human annotators whether the correct concept is entailed by the sentence alone; if a large fraction of items are judged unanswerable or the correct answer is not entailed, the distant-supervision premise fails. A cheaper computational check is to train BERT-CS on the same questions with the correct answer randomly permuted among candidates and measure how much of the CSQA gain survives.","tokens_in":11327,"feed_emoji":"🧠","tokens_out":5649,"duration_ms":50467,"temperature":0.7,"pith_summary":"This paper proposes a way to push commonsense knowledge into pretrained language models without losing their general language abilities. The method, called align-mask-select (AMS), builds a 16.3-million-question multiple-choice dataset by matching ConceptNet triples to Wikipedia sentences, masking one concept, and choosing distractor concepts that share the same relation. Pretraining BERT on this dataset before fine-tuning raises accuracy to 62.2% on CommonsenseQA, 5.5 points above BERT-large and 4.0 points above the prior best, and to 75.5% on the Winograd Schema Challenge, while staying within about one point of BERT on GLUE. The core claim is that commonsense can be injected cheaply and automatically, with no extra labels or architecture changes.","feed_headline":"Commonsense pretraining lifts BERT to 62.2% on CommonsenseQA","feed_subtitle":"The same auto-built multiple-choice pretraining reaches 75.5% on Winograd and keeps GLUE scores steady.","key_machinery":"The machinery is the AMS (align-mask-select) data construction pipeline. Align takes a ConceptNet triple (concept1, relation, concept2) and finds a Wikipedia sentence containing both concepts as exact string matches; mask replaces one concept with the token [QW] to form the question; select pulls four distractors from ConceptNet by querying (concept1, relation, *) or (*, relation, concept2), so every candidate shares the same relation and one concept with the correct answer. This yields 16,324,846 questions. The pretraining objective then scores each candidate by a softmax over the [CLS] representation of the concatenated question-candidate sequence, teaching the model whole-concept selection rather than word-level prediction.","core_discovery":"The central claim is that a pure multi-choice question answering pretraining task over distantly supervised natural-language questions transfers commonsense reasoning to BERT. To create the task, the AMS method aligns each ConceptNet relation triple to sentences containing both concepts as exact strings, masks one concept, and selects four distractors that share the same unmasked concept and relation, so the wrong answers are semantically close rather than random. By replacing the masked-language-model and next-sentence objectives with this QA objective, the model learns to choose the concept that the sentence's commonsense relation actually licenses. On CommonsenseQA, BERT-CS-large reaches 62.2% test accuracy, and on WSC it reaches 75.5%, outscoring the previous best BERT-based system by 3.3 points; on eight GLUE tasks it stays comparable to the original BERT.","pith_inferences":["Editorial inference: if the same AMS pipeline were run on a larger or more recent knowledge graph and corpus, the WSC 'non-associative' subset would be the most informative number to watch, since that subset isolates genuine commonsense from statistical antecedent cues.","Editorial inference: the exact-string align step suggests a concrete stress test: replace aligned sentences with paraphrases or use entity linking, and see whether gains persist; if they vanish, the model may be relying on lexical overlap rather than relation semantics.","Editorial inference: the method should apply to other relation-rich pretraining signals, such as temporal, spatial, or causal text, since the objective only needs triples plus a corpus rather than task-specific supervision."],"forward_implications":["Pretraining on distantly supervised natural-language QA can substitute for human-labeled explanation data: BERT-CS-large beats CoS-E, which uses collected explanations, by 4.0 points on CommonsenseQA.","The approach transfers across commonsense benchmarks: the same pretrained model improves both CommonsenseQA and Winograd Schema Challenge, indicating the learned signal is not tied to one dataset's format.","The semantic closeness of distractors matters: compared with randomly chosen distractors, same-relation distractors improve CSQA development accuracy, so harder choice sets force deeper semantic discrimination.","MCQA alone outperforms MCQA combined with masked language modeling; adding MLM's 15% token masking hurts, likely because it moves inputs away from natural sentences.","Commonsense injection does not degrade general language representation: on eight GLUE tasks, BERT-CS-large matches BERT-large and BERT-CS-base slightly exceeds BERT-base."],"supporting_citations":[{"why":"Supplies the base BERT architecture and initialization for the BERT-CS models.","marker":"Devlin et al. (2019)"},{"why":"Supplies ConceptNet, the commonsense knowledge graph whose triples seed the AMS dataset.","marker":"Speer et al. (2017)"},{"why":"Introduces the distant-supervision assumption that AMS relies on for aligning triples to sentences.","marker":"Mintz et al. (2009)"},{"why":"Provides CommonsenseQA, the main benchmark whose state of the art this method improves.","marker":"Talmor et al. (2019)"},{"why":"Provides the CoS-E prior state of the art and the human-collected explanations baseline on CommonsenseQA.","marker":"Rajani et al. (2019)"},{"why":"Provides the WSC baseline and the loss function that BERT-CS-large plus MCQA outperforms.","marker":"Kocijan et al. (2019)"},{"why":"Supplies the WSCR training and test partitions used to fine-tune for the Winograd Schema Challenge.","marker":"Rahman and Ng (2012)"},{"why":"Defines the Winograd Schema Challenge dataset on which the method reports 75.5%.","marker":"Levesque et al. (2012)"}],"fun_headline_variants":["Align, mask, select: pretraining BERT with commonsense QA","Simple pretraining: BERT gets commonsense from QA tasks","QA pretraining from ConceptNet boosts BERT's commonsense score","AMS pretraining: 62.2% CommonsenseQA, 75.5% Winograd","BERT learns commonsense via auto-built QA pretraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a Wikipedia sentence containing both concepts of a ConceptNet triple actually expresses that triple's relation whenever the two concepts appear as exact strings; if that distant-supervision link is frequently wrong, many of the 16.3 million training questions are unanswerable or mislabeled.","fun_headline_variants_meta":{"raw":{"variants":["Align, mask, select: pretraining BERT with commonsense QA","Simple pretraining: BERT gets commonsense from QA tasks","QA pretraining from ConceptNet boosts BERT's commonsense score","AMS pretraining: 62.2% CommonsenseQA, 75.5% Winograd","BERT learns commonsense via auto-built QA pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001463,"raw_usage":{"total_tokens":5866,"prompt_tokens":908,"completion_tokens":4958,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":4863}},"tokens_in":524,"tokens_out":4958,"duration_ms":37714,"temperature":1.0,"reasoning_tokens":4863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:36:37.287893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 1,000 sentences from the DAMS dataset, show each sentence with the masked concept replaced by a blank and the correct concept plus distractors, and ask human annotators whether the correct concept is entailed by the sentence alone; if a large fraction of items are judged unanswerable or the correct answer is not entailed, the distant-supervision premise fails. A cheaper computational check is to train BERT-CS on the same questions with the correct answer randomly permuted among candidates and measure how much of the CSQA gain survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WSCR training and test partitions used to fine-tune for the Winograd Schema Challenge."}],"review_version":1}