{"id":"9b959766-17b9-4591-bfa8-2dad7e00bb1c","arxiv_id":"1908.04530","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding ConceptNet relation-existence and relation-type auxiliary tasks to a BERT multiple-choice MRC model improves accuracy by about 0.7 points on SemEval-2018 Task 11 and Story Cloze Test.","lead":"This paper adds two side-tasks that predict commonsense relations between words to a BERT-based reading-comprehension model, and reports slightly higher accuracy on two story-comprehension benchmarks. The gains are about one point in a hundred, and the paper does not report statistical tests or released code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed 0.7-point gains are within single-run noise, because no error bars, multiple seeds, or significance tests are reported.","rationale":"The reader's weakest assumption is that the reported accuracy gaps are reproducible effects rather than seed or run-to-run noise. My reading of the numbers supports exactly that concern. A 0.7-point gain on a test set of roughly 2,000 to 3,000 examples is within one standard error of the difference, so without multiple seeds or a significance test the headline result is statistically fragile. The proposed concrete test, multi-seed runs with a paired McNemar test, would settle whether the relation-aware tasks actually help. If the gain vanishes under this test, the paper's central claim would need to be revised; if it survives, the conditional verdict could be lifted. I found no internal inconsistency or circularity in the loss formulations, and the auxiliary-task design is plausible as a regularizer. The secondary concern about the under-specified ConceptNet-to-BPE pair construction is real for reproducibility but is less load-bearing than the statistical issue, because even a perfect implementation would still need to demonstrate that the observed gains are not noise. Since the reader already identified the same central weakness and set a conditional verdict, no change to the verdict is needed.","tokens_in":6448,"tokens_out":6300,"duration_ms":65459,"concrete_test":"Run the same fine-tuning procedure (identical data splits, hyperparameters, and preprocessing) for BERT-base and BERT-base with relation-aware tasks using 10 different random seeds on SemEval-2018 Task 11 and Story Cloze Test, recording test accuracy for each seed. Report the mean, standard deviation, and a paired McNemar test on the test predictions. If the mean gain is less than twice the seed-to-seed standard deviation or the McNemar p-value exceeds 0.05 on either dataset, the claimed improvement is not established and the conclusion should be softened to note the effect is within noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the two relation-aware auxiliary tasks improve MRC accuracy—rests entirely on single-run accuracy comparisons in Tables 2, 3, and 4. The reported gaps are small relative to the test-set sizes. On SemEval-2018 Task 11 (n=2,797), the 0.70-point gain (87.53 to 88.23) corresponds to about 20 additional correct examples; a two-proportion standard error for this difference is about 0.87 percentage points, so the gap is less than one standard error. On Story Cloze Test (n=1,871), the 0.70-point gain (86.7 to 87.4) is about 13 examples with a standard error of about 1.1 percentage points. The dev ablation on SemEval (n=1,411) shows 87.51 to 88.25, again within one standard error. No seed-level variance, confidence intervals, or significance tests are reported, and no code is released to check run-to-run variability. Under the null hypothesis of no true improvement, a positive gap of this size is entirely plausible. Therefore, the empirical foundation of the headline claim is unverified: the observed 'superior performance' may be an artifact of optimizer or initialization noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-task learning extension of BERT-base for multi-choice commonsense reading comprehension. In addition to the standard answer-selection loss (Eq. (1)), the model is trained with two auxiliary objectives derived from ConceptNet: a relation-existence binary task over token pairs from the document/question (sentence A) and option (sentence B) (Eqs. (2)-(3)), and a relation-type classification task over R=34 relation types (Eqs. (4)-(5)), combined in Eq. (6). Experiments on SemEval-2018 Task 11 and Story Cloze Test report accuracy improvements over BERT(base): 88.23 vs. 87.53 on SemEval and 87.4 vs. 86.7 on Story Cloze, with a dev ablation showing +0.74 from adding both auxiliary tasks. The authors conclude that relation-aware multi-task learning improves commonsense MRC.","tokens_in":6656,"tokens_out":8390,"duration_ms":76213,"significance":"The proposed formulation is clean and model-agnostic; applying the same auxiliary tasks to TriAN and obtaining a gain suggests the idea may transfer. The use of ConceptNet-derived labels avoids the circularity that would arise if the auxiliary labels were functions of the MRC answer labels. If the reported gains were robust, the paper would provide a lightweight way to inject relational commonsense knowledge into pre-trained LMs without architectural changes. The main limitation is that the empirical evidence is statistically unverified: single-run accuracy differences of 0.7 points on test sets of about 2,000-3,000 examples are within one standard error. With proper multi-seed evaluation and significance testing, the paper could support its claim.","major_comments":[{"comment":"The central empirical claim is not supported by the reported statistics. All accuracies in Tables 2 and 3 are single-run numbers, and the reported gains are within one standard error of zero under a two-proportion test: on SemEval-2018 Task 11 (n=2,797), the 87.53 vs. 88.23 difference corresponds to about 20 examples with SE ≈ 0.87 percentage points; on Story Cloze Test (n=1,871), the 86.7 vs. 87.4 difference is about 13 examples with SE ≈ 1.1 percentage points; on the SemEval dev set (n=1,411), the 87.51 vs. 88.25 difference in Table 4 is about 10 examples with SE ≈ 1.2 percentage points. No confidence intervals, multiple seeds, or significance tests are reported, and no code is released to assess run-to-run variability. Under the null hypothesis of no improvement, a positive gap of this size is entirely plausible, so the conclusion of 'superior performance' in the abstract is not established. The authors should report mean and standard deviation over at least several fine-tuning seeds and a paired significance test (e.g., McNemar's test) for the test-set comparisons.","section":"§4.3, Tables 2-4"},{"comment":"The construction of the auxiliary relation labels is underspecified and not reproducible as written. The text says that 'the concept' is converted to a set of BPE tokens with beginning indices i and j, but it does not state which spans of the input sequence are treated as concepts, how ConceptNet concepts are matched to BPE tokens (e.g., exact match, stemmed match, or phrase matching), or how negative pairs are sampled under the 1:γ ratio. It is also unclear how the ground-truth relation index k in Eq. (5) is chosen when multiple ConceptNet relations hold for the same token pair. These details are needed to reimplement the method and to assess whether the auxiliary supervision is as the authors intend.","section":"§3.3, Eqs. (2)-(5)"}],"minor_comments":[{"comment":"In the sentence 'Then, we take one step further to decide what is the right type of the relation in relation-existence,' the final phrase should read 'in the relation-type task.'","section":"§3.2, paragraph after Eq. (1)"},{"comment":"The third example in Table 5 is explained incorrectly: the text says 'the relation type (kettle, UsedFor, boil water) in option (A),' but option (A) is 'To drink from' and this fact supports option (B), the correct answer. Please correct the option reference.","section":"§4.3, Table 5"},{"comment":"The citation 'Following [4]' for converting concepts to BPE tokens is misleading because [4] is the BERT paper; please cite the actual source of the concept-to-BPE conversion procedure or describe it directly.","section":"§3.3"},{"comment":"The hyperparameters λ1, λ2, and γ are fixed at 0.5, 0.5, and 4.0 with no sensitivity analysis; given the small reported gains, a sentence stating that the results are stable under reasonable variation would strengthen the paper.","section":"§4.2, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript is a short paper with a simple and sensible idea. The main obstacle is statistical: the reported improvements are within one standard error. I recommend major revision requiring multi-seed results and significance tests. The reproducibility of the auxiliary-label construction also needs to be addressed. The novelty is modest but acceptable for a conference-level paper; there is no circularity issue. No concerns about authorship or citation behavior beyond the [4] citation issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, incremental extension of known relation-aware auxiliary tasks to a BERT-base multi-choice MRC model. The gains it reports are consistently positive on two benchmarks, but the reported 0.7-point gaps are within single-run noise as far as the paper demonstrates, so the empirical claim is not yet load-bearing.\n\nWhat's genuinely new: the specific combination of relation-existence and relation-type prediction as two separate auxiliary tasks jointly trained with BERT for commonsense MRC. It is a small twist on earlier work—TriAN and commonsense-based pre-training [17] already used relation-aware tasks—but the two-task decomposition is neat, and the dev ablation shows that both tasks add a bit and that LRE helps more than LRT. Equations are standard and consistent. Labels come from ConceptNet, independent of the MRC labels, so no circularity. I agree with the reader: no fundamental flaw in the method.\n\nSoft spots: the central claim of \"superior performance\" rests on single-run accuracies. The SemEval test gap (87.53 to 88.23) is about 20 extra correct examples out of 2,797; the Story Cloze gap (86.7 to 87.4) is about 13 out of 1,871. The standard errors on those differences are roughly 0.9 and 1.1 percentage points, so both gaps are less than one standard error. The dev ablation gap is of similar size. Without multiple seeds, error bars, or significance tests, the improvement could easily be optimizer or initialization noise. Also, Section 3.3 is underspecified: how ConceptNet concepts are matched to BPE tokens and how negative pairs are sampled is not stated, which limits reproducibility. No code was released. These are genuine weaknesses, but proportionate: for a 4-page CIKM short paper, single-run reporting is common, and the weaknesses are mostly about evidence quality rather than the method's logic. The method is plausible and the ablation points in the right direction.\n\nWho this is for: people working on knowledge-enhanced MRC or multi-task learning for NLP. They get a small, plausible recipe plus a useful reminder about statistical significance in benchmark differences. I would not cite it as a proven result in my own work, but I would expect it mentioned in a survey of knowledge-aware reading comprehension.\n\nRecommendation: send to peer review. A good reviewer can push for multiple seeds with variance reporting and for a precise description of the relation-pair construction. A serious referee will help the authors make the claim honest.","headline":"A clean incremental recipe for adding relation-aware auxiliary tasks to BERT-base MRC; the reported gains are positive but within single-run noise, so the evidence is provisional rather than demonstrative.","tokens_in":7227,"tokens_out":2385,"would_cite":false,"duration_ms":23639,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding relation-existence and relation-type auxiliary tasks to BERT-based multiple-choice reading improves commonsense accuracy on SemEval-2018 Task 11 and Story Cloze Test.","keywords":["machine reading comprehension","commonsense reasoning","multi-task learning","relation knowledge","ConceptNet","BERT","SemEval-2018 Task 11","Story Cloze Test"],"falsifier":"Run the basic BERT model and the full relation-aware multi-task model on SemEval-2018 Task 11 dev and Story Cloze Test across ten random seeds with the same hyperparameters; if the average accuracy difference is smaller than the standard deviation across seeds, the central claim is not supported.","tokens_in":6195,"feed_emoji":"🧠","tokens_out":7481,"duration_ms":65605,"temperature":0.7,"pith_summary":"The paper tries to show that a multiple-choice machine comprehension model can do better on commonsense questions if, while learning to pick the right answer, it is also trained to predict whether two words from the story and answer option are linked in an external commonsense knowledge base, and what type of relation links them. The two auxiliary tasks turn ConceptNet's triplets into supervised signals that share the BERT encoder, so no new model structure is needed. On SemEval-2018 Task 11 the joint model reaches 88.23% versus 87.53% for BERT-base, and on Story Cloze Test it reaches 87.4% versus 86.7%, which the authors read as evidence that explicit relational knowledge can fill gaps left by co-occurrence-based pretraining. A sympathetic reader would care because the recipe is simple and portable: add relation labels, train jointly, keep the architecture.","feed_headline":"Two relation tasks push BERT past 88% on commonsense reading","feed_subtitle":"On SemEval-2018 Task 11 and Story Cloze Test, joint training beats plain BERT by about 0.7 points.","key_machinery":"The load-bearing mechanism is a multi-task objective over a shared BERT encoder. The answer head classifies the final [CLS] representation among the $N$ options with a linear layer and softmax: $L_{AP} = -\\log \\frac{\\exp(v^\\top \\hat{h}_0)}{\\sum_{k=1}^N \\exp(v^\\top h_0^k)}$. Two auxiliary heads add relation-aware supervision over byte-pair-encoding (BPE) tokens from the document or question and each option: a relation-existence head computes $p^{RE}_{ij} = \\mathrm{sigmoid}(h_i^\\top W_1 h_j)$ with binary cross-entropy, and a relation-type head computes $p^{RT}_{ij} = \\mathrm{softmax}(W_3\\,\\mathrm{ReLU}(W_2[h_i;h_j]))$ over 34 ConceptNet relation types. The joint loss is $L = L_{AP} + \\frac{1}{N}\\sum_{\\ell=1}^{N}(\\lambda_1 L^{RE}_\\ell + \\lambda_2 L^{RT}_\\ell)$, so the BERT parameters are updated by all three tasks at once. ConceptNet triplets provide the positive labels, and negative examples are down-sampled to a $1:4$ positive-to-negative ratio.","core_discovery":"On the paper's own terms, the central discovery is that two relation-aware auxiliary tasks—relation-existence (is there any ConceptNet relation between a token in the context and a token in the option?) and relation-type (which of 34 relation types is it?)—improve commonsense multiple-choice reading comprehension when trained jointly with the answer-prediction objective on top of BERT-base. The authors report 88.23% test accuracy on SemEval-2018 Task 11 against 87.53% for BERT-base, and 87.4% on Story Cloze Test against 86.7%; a dev ablation shows the two tasks together add 0.74 points over the basic model, with relation-existence alone contributing 0.51 and relation-type 0.36. They also show the same auxiliary tasks lift a non-BERT co-attention model (TriAN) on SemEval from 81.94% to 82.84%, suggesting the relation supervision is transferable across base architectures.","pith_inferences":["The reported gains are small enough that they could partly come from multi-task regularization rather than from genuine relation reasoning; a controlled test would be to replace ConceptNet relation labels with random labels and see if the accuracy gain persists.","Because the paper reports single runs without variance, the 0.5- to 0.74-point differences may be within seed noise; the method's real value should be checked with multiple seeds and significance tests before relying on it.","The same auxiliary-task recipe could be applied to more recent encoder-decoder or cross-encoder readers, and to knowledge graphs other than ConceptNet, as long as token-to-concept alignment can be defined."],"forward_implications":["A BERT-based multiple-choice reader can be improved on commonsense benchmarks without changing its architecture, only by adding two linear relation heads and ConceptNet-derived labels.","Relation-existence supervision alone accounts for most of the gain (+0.51 on SemEval dev), so collecting or inferring coarse 'is there a relation' labels may be more valuable than fine-grained type labels.","Folding 'No Relation' into the relation-type task as an extra class hurts performance (+0.41 vs +0.74), suggesting the imbalance of negative examples makes the merged task harder; keeping existence and type separate helps.","The auxiliary tasks transfer to at least one non-BERT architecture (TriAN), so the recipe is not tied to BERT's particular design."],"supporting_citations":[{"why":"It supplies the pre-trained BERT encoder that the auxiliary tasks are attached to and the main baseline model.","marker":"[4]"},{"why":"It provides the ConceptNet triplets from which relation-existence and relation-type labels are constructed.","marker":"[11]"},{"why":"It defines the SemEval-2018 Task 11 benchmark, one of the two datasets used to measure the claimed improvement.","marker":"[7]"},{"why":"It defines the Story Cloze Test benchmark, the second dataset used to measure the claimed improvement.","marker":"[6]"},{"why":"It provides the GPT baseline to beat and the train/dev split of Story Cloze Test that the paper follows.","marker":"[8]"},{"why":"It provides the TriAN model on which the relation-aware tasks are also evaluated, showing the method transfers beyond BERT.","marker":"[12]"}],"fun_headline_variants":["Two relation tasks add 0.7 points to BERT on commonsense","Joint relation training lifts BERT to 88.23% on SemEval","Relation-type and existence tasks boost MRC by 0.7","Multi-task relation learning beats plain BERT on commonsense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported gains of about 0.5 to 0.7 accuracy points come from the relation-aware tasks rather than from seed-to-seed noise, because each configuration is reported from a single run without error bars or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["Two relation tasks add 0.7 points to BERT on commonsense","Joint relation training lifts BERT to 88.23% on SemEval","Relation-type and existence tasks boost MRC by 0.7","Multi-task relation learning beats plain BERT on commonsense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001024,"raw_usage":{"total_tokens":4311,"prompt_tokens":931,"completion_tokens":3380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":3302}},"tokens_in":547,"tokens_out":3380,"duration_ms":24448,"temperature":1.0,"reasoning_tokens":3302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:40:24.864786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the basic BERT model and the full relation-aware multi-task model on SemEval-2018 Task 11 dev and Story Cloze Test across ten random seeds with the same hyperparameters; if the average accuracy difference is smaller than the standard deviation across seeds, the central claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the ConceptNet triplets from which relation-existence and relation-type labels are constructed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the SemEval-2018 Task 11 benchmark, one of the two datasets used to measure the claimed improvement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the Story Cloze Test benchmark, the second dataset used to measure the claimed improvement."},{"cited_title":"Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension","cited_arxiv_id":"1803.00191","evidence_quote":"It provides the TriAN model on which the relation-aware tasks are also evaluated, showing the method transfers beyond BERT."}],"review_version":1}