Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning

Chen Wu; Jiangnan Xia; Ming Yan

REVIEW 2 major objections 4 minor 20 references

Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding relation-existence and relation-type auxiliary tasks to BERT-based multiple-choice reading improves commonsense accuracy on SemEval-2018 Task 11 and Story Cloze Test.

desk verdict A clean incremental recipe for adding relation-aware auxiliary tasks to BERT-base MRC; the reported gains are positive but within single-run noise, so the evidence is provisional rather than demonstrative. read the letter →

arxiv 1908.04530 v2 pith:EVZQSWW6 submitted 2019-08-13 cs.CL

Jiangnan Xia , Chen Wu , Ming Yan This is my paper · ORCID

classification cs.CL

keywords machinereadingcomprehensioncommonsensereasoningmulti-tasklearningrelationknowledgeConceptNetBERTSemEval-2018Task11StoryClozeTest

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a multiple-choice machine comprehension model can do better on commonsense questions if, while learning to pick the right answer, it is also trained to predict whether two words from the story and answer option are linked in an external commonsense knowledge base, and what type of relation links them. The two auxiliary tasks turn ConceptNet's triplets into supervised signals that share the BERT encoder, so no new model structure is needed. On SemEval-2018 Task 11 the joint model reaches 88.23% versus 87.53% for BERT-base, and on Story Cloze Test it reaches 87.4% versus 86.7%, which the authors read as evidence that explicit relational knowledge can fill gaps left by co-occurrence-based pretraining. A sympathetic reader would care because the recipe is simple and portable: add relation labels, train jointly, keep the architecture.

What carries the argument

The load-bearing mechanism is a multi-task objective over a shared BERT encoder. The answer head classifies the final [CLS] representation among the $N$ options with a linear layer and softmax: $L_{AP} = -\log \frac{\exp(v^\top \hat{h}_0)}{\sum_{k=1}^N \exp(v^\top h_0^k)}$. Two auxiliary heads add relation-aware supervision over byte-pair-encoding (BPE) tokens from the document or question and each option: a relation-existence head computes $p^{RE}_{ij} = \mathrm{sigmoid}(h_i^\top W_1 h_j)$ with binary cross-entropy, and a relation-type head computes $p^{RT}_{ij} = \mathrm{softmax}(W_3\,\mathrm{ReLU}(W_2[h_i;h_j]))$ over 34 ConceptNet relation types. The joint loss is $L = L_{AP} + \frac{1}{N}\sum_{\ell=1}^{N}(\lambda_1 L^{RE}_\ell + \lambda_2 L^{RT}_\ell)$, so the BERT parameters are updated by all three tasks at once. ConceptNet triplets provide the positive labels, and negative examples are down-sampled to a $1:4$ positive-to-negative ratio.

What would settle it

Run the basic BERT model and the full relation-aware multi-task model on SemEval-2018 Task 11 dev and Story Cloze Test across ten random seeds with the same hyperparameters; if the average accuracy difference is smaller than the standard deviation across seeds, the central claim is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that two relation-aware auxiliary tasks—relation-existence (is there any ConceptNet relation between a token in the context and a token in the option?) and relation-type (which of 34 relation types is it?)—improve commonsense multiple-choice reading comprehension when trained jointly with the answer-prediction objective on top of BERT-base. The authors report 88.23% test accuracy on SemEval-2018 Task 11 against 87.53% for BERT-base, and 87.4% on Story Cloze Test against 86.7%; a dev ablation shows the two tasks together add 0.74 points over the basic model, with relation-existence alone contributing 0.51 and relation-type 0.36. They also show the same auxiliary tasks lift a non-BERT co-attention model (TriAN) on SemEval from 81.94% to 82.84%, suggesting the relation supervision is transferable across base architectures.

Load-bearing premise

The load-bearing premise is that the reported gains of about 0.5 to 0.7 accuracy points come from the relation-aware tasks rather than from seed-to-seed noise, because each configuration is reported from a single run without error bars or significance tests.

Editorial extensions

If this is right

A BERT-based multiple-choice reader can be improved on commonsense benchmarks without changing its architecture, only by adding two linear relation heads and ConceptNet-derived labels.
Relation-existence supervision alone accounts for most of the gain (+0.51 on SemEval dev), so collecting or inferring coarse 'is there a relation' labels may be more valuable than fine-grained type labels.
Folding 'No Relation' into the relation-type task as an extra class hurts performance (+0.41 vs +0.74), suggesting the imbalance of negative examples makes the merged task harder; keeping existence and type separate helps.
The auxiliary tasks transfer to at least one non-BERT architecture (TriAN), so the recipe is not tied to BERT's particular design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported gains are small enough that they could partly come from multi-task regularization rather than from genuine relation reasoning; a controlled test would be to replace ConceptNet relation labels with random labels and see if the accuracy gain persists.
Because the paper reports single runs without variance, the 0.5- to 0.74-point differences may be within seed noise; the method's real value should be checked with multiple seeds and significance tests before relying on it.
The same auxiliary-task recipe could be applied to more recent encoder-decoder or cross-encoder readers, and to knowledge graphs other than ConceptNet, as long as token-to-concept alignment can be defined.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a clean, incremental extension of known relation-aware auxiliary tasks to a BERT-base multi-choice MRC model. The gains it reports are consistently positive on two benchmarks, but the reported 0.7-point gaps are within single-run noise as far as the paper demonstrates, so the empirical claim is not yet load-bearing.

What's genuinely new: the specific combination of relation-existence and relation-type prediction as two separate auxiliary tasks jointly trained with BERT for commonsense MRC. It is a small twist on earlier work—TriAN and commonsense-based pre-training [17] already used relation-aware tasks—but the two-task decomposition is neat, and the dev ablation shows that both tasks add a bit and that LRE helps more than LRT. Equations are standard and consistent. Labels come from ConceptNet, independent of the MRC labels, so no circularity. I agree with the reader: no fundamental flaw in the method.

Soft spots: the central claim of "superior performance" rests on single-run accuracies. The SemEval test gap (87.53 to 88.23) is about 20 extra correct examples out of 2,797; the Story Cloze gap (86.7 to 87.4) is about 13 out of 1,871. The standard errors on those differences are roughly 0.9 and 1.1 percentage points, so both gaps are less than one standard error. The dev ablation gap is of similar size. Without multiple seeds, error bars, or significance tests, the improvement could easily be optimizer or initialization noise. Also, Section 3.3 is underspecified: how ConceptNet concepts are matched to BPE tokens and how negative pairs are sampled is not stated, which limits reproducibility. No code was released. These are genuine weaknesses, but proportionate: for a 4-page CIKM short paper, single-run reporting is common, and the weaknesses are mostly about evidence quality rather than the method's logic. The method is plausible and the ablation points in the right direction.

Who this is for: people working on knowledge-enhanced MRC or multi-task learning for NLP. They get a small, plausible recipe plus a useful reminder about statistical significance in benchmark differences. I would not cite it as a proven result in my own work, but I would expect it mentioned in a survey of knowledge-aware reading comprehension.

Recommendation: send to peer review. A good reviewer can push for multiple seeds with variance reporting and for a precise description of the relation-pair construction. A serious referee will help the authors make the claim honest.

Referee Report

2 major / 4 minor

Summary. The paper proposes a multi-task learning extension of BERT-base for multi-choice commonsense reading comprehension. In addition to the standard answer-selection loss (Eq. (1)), the model is trained with two auxiliary objectives derived from ConceptNet: a relation-existence binary task over token pairs from the document/question (sentence A) and option (sentence B) (Eqs. (2)-(3)), and a relation-type classification task over R=34 relation types (Eqs. (4)-(5)), combined in Eq. (6). Experiments on SemEval-2018 Task 11 and Story Cloze Test report accuracy improvements over BERT(base): 88.23 vs. 87.53 on SemEval and 87.4 vs. 86.7 on Story Cloze, with a dev ablation showing +0.74 from adding both auxiliary tasks. The authors conclude that relation-aware multi-task learning improves commonsense MRC.

Significance. The proposed formulation is clean and model-agnostic; applying the same auxiliary tasks to TriAN and obtaining a gain suggests the idea may transfer. The use of ConceptNet-derived labels avoids the circularity that would arise if the auxiliary labels were functions of the MRC answer labels. If the reported gains were robust, the paper would provide a lightweight way to inject relational commonsense knowledge into pre-trained LMs without architectural changes. The main limitation is that the empirical evidence is statistically unverified: single-run accuracy differences of 0.7 points on test sets of about 2,000-3,000 examples are within one standard error. With proper multi-seed evaluation and significance testing, the paper could support its claim.

major comments (2)

[§4.3, Tables 2-4] The central empirical claim is not supported by the reported statistics. All accuracies in Tables 2 and 3 are single-run numbers, and the reported gains are within one standard error of zero under a two-proportion test: on SemEval-2018 Task 11 (n=2,797), the 87.53 vs. 88.23 difference corresponds to about 20 examples with SE ≈ 0.87 percentage points; on Story Cloze Test (n=1,871), the 86.7 vs. 87.4 difference is about 13 examples with SE ≈ 1.1 percentage points; on the SemEval dev set (n=1,411), the 87.51 vs. 88.25 difference in Table 4 is about 10 examples with SE ≈ 1.2 percentage points. No confidence intervals, multiple seeds, or significance tests are reported, and no code is released to assess run-to-run variability. Under the null hypothesis of no improvement, a positive gap of this size is entirely plausible, so the conclusion of 'superior performance' in the abstract is not established. The authors should report mean and standard deviation over at least several fine-tuning seeds and a paired significance test (e.g., McNemar's test) for the test-set comparisons.
[§3.3, Eqs. (2)-(5)] The construction of the auxiliary relation labels is underspecified and not reproducible as written. The text says that 'the concept' is converted to a set of BPE tokens with beginning indices i and j, but it does not state which spans of the input sequence are treated as concepts, how ConceptNet concepts are matched to BPE tokens (e.g., exact match, stemmed match, or phrase matching), or how negative pairs are sampled under the 1:γ ratio. It is also unclear how the ground-truth relation index k in Eq. (5) is chosen when multiple ConceptNet relations hold for the same token pair. These details are needed to reimplement the method and to assess whether the auxiliary supervision is as the authors intend.

minor comments (4)

[§3.2, paragraph after Eq. (1)] In the sentence 'Then, we take one step further to decide what is the right type of the relation in relation-existence,' the final phrase should read 'in the relation-type task.'
[§4.3, Table 5] The third example in Table 5 is explained incorrectly: the text says 'the relation type (kettle, UsedFor, boil water) in option (A),' but option (A) is 'To drink from' and this fact supports option (B), the correct answer. Please correct the option reference.
[§3.3] The citation 'Following [4]' for converting concepts to BPE tokens is misleading because [4] is the BERT paper; please cite the actual source of the concept-to-BPE conversion procedure or describe it directly.
[§4.2, Eq. (6)] The hyperparameters λ1, λ2, and γ are fixed at 0.5, 0.5, and 4.0 with no sensitivity analysis; given the small reported gains, a sentence stating that the results are stable under reasonable variation would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MRC targets and ConceptNet-derived auxiliary labels are independent, and the claimed gains are assessed against external benchmarks.

full rationale

The paper's derivation chain is self-contained with respect to circularity. The target MRC loss (Eq. 1) is supervised by the benchmark labels of SemEval-2018 Task 11 and Story Cloze Test, and the auxiliary relation-existence and relation-type losses (Eqs. 2-5) are supervised by ConceptNet triplets, which are external to the target labels. Neither auxiliary objective is a function of the model's own answer predictions, and no fitted parameter is renamed as a prediction: the dev ablation (Table 4) reports the increment from adding LRE and LRT, with the relation labels coming from ConceptNet, not from the MRC labels or predictions. The only self-citation in the bibliography ([13]) is a generic prior MRC architecture reference and is not load-bearing for the proposed method. The observation that the reported 0.7-point gains lack error bars, multiple seeds, or significance tests is a statistical robustness concern, not a circularity concern, because the compared values are produced by independent benchmark evaluation rather than by construction. Therefore, no circular step is identifiable.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new entities are introduced. The method's burden is mainly domain assumptions about ConceptNet matching and the transferability of the auxiliary tasks, plus three hand-set hyperparameters (lambda_1, lambda_2, gamma) and a hand-chosen relation-type set (R=34).

free parameters (4)

lambda_1 (relation-existence task weight) = 0.5
Weight on L_RE in Eq. (6); set by hand in Section 4.2 with no sensitivity analysis.
lambda_2 (relation-type task weight) = 0.5
Weight on L_RT in Eq. (6); set by hand in Section 4.2 with no sensitivity analysis.
gamma (negative sampling ratio) = 4.0
Ratio of negatives to positives in relation-existence task, set in Section 4.2 with no analysis.
R (number of relation types) = 34
Selected ConceptNet relation types excluding RelatedTo, ExternalURL, dbpedia (Section 3.3, footnote 2); no justification for the exclusion.

assumptions (4)

domain assumption ConceptNet triplets, matched to BPE token pairs, provide a useful supervision signal for commonsense MRC.
The whole method rests on these auxiliary tasks transferring to answer prediction; this is tested empirically but not independently motivated.
domain assumption The binary 'relation exists' label for a token pair is a meaningful target for documents and options.
Section 3.3 derives positive/negative pairs from ConceptNet, assuming surface matching between concepts and BPE tokens is reliable.
domain assumption Excluding relation types RelatedTo, ExternalURL, and dbpedia from the type-classification task does not discard information needed for answering.
Section 3.3, footnote 2; no experiments test this exclusion, and Table 5 itself shows RelatedTo relations as useful evidence.
standard math Standard backpropagation training of BERT with the joint loss in Eq. (6) is sufficient.
The paper relies on BERT [4] and standard stochastic gradient training; no formal guarantee is given, which is normal for this class of models.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning." pith.science (2026). https://pith.science/paper/EVZQSWW6

@misc{pith2026190804530,
  author       = {Pith},
  title        = {Pith review of: Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVZQSWW6}},
  note         = {Machine review of arXiv:1908.04530}
}

read the original abstract

This paper focuses on how to take advantage of external relational knowledge to improve machine reading comprehension (MRC) with multi-task learning. Most of the traditional methods in MRC assume that the knowledge used to get the correct answer generally exists in the given documents. However, in real-world task, part of knowledge may not be mentioned and machines should be equipped with the ability to leverage external knowledge. In this paper, we integrate relational knowledge into MRC model for commonsense reasoning. Specifically, based on a pre-trained language model (LM). We design two auxiliary relation-aware tasks to predict if there exists any commonsense relation and what is the relation type between two words, in order to better model the interactions between document and candidate answer option. We conduct experiments on two multi-choice benchmark datasets: the SemEval-2018 Task 11 and the Cloze Story Test. The experimental results demonstrate the effectiveness of the proposed method, which achieves superior performance compared with the comparable baselines on both datasets.

Figures

Figures reproduced from arXiv: 1908.04530 by the authors.

**Figure 1.** MRC model with two relation-aware tasks with special delimiters as one segment, which is then fed into BERT encoder. The input sequence is packed as “[CLS]D(Q)[SEP]O[SEP]” 1 , where [CLS] and [SEP] are the special delimiters. After BERT encoder, we obtain the contextualized word representation h L i ∈ R H for the i-th input token from the final layer of BERT. H is the dimension of hidden state. Next, on top of BERT… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 14 canonical work pages

[17]

Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin

work page
[1]

Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for Generative Multi-Hop Question Answering Tasks. arXiv preprint arXiv:1809.06309 (2018)

arXiv 2018
[2]

Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . 1603–1614

work page 2017
[3]

Zhipeng Chen, Yiming Cui, Wentao Ma, Shijin Wang, Ting Liu, and Guoping Hu

work page
[4]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018
[5]

Elizabeth Merkhofer, John Henderson, David Bloom, Laura Strickhart, and Guido Zarrella. 2018. MITRE at SemEval-2018 Task 11: Commonsense Reasoning with- out Commonsense Knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation. 1078–1082

work page 2018
[6]

Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics. 46–51

work page 2017
[7]

Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 Task 11: Machine Comprehension using Common- sense Knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation. 747–757

work page 2018

Show all 20 references

[8]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Im- proving language understanding by generative pre-training. (2018)

2018
[9]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)

2016 arXiv
[10]

Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A Smith. 2017. Story cloze task: Uw nlp system. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics . 52–55

2017
[11]

Robert Speer, Joshua Chin, and Catherine Havasi. 2016. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. (2016)

2016
[12]

Liang Wang. 2018. Yuanfudao at SemEval-2018 Task 11: Three-way Attention and Relational Knowledge for Commonsense Machine Comprehension. arXiv preprint arXiv:1803.00191 (2018)

2018 arXiv
[13]

Wei Wang, Ming Yan, and Chen Wu. 2018. Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 1705–1714

2018
[14]

Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 1. 189–198

2017
[15]

Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in lstms for improving machine reading. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics , Vol. 1. 1436–1446

2017
[16]

Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mo- hammad Norouzi, and Quoc V Le. 2018. QANet: Combining Local Convolu- tion with Global Self-Attention for Reading Comprehension. arXiv preprint arXiv:1804.09541 (2018)

2018 arXiv
[18]

Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense Knowledge Aware Conversation Generation with Graph Attention.. In IJCAI. 4623–4629

2018
[19]

arXiv preprint arXiv:1809.03568 (2018)

Improving question answering by commonsense-based pre-training. arXiv preprint arXiv:1809.03568 (2018)

2018 arXiv
[2018]

arXiv preprint arXiv:1803.05655 (2018)

HFL-RC System at SemEval-2018 Task 11: Hybrid Multi-Aspects Model for Commonsense Reading Comprehension. arXiv preprint arXiv:1803.05655 (2018)

2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [17]

Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin

work page

[2] [1]

Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for Generative Multi-Hop Question Answering Tasks. arXiv preprint arXiv:1809.06309 (2018)

arXiv 2018

[3] [2]

Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . 1603–1614

work page 2017

[4] [3]

Zhipeng Chen, Yiming Cui, Wentao Ma, Shijin Wang, Ting Liu, and Guoping Hu

work page

[5] [4]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018

[6] [5]

Elizabeth Merkhofer, John Henderson, David Bloom, Laura Strickhart, and Guido Zarrella. 2018. MITRE at SemEval-2018 Task 11: Commonsense Reasoning with- out Commonsense Knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation. 1078–1082

work page 2018

[7] [6]

Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics. 46–51

work page 2017

[8] [7]

Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 Task 11: Machine Comprehension using Common- sense Knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation. 747–757

work page 2018