REVIEW 3 major objections 6 minor 1 cited by
SG-Net: Syntax-Guided Machine Reading Comprehension
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that adding a second self-attention layer whose mask restricts each word to its syntactic ancestors in a dependency tree improves machine reading comprehension beyond the BERT baseline, and demonstrates this on SQuAD 2.0…
desk verdict The syntax-masked attention idea is sound and the gains look real, but equation (2) as written does not implement a hard mask, so the paper needs a correction before it is reproducible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the syntactic dependency of interest (SDOI) mask, a binary matrix that encodes each word's ancestor chain from a dependency parse: $M[i,j]=1$ if token $j$ is an ancestor of token $i$ or is $i$ itself, otherwise $0$. Inserted into multi-head self-attention as $A_i' = \operatorname{Softmax}(M \cdot (Q_i' K_i'^T)/\sqrt{d_k})$, it forces a token to collect context only from syntactically governing words, pruning the rest of the passage. The second mechanism is dual context aggregation, $\bar{h}_i = \alpha h_i + (1-\alpha) h'_i$, which keeps the unpruned BERT representation and the pruned syntax-guided representation in parallel; the ablation shows the combination outperforms either branch alone as well as concatenation or bi-attention fusion.
What would settle it
Train or fine-tune SG-Net on SQuAD 2.0 and RACE with binary SDOI masks built from randomly rewired dependency trees that keep each token's allowed attention count fixed; if the random-mask version matches the syntax-guided scores, the measured gains come from mask density rather than from linguistic structure.
Extended reading notes
Core claim
The core claim is that a hard syntactic mask on self-attention upgrades a pretrained Transformer encoder for reading comprehension. For each sentence, a dependency parser produces a tree; the syntactic dependency of interest (SDOI) of a word is the set of its ancestors plus itself, written as a binary matrix $M$ with $M[i,j]=1$ exactly when $j$ is an ancestor of $i$ or $j=i$. The syntax-guided self-attention layer applies this mask before softmax, so each token gathers information only from its ancestor chain. The syntax-enhanced representation is then the weighted average $\bar{h}_i = \alpha h_i + (1-\alpha) h'_i$ of the original BERT representation and the syntax-guided representation, with $\alpha=0.5$. This dual design outperforms an extra vanilla attention layer, syntax-guided attention alone, concatenation, and bi-attention fusion, and it produces gains in both span extraction on SQuAD 2.0 and multiple-choice on RACE.
Load-bearing premise
The method assumes the automatically computed dependency parse for every passage and question is accurate enough that forcing each word to attend only to its syntactic ancestors removes only noise and never the words that contain the answer; a parse error on the answer span could block the model from ever seeing the evidence.
Editorial extensions
If this is right
- On SQuAD 2.0, SG-Net lifts exact match from 84.1 to 85.1 and F1 from 86.8 to 87.9 over the BERT baseline.
- On RACE, SG-Net lifts accuracy from 72.6 to 74.2, with the larger gain on the high-school subset (70.4 to 72.2).
- The extra gain is not from added parameters: a control with an extra vanilla attention layer gives only 84.2 EM, while syntax-guided attention alone gives 84.4 and the dual combination gives 85.1.
- SG-Net keeps its accuracy on long questions, where the BERT baseline's exact match drops as question length grows.
- The syntax-guided layer sits on top of a generic self-attention encoder, so the same SDOI masking can be attached to any Transformer-based encoder, not only BERT.
Reading between the lines
- One implication the paper leaves implicit: a soft version of the SDOI mask, weighted by parser confidence, would let the model down-weight uncertain dependencies instead of discarding those tokens entirely; the paper only tests the hard binary mask.
- A direct stress test is to keep the mask density fixed but rewire the dependency edges randomly; if a random-mask control matches SG-Net's scores, the gain would come from pruning width rather than from genuine syntactic structure.
- The dual aggregation weight is fixed at $\alpha=0.5$; a learned or question-length-dependent $\alpha$ could show whether the optimal balance between syntax-pruned and unpruned contexts shifts, which the paper does not explore.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. SG-Net augments a Transformer encoder (BERT) with an additional self-attention layer in which each token attends only to itself and its ancestors in a dependency parse tree, via a binary SDOI mask. The syntax-guided representation is then interpolated with the original encoder output through a dual-context aggregation, and the resulting representation is passed to task-specific heads for SQuAD 2.0 and RACE. The paper reports EM/F1 improvements on SQuAD 2.0 (85.1/87.9 vs. a baseline of 84.1/86.8) and accuracy improvement on RACE (74.2 vs. 72.6), together with an ablation showing that adding an extra vanilla attention layer does not reproduce the gain.
Significance. If the reported effects are robust, the paper offers a simple and portable way to inject syntactic tree structure into Transformer-based encoders, with evidence from two different MRC settings. The use of a fixed external parser (trained on Penn Treebank, not on MRC labels) and evaluation on held-out test sets are strengths, as is the explicit ablation against an equal-size vanilla attention layer. The main weaknesses are the ambiguous mask formula in Eq. (2), the missing wordpiece-to-dependency-node alignment details, and the insufficient statistical reporting behind the significance claims.
major comments (3)
- [Section 3.1, Eq. (2)] Equation (2) cannot implement the hard SDOI mask claimed in the text. Since M in Eq. (1) is a binary 0/1 matrix, elementwise multiplication into the pre-softmax logits leaves every disallowed position with logit 0, contributing exp(0)=1 to the softmax numerator, while ordinary matrix multiplication would produce a linear recombination of row scores rather than a mask. Please correct the formula to add -inf (or a large negative constant) to masked logits before softmax, or else zero the post-softmax attention weights, and check the released code to state which operation actually ran, because the reported gains are attributed to hard syntactic pruning.
- [Section 3.1 and Section 4.2] The paper does not specify how word-level dependency tree nodes are aligned to the BERT wordpiece tokens that form the actual input sequence S. The text says all texts are tokenized using wordpieces, but the SDOI mask is defined over word tokens; no mapping is provided for subword pieces, multiword expressions, or regular tokens (the footnote addresses only [CLS], [SEP], and [PAD]). This makes Eq. (1) ill-defined for the real input and is essential for reproducibility; please specify the alignment procedure explicitly.
- [Section 4.2 and Tables 1 and 2] The claims that SG-Net is 'significantly better than the baseline BERT with p-value < 0.01' are unsupported as reported. State the test used, the number of independent runs or seeds, how variance is estimated, and report error bars or confidence intervals. Also, the two main hyperparameters, alpha=0.5 in Eq. (4) and the SQuAD 2.0 no-answer threshold delta, are set without any sensitivity analysis, so it is unclear whether the reported gains depend on these choices.
minor comments (6)
- [Abstract] The abstract contains a typo: 'getting ride of the noises' should be 'getting rid of the noise'.
- [Table 2, footnote 7] The RACE baseline uses a different concatenation order and is therefore 'improved'; please give the exact input format so readers can place the 72.6 baseline relative to the public BERT result.
- [Figure 5 caption] The caption says weights are 'normalized by SoftMax for each row'; with a hard mask, normalization should be over allowed positions only, and the caption should say this explicitly.
- [Section 5.3, footnote 9] The statement that special-token weights 'will be masked in the following aggregation layer' is not described in Section 3.1; if such masking exists, include it in the model definition and ablation.
- [Table 3] The Concatenation and Bi-attention variants are listed without specifying how the representations are merged and projected to match the task-specific heads; please add these architectural details.
- [Section 5.1] The claim that SG-Net shows 'positive correlation' between accuracy and question length is presented without any correlation statistic or confidence interval; please support or soften this observation.
Circularity Check
No circularity: the syntax masks come from an externally trained PTB dependency parser, and the reported gains are measured on held-out dev/test sets; the Eq. (2) masking issue is a reproducibility concern, not an input-output equivalence.
full rationale
The paper's derivation chain is self-contained against external benchmarks. The SDOI masks are produced by a dependency parser pretrained on the Penn Treebank (Section 4.2), with reported UAS/LAS numbers on the PTB test set; the parser is not updated with the MRC models and the masks are not derived from the answer labels or from the BERT representations being evaluated. The SG-Net output is a fixed-weight convex combination (Eq. 4, alpha = 0.5) of the original BERT hidden states and the syntax-guided attention output, and final predictions go through task-specific linear/softmax layers trained with cross-entropy. The reported improvements on SQuAD 2.0 and RACE are measured on the official dev/test sets, so no quantity central to the claim is defined in terms of the target result. The only near self-citation is the dependency parser of Zhou and Zhao (2019), whose authors overlap with this paper; however, that parser's accuracy is an external PTB benchmark result and is not equivalent to SG-Net's MRC outputs, so under the stated rules it does not constitute circularity. The skeptical observation about Eq. (2) is a substantive correctness/reproducibility issue: multiplying a binary 0/1 mask into pre-softmax logits is not the same as a hard mask excluding disallowed positions, because exp(0) mass remains. But that is a mismatch between formula and intended operation, not a case where the claimed prediction reduces by construction to its own input; therefore it does not raise the circularity score.
Assumptions & free parameters
free parameters (2)
- alpha (dual context aggregation weight) =
0.5
- delta (SQuAD 2.0 no-answer threshold) =
heuristic, computed on development set
assumptions (3)
- domain assumption The dependency parser from Zhou and Zhao (2019) provides sufficiently accurate parse trees for SQuAD and RACE inputs.
- domain assumption The SDOI ancestor set is the correct contextual subset for reading comprehension, and all non-ancestor tokens are safely pruned.
- domain assumption Word-level dependency parse nodes can be mapped to BERT wordpiece tokens.
Cite this review
Pith. "Pith review of SG-Net: Syntax-Guided Machine Reading Comprehension." pith.science (2026). https://pith.science/paper/ZLGWIDLM
@misc{pith2026190805147,
author = {Pith},
title = {Pith review of: SG-Net: Syntax-Guided Machine Reading Comprehension},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZLGWIDLM}},
note = {Machine review of arXiv:1908.05147}
}
read the original abstract
For machine reading comprehension, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy passages and getting ride of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanism for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. To verify its effectiveness, the proposed SG-Net is applied to typical pre-trained language model BERT which is right based on a Transformer encoder. Extensive experiments on popular benchmarks including SQuAD 2.0 and RACE show that the proposed SG-Net design helps achieve substantial performance improvement over strong baselines.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
Semantics-aware BERT for Language Understanding
Feeding semantic role labels into BERT alongside the text improves performance on ten NLU benchmarks over the BERT baseline.
Reference graph
Works this paper leans on
-
[1]
[Bahdanau, Cho, and Bengio 2015] Bahdanau, D.; Cho, K.; and Bengio, Y
work page 2015
-
[7]
arXiv preprint arXiv:1810.04805
BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [Dhingra et al. 2017] Dhingra, B.; Liu, H.; Yang, Z.; Cohen, W. W.; and Salakhutdinov, R
arXiv 2017
-
[8]
Gated-attention readers for text comprehension. ACL. [Duan et al. 2019] Duan, S.; Zhao, H.; Zhou, J.; and Wang, R
work page 2019
-
[9]
Syntax-aware transformer encoder for neural machine translation. In IALP. [Hendrycks and Gimpel 2016] Hendrycks, D., and Gimpel, K
work page 2016
-
[10]
Bridging nonlinearities and stochastic regularizers with gaussian error linear units. ICLR. [Hermann et al. 2015] Hermann, K. M.; Kocisky, T.; Grefen- stette, E.; Espeholt, L.; Kay, W.; Suleyman, M.; and Blunsom, P
work page 2015
-
[11]
2015] Hill, F.; Bordes, A.; Chopra, S.; and Weston, J
[Hill et al. 2015] Hill, F.; Bordes, A.; Chopra, S.; and Weston, J
work page 2015
-
[12]
arXiv preprint arXiv:1511.02301
The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. [Joshi et al. 2017] Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L
arXiv 2017
-
[13]
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. ACL. [Kadlec et al. 2016] Kadlec, R.; Schmid, M.; Bajgar, O.; and Kleindienst, J
work page 2016
Show all 39 references
-
[14]
Text understanding with the attention sum reader network. ACL. [Kasai et al. 2019] Kasai, J.; Friedman, D.; Frank, R.; Radev, D.; and Rambow, O
2019
-
[15]
In NAACL
Syntax-aware neural semantic role labeling with supertags. In NAACL. [Kitaev and Klein 2018] Kitaev, N., and Klein, D
2018
-
[16]
Constituency Parsing with a Self-Attentive Encoder. InACL. [Lai et al. 2017] Lai, G.; Xie, Q.; Liu, H.; Yang, Y .; and Hovy, E
2017
-
[17]
In EMNLP
Race: Large-scale reading comprehension dataset from examinations. In EMNLP. [Li et al. 2018] Li, Z.; Cai, J.; He, S.; and Zhao, H
2018
-
[18]
In COLING
Seq2seq dependency parsing. In COLING. [Li, Zhao, and Parnow 2020] Li, Z.; Zhao, H.; and Parnow, K
2020
-
[20]
Stack-Pointer Networks for Dependency Parsing. In ACL. [Marcus, Santorini, and Marcinkiewicz 1993] Marcus, M. P.; Santorini, B.; and Marcinkiewicz, M. A
1993
-
[22]
[Nguyen et al
Did the model understand the question? ACL. [Nguyen et al. 2016] Nguyen, T.; Rosenberg, M.; Song, X.; Gao, J.; Tiwary, S.; Majumder, R.; and Deng, L
2016
-
[23]
ArXiv:1611.09268v2
Ms marco: A human generated machine reading comprehension dataset. ArXiv:1611.09268v2. [Radford et al. 2018] Radford, A.; Narasimhan, K.; Sali- mans, T.; and Sutskever, I
2018 arXiv
-
[24]
Technical report
Improving language understanding by generative pre-training. Technical report. [Rajpurkar et al. 2016] Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P
2016
-
[25]
SQuAD: 100,000+ questions for machine comprehension of text. EMNLP. [Rajpurkar, Jia, and Liang 2018] Rajpurkar, P.; Jia, R.; and Liang, P
2018
-
[26]
Know what you don’t know: Unanswerable questions for SQuAD. ACL. [Ran et al. 2019] Ran, Q.; Li, P.; Hu, W.; and Zhou, J
2019
-
[27]
arXiv preprint arXiv:1903.03033
Option comparison network for multiple-choice reading comprehension. arXiv preprint arXiv:1903.03033. [Seo et al. 2016] Seo, M.; Kembhavi, A.; Farhadi, A.; and Hajishirzi, H
1903 arXiv
-
[28]
arXiv preprint arXiv:1611.01603
Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. [Strubell et al. 2018] Strubell, E.; Verga, P.; Andor, D.; Weiss, D.; and McCallum, A
2018 arXiv
-
[29]
In EMNLP
Linguistically-informed self-attention for semantic role labeling. In EMNLP. [Sun et al. 2018] Sun, K.; Yu, D.; Yu, D.; and Cardie, C
2018
-
[30]
arXiv preprint arXiv:1810.13441
Improving machine reading comprehension with gen- eral reading strategies. arXiv preprint arXiv:1810.13441. [Vaswani et al. 2017] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I
2017 arXiv
-
[31]
Attention is all you need. In NIPS. [Wang et al. 2017] Wang, W.; Yang, N.; Wei, F.; Chang, B.; and Zhou, M
2017
-
[32]
Gated self-matching networks for reading comprehension and question answering. ACL. [Wang, Zhang, and Zong 2017] Wang, S.; Zhang, J.; and Zong, C
2017
-
[33]
In IJCAI
Learning sentence representation with guidance of human attention. In IJCAI. [Yu, Lee, and Le 2017] Yu, A. W.; Lee, H.; and Le, Q
2017
-
[34]
Learning to skim text. In ACL. [Zhang et al. 2018] Zhang, Z.; Li, J.; Zhu, P.; Zhao, H.; and Liu, G
2018
-
[35]
In COLING
Modeling multi-turn conversation with deep utterance aggregation. In COLING. arXiv preprint arXiv:1806.09102. [Zhang et al. 2019] Zhang, Z.; Wang, R.; Chen, K.; Utiyama, M.; Sumita, E.; and Zhao, H
2019 arXiv
-
[36]
arXiv preprint arXiv:1911.02971
Probing contextualized sentence representations with visual awareness. arXiv preprint arXiv:1911.02971. [Zhang et al. 2020a] Zhang, S.; Zhao, H.; Wu, Y .; Zhang, Z.; Zhou, X.; and Zhou, X. 2020a. Dual co-matching network for multi-choice reading comprehension. In AAAI. arXiv p...
1911 arXiv
-
[37]
Probabilistic graph-based dependency parsing with convolutional neural network. In ACL. [Zhou and Zhao 2019] Zhou, J., and Zhao, H
2019
-
[38]
Head- driven phrase structure grammar parsing on penn treebank. In ACL. [Zhou, Zhang, and Zhao 2019] Zhou, J.; Zhang, Z.; and Zhao, H
2019
-
[39]
arXiv preprint arXiv:1910.14296
LIMIT-BERT: Linguistic informed multi- task bert. arXiv preprint arXiv:1910.14296
1910 arXiv
-
[1993]
Computational Linguistics 19(2)
Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19(2). [Mudrakarta et al. 2018] Mudrakarta, P. K.; Taly, A.; Sun- dararajan, M.; and Dhamdhere, K
2018
-
[2015]
Neural machine translation by jointly learning to align and translate. In ICLR. [Bowman et al. 2016] Bowman, S. R.; Gauthier, J.; Rastogi, A.; Gupta, R.; Manning, C. D.; and Potts, C
2016
-
[2016]
[Chen et al
A fast unified model for parsing and sentence understanding.arXiv preprint arXiv:1603.06021. [Chen et al. 2017a] Chen, K.; Wang, R.; Utiyama, M.; Liu, L.; Tamura, A.; Sumita, E.; and Zhao, T. 2017a. Neural machine translation with source dependency representation. In EMNLP. [Ch...
2018 arXiv
-
[2017]
Attention-over-attention neural networks for reading comprehension. ACL. [Devlin et al. 2018] Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K
2018
-
[2018]
Syntax-directed attention for neural machine translation. In AAAI. [Clark et al. 2019] Clark, K.; Khandelwal, U.; Levy, O.; and Manning, C. D
2019
-
[2019]
arXiv preprint arXiv:1906.04341
What does BERT look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. [Cui et al. 2017] Cui, Y .; Chen, Z.; Wei, S.; Wang, S.; Liu, T.; and Hu, G
1906 arXiv
-
[2020]
Global greedy dependency parsing. In AAAI. [Ma et al. 2018] Ma, X.; Hu, Z.; Liu, J.; Peng, N.; Neubig, G.; and Hovy, E
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.