Source Dependency-Aware Transformer with Supervised Self-Attention

Chengyi Wang; Shuangzhi Wu; Shujie Liu

REVIEW 3 major objections 4 minor 16 references

Source Dependency-Aware Transformer with Supervised Self-Attention

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Supervising two encoder attention heads with source dependency matrices improves Transformer translation quality across Chinese-English, English-Japanese, and English-German tasks, with no parser needed at inference.

desk verdict A simple, plausible way to make two Transformer heads follow dependency trees—consistent BLEU gains, but no control or significance testing, so the syntax-specific claim is not yet proven. read the letter →

arxiv 1909.02273 v1 pith:XDO7UZ45 submitted 2019-09-05 cs.CL

Chengyi Wang , Shuangzhi Wu , Shujie Liu This is my paper · ORCID

classification cs.CL

keywords neuralmachinetranslationTransformersourcedependencytreesyntax-awaresupervisedself-attentionattentionheadscross-entropyregularization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a Transformer encoder can be made explicitly syntax-aware by supervising two of its self-attention heads with dependency-tree information, and that this improves translation quality across four language-pair tasks. The authors define a child-attention matrix and a parent-attention matrix from a source dependency tree, add two cross-entropy losses that pull two selected attention heads toward those matrices, and keep all other heads unsupervised. If the claim holds, a standard Transformer can exploit source syntax without any extra network modules and without requiring parsed input at inference time, making syntax-aware translation nearly free in terms of architecture and decoding cost.

What carries the argument

The load-bearing mechanism is a pair of attentional adjacency matrices derived from the source dependency tree: $W^c$ (child matrix) sets $W^c_{ij} = 1/n_i$ when word $j$ is a child of word $i$, and $W^p$ (parent matrix) sets $W^p_{ij} = 1$ when word $j$ is the parent of word $i$, with self-alignment for leaf and root nodes. Two self-attention heads in the top encoder layer, the child supervised head (CSH) and parent supervised head (PSH), are trained with additional cross-entropy losses $L_c$ and $L_p$ that minimize the divergence between the heads' softmax attention distributions and these matrices. These two losses are added to the standard translation likelihood with weights $\alpha = 0.4$ and $\beta = 0.4$, so the heads are pushed to act as dependency selectors while the remaining heads keep learning unconstrained patterns.

What would settle it

Take the same model and replace the two supervised heads' target matrices $W^c$ and $W^p$ with random matrices of the same shape, keeping all other settings identical; if BLEU on the NIST Chinese-to-English test sets does not drop back toward the Transformer baseline, then the dependency structure itself is not what causes the improvement. Alternatively, train with gold-standard dependency trees from the treebanks instead of pseudo parses and check whether BLEU rises further or falls, which would isolate parser noise as a limiting factor.

Watch

Extended reading notes

Core claim

The central discovery is that encoding child and parent dependency relations into two attention heads via auxiliary cross-entropy losses improves translations on NIST Chinese-to-English, WAT English-to-Japanese, WMT2017 bidirectional Chinese-English, and WMT2014 English-to-German tasks. On the NIST Chinese-to-English sets, the combined child-plus-parent supervision raises average BLEU from 37.10 to 38.63, and the same model outperforms RNN-based syntax-aware systems as well as the plain Transformer. The supervised heads learn interpretable alignment patterns; on a held-out subset the predicted source trees match the in-house parser at a UAS of 83.25%, close to the parser's own 83.7% on Chinese.

Load-bearing premise

The entire training signal for syntax rests on pseudo-golden dependency parses produced by an in-house parser (92.3% UAS for English, 83.7% for Chinese); if those parses are too noisy, the auxiliary losses inject a wrong inductive bias and the claimed gains could reverse.

Editorial extensions

If this is right

Child-only and parent-only supervision each improve BLEU by about 1.0 point on the NIST Chinese-to-English test sets, and combining them adds a further ~0.5 point.
No parse tree is required during decoding: the supervised heads predict dependency relations from the source sentence alone, preserving Transformer decoding efficiency.
The method works on large-scale WMT tasks with BPE subword vocabularies, gaining 0.86 (Zh-En), 1.05 word-level (En-Zh), and 0.6 (En-De) BLEU points.
The supervised heads can construct source dependency trees: predicted trees have 83.25% UAS against the parser's outputs on a curated Chinese subset, close to the parser's own reported accuracy.
Lower layers are less effective for supervision; the authors find the best results when the supervised heads are placed in the top encoder layer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to apply the same supervision at several encoder layers or to label the dependencies (e.g., subject, object) rather than only unlabeled child/parent links; the paper does not test these variants.
Because the supervised heads produce explicit parent predictions, the same training objective could be evaluated as an end-to-end dependency parser inside the encoder, comparing directly against gold treebanks.
The approach could transfer to other sequence-to-sequence tasks where structural relations are available, such as abstractive summarization with discourse trees or code generation with abstract syntax trees.
The near-parity between predicted tree UAS and the parser's own UAS hints that the model may be learning to imitate parser noise; testing on gold trees would separate genuine syntax learning from parser replication.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a clean idea and a decent paper, but the headline claim outruns the experiments. The method builds child/parent adjacency matrices from a dependency parse, supervises two encoder self-attention heads with cross-entropy, and keeps the rest of the Transformer unchanged. At inference you don't need a parser. That is genuinely new for Transformer NMT, as far as I know, and the presentation is refreshingly simple. The CSH/PSH split is a nice ablation: either alone helps, both together help more.

The positives first. The idea is a natural transfer of head-selection from parsing (Zhang et al. 2017) to syntax-aware translation. It avoids extra modules, keeps decoding unchanged, and gives you interpretable attention heads. Across four language pairs the gains are consistently positive, around 0.5–1.5 BLEU over a strong Transformer baseline. The related-work citations to RNN-based syntax-aware NMT and to head-selection parsing are appropriate. If I worked in this area, I'd want to try this.

Now the soft spots, in order of importance. First, no significance tests and no multi-seed variance anywhere. With single runs, the smaller gains (NIST2005 +0.32 in Table 1) are within run-to-run noise, so "significantly improve" in the abstract is not supported by the reported numbers. Second, there is no non-syntactic control. The auxiliary losses force two heads to produce very sparse distributions and add a training signal regardless of whether the target encodes syntax. A control with random or dummy sparse targets would separate syntax-specific benefit from generic regularization. Without it, the causal story is underdetermined. Third, Section 4.5's tree-quality evaluation is circular: the reference is the same in-house parser that produced the pseudo-gold supervision, so a UAS of 83.25% against an 83.7% parser mostly shows the model learned to imitate its teacher. That section should be labeled as a consistency check, not quality estimation. Minor: the Chinese parser UAS is 83.7%, so the supervision itself is noisy; a robustness test or a sentence-level analysis would help.

None of this sinks the method. The central idea is sound and the consistent direction of gains is a real signal. But the paper currently cannot distinguish "dependency syntax helps" from "extra sparse supervision helps." A revision with a dummy-target control, multi-seed variance, and significance testing would make the claim much stronger.

This paper deserves a serious referee; it should not be desk-rejected. I'd send it out and in the review ask for the control and the statistics. For a reading group, it is a good case study in why ablation controls matter.

Referee Report

3 major / 4 minor

Summary. The paper proposes a method to incorporate source dependency knowledge into the Transformer encoder by supervising two self-attention heads, one for child dependencies and one for parent dependencies. Two cross-entropy auxiliary losses, Lc and Lp, are added to the standard translation objective. The method requires no extra modules and no parser at inference, since the supervised heads are learned to reproduce dependency matrices from the input. The paper reports BLEU improvements over a strong Transformer baseline across four tasks: NIST Chinese-to-English, English-to-Japanese (ASPEC), WMT17 Chinese-English (both directions), and WMT14 English-to-German. It also evaluates quality of the dependency trees constructed from the supervised heads and reports a UAS of 83.25% against a standalone parser.

Significance. The method is conceptually elegant and practically attractive: it injects syntactic information into the Transformer without changing the architecture or requiring parsing at inference, and the experiments cover several language pairs with consistent positive results. The supervised-head idea is a clean way to leverage the multi-head redundancy of the Transformer. However, the current evidence does not yet fully support the causal claim that dependency structure is responsible for the gains. The absence of significance testing and of a non-syntactic control means that a generic regularization or sparsification effect remains a viable explanation. The results are nonetheless encouraging and the experimental setup is standard, so the approach is likely reproducible from the description. The paper does not provide code, data, or machine-checked proofs, but the methodology is sufficiently precise to be re-implemented.

major comments (3)

[§4, Tables 1–3] The paper's central claim is that the proposed method 'significantly improve[s] the Transformer baseline' (abstract), but no significance tests, confidence intervals, or multiple runs are reported. For example, in Table 1 the smallest per-test-set gain with a single supervised head is +0.32 BLEU on NIST2005 (44.21 vs. 43.89), which is within the range of run-to-run noise typical for BLEU on such systems. Without an estimate of variance or a paired bootstrap test, the word 'significantly' is not supported by the reported numbers. Please provide variance over at least three seeds or conduct paired bootstrap significance tests, and adjust the claims in the abstract and conclusion accordingly.
[§3.2, Eqs. (6)–(9); Tables 1–3] The causal claim that source dependency structure is responsible for the BLEU gains is underdetermined because no non-syntactic control is tested. The auxiliary losses force the selected heads toward extremely sparse adjacency matrices (one parent per row, averaged child sets, and self-loops for leaves and root), which changes the entropy and gradient signal of those heads independently of whether the targets encode syntactic relations. An ablation with matched-sparsity non-syntactic targets (e.g., random trees, reversed trees, or fixed sparse patterns) is necessary to attribute the gains specifically to dependency knowledge. The consistent direction of improvements across tasks is encouraging, but it does not exclude a generic regularization or sparsification explanation.
[§4.5] The 'quality estimation' of the constructed dependency trees is circular. The reference for the UAS computation is the output of the in-house parser that generated the pseudo-golden matrices used as supervision for PSH. Since the model is trained to imitate that parser, the high agreement (83.25% vs. 83.7%) is partially by construction and cannot independently demonstrate that the model has learned linguistically valid dependencies. The evaluation should instead use a held-out gold-standard treebank, or be re-framed as 'agreement with the training-time parser' rather than as an independent validation of dependency quality.

minor comments (4)

[§4.1] The rule for handling BPE-segmented text — linking all subword pieces of a word to its first piece — is adopted without empirical justification. Since all experiments except the NIST Chinese-to-English task use BPE, this assumption affects most of the reported results; please discuss its potential effect or validate it with a small comparison.
[§4.1] There is a typo: 'CHS and PSH are selected from the top layer' should be 'CSH and PSH'.
[§4.6, Figure 3] The text refers to Figure 4(a)–(c), but the caption says Figure 3; please fix the cross-referencing.
[§5] The phrase 'they are usually lead to much longer input' is ungrammatical; it should be 'they usually lead to much longer input'.

Circularity Check

1 steps flagged · score 4.0 of 10

The main translation results are externally evaluated, but the paper's Section 4.5 'quality estimation' of constructed dependency trees is circular because it compares supervised attention heads against the same in-house parser outputs that generated their training targets.

self definitional [Section 4.5, 'Quality Estimation of Source Dependency Tree Construction']
"We estimate the consistency between the predicted trees and the parsing results of our stand-alone dependency parser due to the unavailable golden references. The higher the consistency is, the closer the performances are. ... We obtain a UAS of 83.25%, which demonstrates that the predicted dependency trees are highly similar to the parsing results from the stand-alone parser (the UAS of our stand-alone Chinese parser is 83.7%)."

The reference trees in this evaluation are the outputs of the same in-house arc-eager parser that produced the pseudo-golden dependency trees used to construct the supervision matrices Wc and Wp (Section 3.1, Eqs. 4-5; Section 4.1). The supervised attention heads CSH and PSH are trained by cross-entropy losses Lc and Lp (Eqs. 6, 7, 9) to match exactly those matrices, so computing agreement between predicted parent choices and the parser's parent choices measures how well the heads reproduced their own training labels. The reported UAS of 83.25% therefore cannot independently establish that the model learned real syntax; it only shows close fit to the supervision source.

full rationale

The central claim of the paper is that adding child and parent dependency supervision losses to two Transformer encoder attention heads improves translation quality on four language pairs. That claim is supported by BLEU/RIBES comparisons against the Transformer baseline (Tables 1-3), which are independent, externally defined metrics and not constructed from the method's own inputs. No fitted parameter is renamed as a prediction: the only hyper-parameters alpha and beta are selected on validation data, and the reported gains are not shown to be forced by the supervision construction. The main potential circularity is confined to Section 4.5, where the learned dependency trees are evaluated against the same parser outputs that generated the training supervisions. That evaluation is circular by construction and should not be cited as evidence that real syntax was learned. The paper also self-cites Wu et al. (2017) and re-implements it as SE-NMT, but that citation is used as a baseline comparison, not as load-bearing support for the proposed method. Overall, the translation-quality result is self-contained and externally falsifiable; only the auxiliary tree-quality sub-claim reduces to its training input. This warrants a moderate circularity score rather than a high one.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No invented entities. Two hyperparameters are fitted on validation sets. Auxiliary supervision relies on imperfect parser output, which is a domain assumption. The BPE linking rule is an ad hoc adaptation.

free parameters (2)

alpha (parent loss weight) = 0.4
Selected based on validation set to weight the child-dependency auxiliary loss in the training objective (Equation 9).
beta (child loss weight) = 0.4
Selected based on validation set to weight the parent-dependency auxiliary loss in the training objective (Equation 9).

assumptions (3)

domain assumption Dependency parses provide a useful inductive bias for NMT.
The whole method assumes that attending to parent/child nodes helps translation; this is supported by prior syntax-aware NMT work but not proven here (Sections 1, 3.2).
domain assumption In-house parser output (UAS 92.3/83.7) is trustworthy enough to serve as pseudo-golden supervision.
The auxiliary losses are trained against these parses; parser errors propagate into the model (Section 4.1).
ad hoc to paper For BPE-segmented text, linking all subword pieces to the first piece preserves the dependency structure.
This rule is introduced to adapt dependency trees to BPE; it may distort relations for multi-piece words (Section 4.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Source Dependency-Aware Transformer with Supervised Self-Attention." pith.science (2026). https://pith.science/paper/XDO7UZ45

@misc{pith2026190902273,
  author       = {Pith},
  title        = {Pith review of: Source Dependency-Aware Transformer with Supervised Self-Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDO7UZ45}},
  note         = {Machine review of arXiv:1909.02273}
}

read the original abstract

Recently, Transformer has achieved the state-of-the-art performance on many machine translation tasks. However, without syntax knowledge explicitly considered in the encoder, incorrect context information that violates the syntax structure may be integrated into source hidden states, leading to erroneous translations. In this paper, we propose a novel method to incorporate source dependencies into the Transformer. Specifically, we adopt the source dependency tree and define two matrices to represent the dependency relations. Based on the matrices, two heads in the multi-head self-attention module are trained in a supervised manner and two extra cross entropy losses are introduced into the training objective function. Under this training objective, the model is trained to learn the source dependency relations directly. Without requiring pre-parsed input during inference, our model can generate better translations with the dependency-aware context information. Experiments on bi-directional Chinese-to-English, English-to-Japanese and English-to-German translation tasks show that our proposed method can significantly improve the Transformer baseline.

Figures

Figures reproduced from arXiv: 1909.02273 by the authors.

**Figure 1.** (a). A translation example from the Chinese-to-English task. Text highlighted in the rectangle is the incorrect translation part. (b). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** The architecture of our syntax-aware Transformer. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

[1]

Neural machine transla- tion by jointly learning to align and translate

[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine transla- tion by jointly learning to align and translate. CoRR, abs/1409.0473,

arXiv 2014
[4]

On the shortest arborescence of a directed graph

[Chu, 1965] Yoeng-Jin Chu. On the shortest arborescence of a directed graph. Scientia Sinica, 14:1396–1400,

work page 1965
[7]

[Gehring et al., 2017b] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolu- tional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , pages 1243–1252,

work page 2017
[8]

Modeling source syntax for neural machine translation

[Li et al., 2017] Junhui Li, Deyi Xiong, Zhaopeng Tu, Muhua Zhu, Min Zhang, and Guodong Zhou. Modeling source syntax for neural machine translation. In Proceed- ings of ACL 2017, pages 688–697,

work page 2017
[9]

Le, Oriol Vinyals, and Wojciech Zaremba

[Luong et al., 2015] Thang Luong, Ilya Sutskever, Quoc V . Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. In Pro- ceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of...

work page 2015
[11]

Linguistic input features improve neural machine translation

[Sennrich and Haddow, 2016] Rico Sennrich and Barry Had- dow. Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Ma- chine Translation, WMT 2016, colocated with ACL 2016, August 11-12, Berlin, Germany, pages 83–91,

work page 2016
[12]

Neural machine translation of rare words with subword units

[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of ACL 2016,

work page 2016
[13]

Gomez, Lukasz Kaiser, and Illia Polosukhin

[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, pages 6000–6010,

work page 2017

Show all 16 references

[14]

Improved neural machine translation with source syntax

[Wu et al., 2017] Shuangzhi Wu, Ming Zhou, and Dongdong Zhang. Improved neural machine translation with source syntax. In Proceedings of IJCAI 2017, pages 4179–4185,

2017
[15]

Transition-based dependency parsing with rich non-local features

[Zhang and Nivre, 2011] Yue Zhang and Joakim Nivre. Transition-based dependency parsing with rich non-local features. In The 49th Annual Meeting of the Associ- ation for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Po...

2011
[1965]

Tree-to-sequence attentional neural machine translation

[Eriguchi et al., 2016] Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. Tree-to-sequence attentional neural machine translation. In Proceedings of ACL 2016,

2016
[2011]

Dependency parsing as head selection

[Zhang et al., 2017] Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. Dependency parsing as head selection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 665–676, 2017

2017
[2014]

Graph convolutional encoders for syntax-aware neural machine translation

[Bastings et al., 2017] Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima’an. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of EMNLP 2017, pages 1957– 1967,

2017
[2015]

ASPEC: asian scientiﬁc paper excerpt corpus

[Nakazawa et al., 2016] Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. ASPEC: asian scientiﬁc paper excerpt corpus. InProceedings of the Tenth International Conference on Language Resources and Evalua...

2016
[2016]

A convolutional encoder model for neural machine translation

[Gehring et al., 2017a] Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. A convolutional encoder model for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2017, Vancouver, Canada, July ...

2017
[2017]

Improved neural machine trans- lation with a syntax-aware encoder and decoder

[Chen et al., 2017] Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. Improved neural machine trans- lation with a syntax-aware encoder and decoder. In Pro- ceedings of the ACL 2017, pages 1936–1945,

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Neural machine transla- tion by jointly learning to align and translate

[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine transla- tion by jointly learning to align and translate. CoRR, abs/1409.0473,

arXiv 2014

[2] [4]

On the shortest arborescence of a directed graph

[Chu, 1965] Yoeng-Jin Chu. On the shortest arborescence of a directed graph. Scientia Sinica, 14:1396–1400,

work page 1965

[3] [7]

[Gehring et al., 2017b] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolu- tional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , pages 1243–1252,

work page 2017

[4] [8]

Modeling source syntax for neural machine translation

[Li et al., 2017] Junhui Li, Deyi Xiong, Zhaopeng Tu, Muhua Zhu, Min Zhang, and Guodong Zhou. Modeling source syntax for neural machine translation. In Proceed- ings of ACL 2017, pages 688–697,

work page 2017

[5] [9]

Le, Oriol Vinyals, and Wojciech Zaremba

[Luong et al., 2015] Thang Luong, Ilya Sutskever, Quoc V . Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. In Pro- ceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of...

work page 2015

[6] [11]

Linguistic input features improve neural machine translation

[Sennrich and Haddow, 2016] Rico Sennrich and Barry Had- dow. Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Ma- chine Translation, WMT 2016, colocated with ACL 2016, August 11-12, Berlin, Germany, pages 83–91,

work page 2016

[7] [12]

Neural machine translation of rare words with subword units

[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of ACL 2016,

work page 2016

[8] [13]

Gomez, Lukasz Kaiser, and Illia Polosukhin

[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, pages 6000–6010,

work page 2017