Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data

Chris Reale; Jacob Harer; Peter Chin

Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data

T0 review · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Transformer variant with parent-sibling tree convolution improves code and grammar correction over sequence baselines and achieves the best reported F0.5 on the AESW benchmark.

desk verdict A genuinely novel tree-to-tree Transformer architecture with a striking code-correction result, but the GEC numbers are unverifiable until the authors specify the tree-to-text step. read the letter →

arxiv 1908.00449 v1 pith:QKYZBZ7J submitted 2019-08-01 cs.LG cs.CLstat.ML

Jacob Harer , Chris Reale , Peter Chin This is my paper · ORCID

classification cs.LGcs.CLstat.ML

keywords naturalcodecorrectiondatalanguagesourcetree-structuredachieved

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most correction systems treat sentences and source code as flat sequences of words or characters. But these inputs have a natural grammar structure that can be drawn as a tree. This paper builds a neural network that reads and writes those trees directly. The model is based on the Transformer, a popular sequence-processing architecture, with one key change: a tree convolution layer lets each node look at its parent and its left sibling, while a self-attention mechanism still lets every node see all earlier nodes. Nodes are ordered depth-first, so the network generates a corrected tree one node at a time. The authors removed the usual positional encoding because it hurt performance on their small datasets.

The method was tested in two domains. On the SATE IV dataset of vulnerable C and C++ functions, the Tree-Transformer scored an F0.5 of 84.7, well above the 63.5 scored by a sequence-based Transformer. On the CoNLL 2014 grammar correction test, it achieved a higher recall than prior systems (43.2 versus 38.9) but a lower F0.5 than the strongest previous system (55.09 versus 56.1). On the AESW scientific-writing benchmark, it reports the highest F0.5 so far, 50.43.

The paper does not release code or data, and one abstract number does not match the reported table. The core idea, making Transformers tree-aware, is plausible and could generalize to other tree-to-tree tasks.

Extended reading notes

Core claim

The paper's central claim is that replacing the feed-forward sublayers of a Transformer with a parent-sibling Tree Convolution Block (TCB), combined with depth-first node ordering and masked self-attention, yields a network that 'translate[s] between arbitrary input and output trees' and outperforms sequence-based models on correction tasks. Concretely, the authors claim a 25% F0.5 improvement over the best sequential method on SATE IV, comparable results on CoNLL 2014 with a 10% recall gain, and 'the highest to date F0.5 score on the AESW benchmark of 50.43' (Abstract; Tables 1, 2, 4). If correct, tree structure is a reliable inductive bias for code and grammar correction.

Load-bearing premise

The most fragile premise is representational: every GEC input is a constituency parse produced by the Stanford shift-reduce parser, and every output tree must be converted back to a surface sentence for scoring, but the paper never specifies the tree-to-text reconstruction or checks its quality (Sections 6.2, 3.3). If parser errors corrupt input trees, or if the linearization of generated trees is lossy or ambiguous, the reported F0.5 and recall numbers would not measure actual correction quality. This assumption is distinct from the architecture claim and is load-bearing for all natural-language results.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Honest take: this is a plausible architectural contribution with a real bug in the headline numbers and a missing piece in the GEC pipeline that keeps me from trusting the natural-language results. The Tree-Transformer idea—replacing the Transformer's feed-forward sublayers with a parent-sibling tree convolution block, using top-down depth-first ordering and self-attention—is genuinely new relative to the tree-LSTM and sequence-Transformer work they cite. The SATE IV preprocessing is described in enough detail to be reproducible-ish, and the code-correction result (84.7 F0.5 vs 63.5 for the sequence Transformer) is a large gain, even if the abstract's "25%" doesn't match Table 1 (33% relative). The architecture's motivation is clear and the batching trick that cuts training from months to a day is a practical plus.

On GEC, the missing tree-to-text step is the load-bearing problem. The paper never says how a generated constituency tree becomes a surface sentence for M2scorer. They parse input sentences, generate corrected trees, but the inverse linearization is absent. Without it, the CoNLL/AESW F0.5 numbers are not verifiable, and the parser-artifact confound is uncontrolled. This is not a minor omission. Also, the best CoNLL F0.5 (55.09) is below the published 56.1 they list, so "comparable" is doing work; the 10% recall gain is real but precision drops. No error bars or significance tests anywhere, no code release, and the AESW claim of "highest to date" depends only on the baselines in their own table. The SATE IV number is striking but rests on a nonstandard preprocessing pipeline—dead-code removal, de-duplication—that could inflate gains if not applied identically to the sequence baselines, and the paper doesn't say.

The architecture itself is not circular; hyperparameters tuned on validation are standard practice, and the self-citations to prior code-repair work are not a problem. The central architectural argument holds up. What doesn't hold up yet is the empirical case for the natural-language half. The code result is enough to warrant a serious look, but the GEC evidence should be treated as unverified until the linearizer and parser-artifact controls are described.

I would send this to peer review—the architecture is novel and the code-correction gain is large enough to demand a careful referee. The authors should fix the abstract's 25%, release code, and spell out tree-to-text reconstruction. A serious referee would focus on the GEC pipeline.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's empirical results rest on a standard Transformer training stack plus several domain-specific representation choices: parse trees for GEC, ASTs for code, and a custom SATE IV preprocessing pipeline. None of these are formalized or independently verified, so the ledger is dominated by domain assumptions rather than theoretical axioms.

free parameters (4)

alpha (monolingual ensemble weight) = 0.15
Selected on the CoNLL 2013 validation set (Section 6.2.1); controls the interpolation with the pretrained language model during beam search. Variants labeled +Mon-Ens use alpha greater than zero.
beam width = 6
Chosen empirically; the paper states that larger beam widths did not improve results (Section 5.2).
edit-weight lambda = 3 for edited tokens, 1 otherwise
Adopted from Junczys-Dowmunt et al. [8] for the edit-weighted MLE loss (Section 6.2.1); not fitted in this paper but still a hyperparameter affecting the objective.
core model hyperparameters (N=6, d_model=512, d_ff=2048, heads=8, dropout=0.3, attention dropout=0.1, label… = see Appendix A, Tables 5 and 6
Standard Transformer settings from Vaswani et al. [2], retained here. Several rows in the appendix tables are blank, so the exact configuration for each run is not fully documented.

assumptions (4)

domain assumption Constituency parse trees for incorrect sentences produced by the Stanford shift-reduce parser faithfully represent the syntactic structure available for correction.
Used to build all GEC inputs (Section 6.2). If the parser's output on ungrammatical input is unreliable, the model sees corrupted structure.
domain assumption The depth-first ordering with masked self-attention makes p(y|x)=prod_t p(y_t|y_<t,x) an adequate factorization for tree generation.
Equation (1), Sections 3.3, 3.4, and 4.2. This is the standard autoregressive assumption, but for tree outputs it depends on the chosen node order.
ad hoc to paper SATE IV preprocessing, including dead-code removal and deduplication of identical bad functions, preserves the difficulty of the code repair task.
Section 6.1. This is a new processing choice specific to this paper; if it makes the data easier, the reported F0.5 may not transfer to real-world repair.
domain assumption Clang ASTs and Clang source reconstruction provide an adequately invertible representation of code edits.
Section 6.1. The model is trained on ASTs and evaluated on source edits after Clang reconstruction; lossy inversion would bias the edit scores.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data." pith.science (2026). https://pith.science/paper/QKYZBZ7J

@misc{pith2026190800449,
  author       = {Pith},
  title        = {Pith review of: Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKYZBZ7J}},
  note         = {Machine review of arXiv:1908.00449}
}

abstract

Many common sequential data sources, such as source code and natural language, have a natural tree-structured representation. These trees can be generated by fitting a sequence to a grammar, yielding a hierarchical ordering of the tokens in the sequence. This structure encodes a high degree of syntactic information, making it ideal for problems such as grammar correction. However, little work has been done to develop neural networks that can operate on and exploit tree-structured data. In this paper we present the Tree-Transformer \textemdash{} a novel neural network architecture designed to translate between arbitrary input and output trees. We applied this architecture to correction tasks in both the source code and natural language domains. On source code, our model achieved an improvement of $25\%$ $\text{F}0.5$ over the best sequential method. On natural language, we achieved comparable results to the most complex state of the art systems, obtaining a $10\%$ improvement in recall on the CoNLL 2014 benchmark and the highest to date $\text{F}0.5$ score on the AESW benchmark of $50.43$.

Figures

Figures reproduced from arXiv: 1908.00449 by the authors.

**Figure 2.** Tree-Transformer State Transfer models follow the 6 layer architecture of the base Transformer model with sublayer outputs, dmodel, of size 512 and tree convolution layers, df f , of size 2048. 3.1 Parent-Sibling Tree Convolution Tree convolution is computed for each node as: T CB(xt, xp, xs) = relu(xtWt + xpWp + xsWs + b)W2 + b2 The inputs xt, xp, and xs all come from the previous sublayer, xt from the same node, x… view at source ↗

**Figure 3.** Example Constituency Parse Tree. The index of the node in depth-first ordering is shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 45 canonical work pages

[1]

Bahdanau, K

D. Bahdanau, K. Cho, and Y . Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations (ICLR), 2015

work page 2015
[2]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. Neural Information Processing Systems (NIPS) , 2017

work page 2017
[3]

Z. Xie, A. Avati, N. Arivazhagan, D. Jurafsky, and A. Y . Ng. Neural Language Correction with Character-Based Attention. arXiv.org, 2016

work page 2016
[4]

Yuan and T

Z. Yuan and T. Briscoe. Grammatical error correction using neural machine translation. North American Chapter of the Association of Computational Linguistics (NAACL), 2016

work page 2016
[5]

J. Ji, Q. Wang, K. Toutanova, Y . Gong, S. Truong, and J. Gao. A Nested Attention Neural Hybrid Model for Grammatical Error Correction. Association of Computational Linguistics (ACL), 2017

work page 2017
[6]

Schmaltz, Y

A. Schmaltz, Y . Kim, A. M. Rush, and S. M. Shieber. Adapting Sequence Models for Sentence Correction. Empirical Methods in Natural Language (EMNLP), 2017

work page 2017
[7]

Chollampatt and H

S. Chollampatt and H. T. Ng. A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction. Association for the Advancement of Artiﬁcial Intelligence (AAAI), 2018

work page 2018
[8]

Junczys-Dowmunt, R

M. Junczys-Dowmunt, R. Grundkiewicz, S. Guha, and K. Heaﬁeld. Approach neural grammati- cal error correction as a low-resource machine translation task. North American Chapter of the Association of Computational Linguistics (NAACL-HLT), 2018

work page 2018

Show all 47 references

[9]

K. S. Tai, R. Socher, and C. D. Manning. Improved Semantic Representations From Tree- Structured Long Short-Term Memory Networks. Association of Computational Linguistics (ACL), 2015

2015
[10]

X. Zhu, P. Sobhani, and H. Guo. Long Short-Term Memory Over Tree Structures. International Conference on Machine Learning (ICML), 2015

2015
[11]

Socher, C

R. Socher, C. C.-Y . Lin, A. Y . Ng, and C. D. Manning. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. International Conference on Machine Learning (ICML), 2011

2011
[12]

Eriguchi, K

A. Eriguchi, K. Hashimoto, and Y . Tsuruoka. Tree-to-Sequence Attentional Neural Machine Translation. Association of Computational Linguistics (ACL), 2016

2016
[13]

Dong and M

L. Dong and M. Lapata. Language to Logical Form with Neural Attention. Association of Computational Linguistics (ACL), 2016

2016
[14]

D. A.-M. . T. S. Jaakkola. Tree-structured decoding with doubly-recurrent neural networks. International Conference on Learning Representations (ICLR), 2017

2017
[15]

Vinyals, L

O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton. Grammar as a Foreign Language. Neural Information Processing Systems (NIPS), 2015

2015
[16]

Aharoni and Y

R. Aharoni and Y . Goldberg. Towards String-to-Tree Neural Machine Translation.Association of Computational Linguistics (ACL), 2017

2017
[17]

Rabinovich, M

M. Rabinovich, M. Stern, and D. Klein. Abstract Syntax Networks for Code Generation and Semantic Parsing. Association of Computational Linguistics (ACL), 2017

2017
[18]

Parisotto, A.-r

E. Parisotto, A.-r. Mohamed, R. Singh, L. Li, D. Zhou, and P. Kohli. Neuro-Symbolic Program Synthesis. International Conference on Learning Representations (ICLR), 2017

2017
[19]

Yin and G

P. Yin and G. Neubig. A Syntactic Neural Model for General-Purpose Code Generation. Association of Computational Linguistics (ACL), 2017

2017
[20]

Zhang, L

X. Zhang, L. Lu, and M. Lapata. Top-down Tree Long Short-Term Memory Networks. North American Chapter of the Association of Computational Linguistics (NAACL), 2016

2016
[21]

X. Chen, C. Liu, and D. Song. Tree-to-tree Neural Networks for Program Translation. Neural Information Processing Systems (NeurIPS), 2018

2018
[22]

Chakraborty, M

S. Chakraborty, M. Allamanis, and B. Ray. Tree2tree neural translation model for learning source code changes. arXiv pre-print, 2018. 9

2018
[23]

Monperrus

M. Monperrus. Automatic software repair: A bibliography. ACM Computing Surveys (CSUR), 2018

2018
[24]

X. B. D. Le, D. Lo, and C. Le Goues. History driven program repair. Software Analysis, Evolution, and Reengineering (SANER), 2016

2016
[25]

Long and M

F. Long and M. Rinard. Automatic patch generation by learning correct code. Principles of Programming Languages (POPL), 2016

2016
[26]

Semantic Code Repair using Neuro-Symbolic Transformation Networks

Devlin, Jacob, Uesato, Jonathan, Singh, Rishabh, and Kohli, Pushmeet. Semantic Code Repair using Neuro-Symbolic Transformation Networks. arXiv:1710.11054, 2017

2017 arXiv
[27]

Gupta, S

R. Gupta, S. Pal, A. Kanade, and S. Shevade. Deepﬁx: Fixing common c language errors by deep learning. Association for the Advancement of Artiﬁcal Intelligence (AAAI) , pages 1345–1351, 2017

2017
[28]

Harer, O

J. Harer, O. Ozdemir, T. Lazovich, C. P. Reale, R. L. Russell, L. Y . Kim, and P. Chin. Learning to Repair Software Vulnerabilities with Generative Adversarial Networks. Neural Information Processing Systems (NeuroIPS), 2018

2018
[29]

Junczys-Dowmunt and R

M. Junczys-Dowmunt and R. Grundkiewicz. Phrase-based Machine Translation is State-of-the- Art for Automatic Grammatical Error Correction. Empirical Methods in Natural Language (EMNLP), 2016

2016
[30]

Chollampatt and H

S. Chollampatt and H. T. Ng. Connecting the Dots: Towards Human-Level Grammatical Error Correction. The 12th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics (ACL), 2017

2017
[31]

Dahlmeier, H

D. Dahlmeier, H. T. Ng, and S. M. Wu. Building a Large Annotated Corpus of Learner English - The NUS Corpus of Learner English. North American Chapter of the Association of Computational Linguistics (NAACL), 2013

2013
[32]

V . Okun, A. Delaitre, and P. Black. Report on the static analysis tool exposition (sate) iv. Technical Report, 2013

2013
[33]

Lattner and V

C. Lattner and V . S. Adve. LLVM - A Compilation Framework for Lifelong Program Analysis & Transformation. CGO, 2004

2004
[34]

clang.llvm.org, 2011

Clang library. clang.llvm.org, 2011

2011
[35]

https://github.com/nusnlp/m2scorer/ releases, 2014

Ofﬁcal scorer for conll 2014 shared task. https://github.com/nusnlp/m2scorer/ releases, 2014

2014
[36]

Goller and A

C. Goller and A. Kuchler. Learning task-dependent distributed representations by backpropaga- tion through structure. International Conference on Neural Networks (ICNN’96), 1996

1996
[37]

Chen and C

D. Chen and C. D. Manning. A fast and accurate dependency parser using neural networks. Emperical Methods in Natural Language Processing (EMNLP), 2014

2014
[38]

Socher, J

R. Socher, J. Bauer, and A. Y . Manning, Christopher D.and Ng. Parsing with compositional vector grammars. Association for Computational Linguistics (ACL), 2013

2013
[39]

Klein and C

D. Klein and C. Manning. Accurate unlexicalized parsing. Association for Computational Linguistics (ACL), 2003

2003
[40]

https://nlp.stanford.edu/software/srparser

Shift-reduce constituency parser. https://nlp.stanford.edu/software/srparser. html, 2014

2014
[41]

https://web.archive.org/web/20130517134339/http://bulba

Penn treebank ii tags. https://web.archive.org/web/20130517134339/http://bulba. sdsu.edu/jeanette/thesis/PennTags.html, 2016

2016
[42]

Heinzerling and M

B. Heinzerling and M. Strube. BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018

2018
[43]

H. Ng, S. Wu, T. Briscoe, C. Hadiwinoto, R. Susanto, and C. Bryant. The conll-2014 shared task on grammatical error correction. Conference on Computational Natural Language Learning, Association for Computational Linguistics (ACL), 2014

2014
[44]

Daudaravicius, R

V . Daudaravicius, R. Banchs, E. V olodine, and C. Napoles. A report on the automatic evaluation of scientiﬁc writing shared task. 11th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics (ACL), 2016. 10

2016
[45]

https://pypi.org/project/apted/, 2015

Apted python library. https://pypi.org/project/apted/, 2015

2015
[46]

Pawlik and N

M. Pawlik and N. Augsten. Tree edit distance: Robust and memory-efﬁcient. Information Systems 56, 2016

2016
[47]

Pawlik and N

M. Pawlik and N. Augsten. Efﬁcient computation of the tree edit distance. ACM Transactions on Database Systems, 2015. 11 Appendix A hyperparameters Hyperparameters utilized are listed in tables 5 and 6. Default hyperparameters are listed at the top of each table. A blank means...

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Bahdanau, K

D. Bahdanau, K. Cho, and Y . Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations (ICLR), 2015

work page 2015

[2] [2]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. Neural Information Processing Systems (NIPS) , 2017

work page 2017

[3] [3]

Z. Xie, A. Avati, N. Arivazhagan, D. Jurafsky, and A. Y . Ng. Neural Language Correction with Character-Based Attention. arXiv.org, 2016

work page 2016

[4] [4]

Yuan and T

Z. Yuan and T. Briscoe. Grammatical error correction using neural machine translation. North American Chapter of the Association of Computational Linguistics (NAACL), 2016

work page 2016

[5] [5]

J. Ji, Q. Wang, K. Toutanova, Y . Gong, S. Truong, and J. Gao. A Nested Attention Neural Hybrid Model for Grammatical Error Correction. Association of Computational Linguistics (ACL), 2017

work page 2017

[6] [6]

Schmaltz, Y

A. Schmaltz, Y . Kim, A. M. Rush, and S. M. Shieber. Adapting Sequence Models for Sentence Correction. Empirical Methods in Natural Language (EMNLP), 2017

work page 2017

[7] [7]

Chollampatt and H

S. Chollampatt and H. T. Ng. A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction. Association for the Advancement of Artiﬁcial Intelligence (AAAI), 2018

work page 2018

[8] [8]

Junczys-Dowmunt, R

M. Junczys-Dowmunt, R. Grundkiewicz, S. Guha, and K. Heaﬁeld. Approach neural grammati- cal error correction as a low-resource machine translation task. North American Chapter of the Association of Computational Linguistics (NAACL-HLT), 2018

work page 2018