Pith. sign in

REVIEW 2 major objections 4 minor 31 references

A survey of cross-lingual features for zero-shot cross-lingual semantic parsing

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A semantic parser trained only on English can parse German, Italian, and Dutch when each word carries its universal dependency relation label; modeling the dependency tree itself does not help.

desk verdict A useful empirical comparison of cross-lingual features for DRS parsing, but the zero-shot claim is compromised by an unstated parallel-corpus overlap and missing significance tests. read the letter →

arxiv 1908.10461 v1 pith:BPBUQTVR submitted 2019-08-27 cs.CL

Jingfeng Yang , Federico Fancellu , Bonnie Webber This is my paper · ORCID
classification cs.CL
keywords zero-shotcross-lingualsemanticparsinguniversaldependenciesdependencyrelationembeddingstree-LSTMDiscourseRepresentationStructurewordmultilingualtransfercoarse-to-finedecoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Zero-shot cross-lingual semantic parsing asks whether a parser trained on English alone can interpret sentences in another language. This paper answers yes for German, Italian, and Dutch, with one major qualification: adding universal dependency relation labels to each input word sharply improves accuracy, while encoders that model the dependency tree itself do not beat a sequential baseline. The finding matters because it offers a path to semantic parsers for languages with no annotated meaning representations, without relying on parallel corpora or machine-translation projection. The practical lesson is that a flat, easily predicted feature carries the transfer, so complex syntax-aware architectures are not necessary.

What carries the argument

The central object is the encoder's token representation. Each input word is embedded as a concatenation of a multilingual word embedding, a universal POS-tag embedding, and a universal dependency-relation embedding, fed into either a sequential BiLSTM or a child-sum tree-LSTM operating over the dependency tree. The dependency-relation embedding is the load-bearing feature, and the comparison isolates flat relation labels from tree geometry. The decoder is a three-stage coarse-to-fine reconstructor for Discourse Representation Structures that first builds the box skeleton, then predicates, then referents, with a copy mechanism that transfers lexical predicates from the input when their lemmas align.

What would settle it

Check the multilingual graphbank's item identifiers to see whether any English training sentence is a translation of a German, Italian, or Dutch test sentence; if overlap exists, re-run the evaluation with those English sentences removed. If the dependency-feature boost disappears, the reported transfer is partly an artifact of leakage.

Watch

Extended reading notes

Core claim

The paper's central claim is that the cross-lingual transfer in zero-shot semantic parsing comes from universal dependency relation labels, not from the shape of the dependency tree. Across German, Italian, and Dutch, adding dependency-relation embeddings to a sequential BiLSTM encoder raises F1 by roughly 13 to 17 points over word embeddings with universal POS tags alone, and dependency relations alone outperform those two lexical features. Every tree-structured encoder tried—child-sum tree-LSTM, tree-LSTM with positional encodings, and a BiLSTM-initialized tree-LSTM—either underperforms or matches the sequential BiLSTM, both in zero-shot and monolingual settings. The authors conclude that dependency relation labels are the key transferable feature and that modeling the dependency structure directly when encoding the input does not help.

Load-bearing premise

The load-bearing premise is that no English training sentence is a translation of a target-language test sentence; if such overlap exists in the parallel corpus, the model could copy semantic content it has already seen and the zero-shot scores would be inflated.

Editorial extensions

If this is right

  • Languages without any semantic-parsing annotations can be served by an English-trained parser once universal dependency relation labels are available, sidestepping parallel corpora and machine-translation projection.
  • The dependency-relation embedding is the main source of cross-lingual transfer: adding it to a sequential BiLSTM gives larger gains than multilingual word embeddings or universal POS tags.
  • Tree-structured encoders are not worth their added complexity for zero-shot transfer, since sequential BiLSTMs match or beat them on German, Italian, and Dutch.
  • The copy mechanism lets the decoder output correct lexical predicates even for tokens unseen during training, provided the predicate aligns to an input lemma.
  • Operators and binary predicates remain the weak points of cross-lingual DRS parsing, while lexical predicates transfer well.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If dependency relation labels are the transferable signal, then automatically parsed UD labels in a new language may already be enough to extend the approach to typologically distant languages, even when the tree structure is unreliable.
  • The same feature decomposition could be tested on other meaning representations, such as Abstract Meaning Representation; the prediction would be that flat relation features transfer while tree encoders do not.
  • Because the multilingual word embeddings were kept fixed, the conclusion may be specific to static embeddings; fine-tuned contextual cross-lingual embeddings could shift the balance between lexical and syntactic features.
  • A data-efficiency experiment varying the amount of English training data would separate a genuine cross-lingual signal from simple overfitting to the source language.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper studies zero-shot cross-lingual semantic parsing, training a coarse-to-fine DRS parser (Liu et al., 2018) on English and testing it on German, Italian, and Dutch using the Parallel Meaning Bank. The authors replace English-only lexical features with language-independent features: MUSE multilingual word embeddings, universal POS tags, and universal dependency relation embeddings, and compare a sequential BiLSTM encoder with child-sum tree-LSTM variants. They report that adding dependency relation features substantially improves F1 across all three target languages, while tree-structured encoders do not reliably outperform the BiLSTM baseline. The paper concludes that dependency relation labels are the key transferable feature for zero-shot cross-lingual semantic parsing, and that directly modelling the dependency tree is not worth the added complexity.

Significance. If the central empirical claim holds, the paper provides a practically useful result: for zero-shot cross-lingual semantic parsing, dependency relation labels transfer well across languages, while tree encoders add complexity without consistent gains. The comparison spans three typologically related but distinct target languages, and the error analysis by predicate type is informative. The paper also demonstrates a parameter-shared alternative to machine-translation-projection approaches that does not require parallel data for alignment. On the other hand, the zero-shot claim is only meaningful if the English training split does not overlap semantically with the target-language test sets; the paper does not establish this, and because PMB is a parallel corpus the risk of leakage is concrete. The absence of significance testing also weakens the negative claim about tree encoders. These issues are addressable, but they are load-bearing for the paper's central conclusions.

major comments (2)
  1. [§2.2 (Data split)] The zero-shot claim requires that the meanings of the German, Italian, and Dutch test sentences were never seen during English training. Because PMB is a parallel translation corpus, every non-English test sentence has an English counterpart, and the paper does not state that the English training split (3072/663/670 from 4405 sentences) excludes those counterparts. If the split is random, a large fraction of the target test sentences have their exact English translations in the training data, so the model has already seen the semantic content in English. The paper must either provide sentence-ID-based evidence of no overlap or re-run the experiments with a leakage-free split; this is essential for both the positive result on dependency features and the negative result on tree encoders.
  2. [§3, Tables 1 and 2] The abstract claims that UD features 'significantly boost' performance and that modelling the UD structure directly 'does not' help, but no significance tests, confidence intervals, or multiple runs are reported. The positive gap between BiWE,PE and BiWE,PE,DE is large in all three languages, but the negative claim about tree encoders rests on small differences (e.g., German F1: BiWE,PE,DE 0.6392 vs. treeWE,PE,DE 0.6405; BiWE,DE 0.6290 vs. treeWE,DE 0.6400) that could easily arise from random seed variation. Additionally, the Bi/treeDE condition is missing from both tables, so the conclusion that combining the BiLSTM and treeLSTM does not help is not directly tested. Please add multiple seeds or significance tests and complete the missing condition.
minor comments (4)
  1. [Abstract and Introduction] There are several language and typos issues, e.g., 'has lead' should be 'has led', and the sentence 'test it where in another where annotations are not available' is garbled and should be rephrased.
  2. [Table 1 caption] The caption contains a typo: 'parsi ng' should be 'parsing'.
  3. [§2.3 (Cross-lingual features)] The paper does not report the dimensions of the MUSE embeddings or the POS/dependency relation embeddings, nor training hyperparameters such as learning rate, batch size, and number of epochs; these details are needed for reproducibility.
  4. [§3 (Results)] The paper reports results for models with and without dependency relation features, but does not discuss the fact that using dependency relations alone (BiDE) yields F1 comparable to using word and POS embeddings alone in some settings; a sentence explaining this would help readers interpret the feature ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's F1 comparisons are empirical measurements, not quantities defined by the feature inputs.

full rationale

The paper is an empirical comparison study. It takes an existing coarse-to-fine DRS parser (Liu et al., 2018), swaps encoder features (fixed multilingual embeddings, UD relation embeddings, universal POS embeddings) and encoder architectures (BiLSTM, child-sum TreeLSTM, variants), and measures Counter F1 on fixed PMB test splits. No equation in the paper defines the reported F1 scores in terms of the feature choices; the results are contingent on data and training. The only dependencies on prior work are the Liu et al. parser, MUSE embeddings, and UDPipe parses, all external resources whose outputs are not equivalent to the paper's conclusions. The self-citation to Fancellu et al. (2018) is a motivating analogy from negation scope detection, not a load-bearing argument for the paper's own result. There is no fitted-parameter-then-prediction move: all models are trained on the same English training split and evaluated on target-language test sets, and the abstract's claim is simply the observed ordering of those F1 values. A legitimate methodological concern - that the PMB parallel-corpus split may let English training sentences share content with target test sentences - would bear on the validity of the zero-shot claim, but it is not circularity: even under such leakage, the reported F1 values are not equivalent by construction to the feature inputs. No self-definitional step, renamed known result, or self-citation chain forces the outcome. Therefore the paper contains no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and no fitted constants; the model weights are trained on English, which is the standard supervised setup. The central claims depend on the quality of external tools (MUSE embeddings, UDPipe parsing) and on the PMB's gold token-predicate alignments. The most consequential unstated premise is the overlap-free split of the parallel data, which is listed as the weakest assumption.

assumptions (4)
  • domain assumption MUSE embeddings provide a shared cross-lingual vector space of sufficient quality for English-German/Italian/Dutch.
    Section 2.3: 'We use the MUSE pre-trained multilingual word embeddings and keep them fixed during training.'
  • domain assumption UDPipe provides reliable universal dependency trees and universal POS tags for all four languages.
    Section 2.3: UD relations and structure and Universal POS tags are obtained with UDPipe.
  • domain assumption Gold token-to-predicate alignments in the PMB are correct, and reverting lexical predicates to source-language lemmas is a valid preprocessing step.
    Section 2.2: 'we revert predicates to their original language by substituting them with the lemmas of the tokens they are aligned to (since gold alignment information is included in the PMB)'.
  • domain assumption The coarse-to-fine parser of Liu et al. (2018) is a sound baseline architecture for this task.
    Section 2.1: the model is built on the Liu et al. (2018) encoder-decoder with no decoder modifications.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A survey of cross-lingual features for zero-shot cross-lingual semantic parsing." pith.science (2026). https://pith.science/paper/BPBUQTVR

@misc{pith2026190810461,
  author       = {Pith},
  title        = {Pith review of: A survey of cross-lingual features for zero-shot cross-lingual semantic parsing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BPBUQTVR}},
  note         = {Machine review of arXiv:1908.10461}
}
read the original abstract

The availability of corpora to train semantic parsers in English has lead to significant advances in the field. Unfortunately, for languages other than English, annotation is scarce and so are developed parsers. We then ask: could a parser trained in English be applied to language that it hasn't been trained on? To answer this question we explore zero-shot cross-lingual semantic parsing where we train an available coarse-to-fine semantic parser (Liu et al., 2018) using cross-lingual word embeddings and universal dependencies in English and test it on Italian, German and Dutch. Results on the Parallel Meaning Bank - a multilingual semantic graphbank, show that Universal Dependency features significantly boost performance when used in conjunction with other lexical features but modelling the UD structure directly when encoding the input does not.

Figures

Figures reproduced from arXiv: 1908.10461 by the authors.

Figure 1
Figure 1. The Discourse Representation Structure (DRS) for “I sat down and opened my laptop”. For simplicity, we have omitted any time reference. building multilingual representations, e.g. PMB (Abzianidze et al., 2017), MRS (Copestake et al., 1995) and FrameNet(Pad´o and Lapata, 2005). However, manually annotating meaning represen￾tations in a new language is a painstaking process which explains why there are only a few data… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 20 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik Van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. arXiv preprint arXiv:1702.03964

    work page Pith review arXiv 2017

  4. [4]

    Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178--186

    work page 2013

  5. [5]

    Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. arXiv preprint arXiv:1707.05436

    work page Pith review arXiv 2017

  6. [6]

    Jianpeng Cheng, Siva Reddy, Vijay Saraswat, and Mirella Lapata. 2017. Learning structured natural language representations for semantic parsing. arXiv preprint arXiv:1704.08387

    work page Pith review arXiv 2017

  7. [7]

    Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv \'e J \'e gou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087

    arXiv 2017

  8. [8]

    Ann Copestake, Dan Flickinger, Rob Malouf, Susanne Riehemann, and Ivan Sag. 1995. Translation using minimal recursion semantics. In Proceedings of the Sixth International Conference on Theoretical and Methodological Issues in Machine Translation, pages 15--32

    work page 1995

Show all 31 references
  1. [9]

    Marco Damonte and Shay B Cohen. 2017. Cross-lingual abstract meaning representation parsing. arXiv preprint arXiv:1704.04539

    2017 arXiv

  2. [10]

    Marco Damonte and Shay B Cohen. 2018. Cross-lingual abstract meaning representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1146--1155

    2018

  3. [11]

    Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. arXiv preprint arXiv:1601.01280

    2016 arXiv

  4. [12]

    Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. arXiv preprint arXiv:1805.04793

    2018 arXiv

  5. [13]

    Long Duong, Hadi Afshar, Dominique Estival, Glen Pink, Philip Cohen, and Mark Johnson. 2017. Multilingual semantic parsing and code-switching. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 379--389

    2017

  6. [14]

    Federico Fancellu, Adam Lopez, and Bonnie Webber. 2018. Neural networks for cross-lingual negation scope detection. arXiv preprint arXiv:1810.02156

    2018 arXiv

  7. [15]

    Robin Jia and Percy Liang. 2016. http://aclweb.org/anthology/P/P16/P16-1002.pdf Data recombination for neural semantic parsing . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: ...

    2016

  8. [16]

    Hans Kamp and Uwe Reyle. 2013. From discourse to logic: Introduction to modeltheoretic semantics of natural language, formal logic and discourse representation theory, volume 42. Springer Science & Business Media

    2013

  9. [17]

    Percy Liang, Michael I Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389--446

    2013

  10. [18]

    Jiangming Liu, Shay B Cohen, and Mirella Lapata. 2018. Discourse representation structure parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 429--439

    2018

  11. [19]

    Phoebe Mulcaire, Swabha Swayamdipta, and Noah Smith. 2018. Polyglot semantic role labeling. arXiv preprint arXiv:1805.11598

    2018 arXiv

  12. [20]

    Rik van Noord, Lasha Abzianidze, Antonio Toral, and Johan Bos. 2018. Exploring neural methods for parsing discourse representation structures. Transactions of the Association for Computational Linguistics, 6:619--633

    2018

  13. [21]

    Sebastian Pad \'o and Mirella Lapata. 2005. Cross-linguistic projection of role-semantic information. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pages 859--866. Association for Computational Linguistics

    2005

  14. [22]

    Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086

    2011 arXiv

  15. [23]

    a ckstr \

    Siva Reddy, Oscar T \"a ckstr \"o m, Slav Petrov, Mark Steedman, and Mirella Lapata. 2017. Universal semantic parsing. arXiv preprint arXiv:1702.03196

    2017 arXiv

  16. [24]

    Milan Straka and Jana Strakov\' a . 2017. http://www.aclweb.org/anthology/K/K17/K17-3009.pdf Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe . In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages ...

    2017

  17. [25]

    Raymond Hendy Susanto and Wei Lu. 2017. Neural architectures for multilingual semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 38--44

    2017

  18. [26]

    Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075

    2015 arXiv

  19. [27]

    Rik Van Noord, Lasha Abzianidze, Hessel Haagsma, and Johan Bos. 2018. Evaluating scoped meaning representations. arXiv preprint arXiv:1802.08599

    2018 arXiv

  20. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008

    2017

  21. [29]

    Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. arXiv preprint arXiv:1704.01696

    2017 arXiv

  22. [30]

    Luke S Zettlemoyer and Michael Collins. 2012. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv:1207.1420

    2012 arXiv

  23. [31]

    Sheng Zhang, Kevin Duh, and Benjamin Van Durme. 2018. Cross-lingual semantic parsing. arXiv preprint arXiv:1804.08037

    2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.