{"id":"4aaae2e0-e19d-49b2-819e-de8f539695f1","arxiv_id":"1908.10461","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Universal dependency relation features, but not explicit dependency tree structure, improve zero-shot cross-lingual semantic parsing on the Parallel Meaning Bank.","lead":"This paper tests whether a semantic parser trained only on English can understand Italian, German, and Dutch without any labeled training data in those languages. It finds dependency relation features help a lot, but building the input as a syntax tree does not beat a plain sequence model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The parallel-corpus split likely leaks target-language semantics through English training sentences, undermining the zero-shot claim.","rationale":"The reader's weakest assumption exactly matches the load-bearing concern: without an overlap-free split, the zero-shot evaluation is invalid. My stress test sharpens this from a mere possibility to a structural expectation. Because PMB is explicitly a parallel corpus and the English split is described without any cross-lingual exclusion, the default is that most target test sentences have English translations in training. This affects the validity of every result in Tables 1 and 2, not just the confidence intervals. The reader's CONDITIONAL verdict is reasonable, but I recommend UNVERDICTED rather than CONDITIONAL because the paper currently provides no evidence that the condition can be satisfied; the proposed overlap check is a prerequisite before any empirical claim can be evaluated. If the check shows zero overlap, the paper could move to CONDITIONAL pending significance testing; if it shows substantial overlap, the central claims would need to be retracted or re-scoped to a non-zero-shot setting. Thus the verdict should be held as UNVERDICTED until the alignment-based check is performed.","tokens_in":7732,"tokens_out":4735,"duration_ms":48613,"concrete_test":"Use PMB v2.1.0 sentence and document alignment IDs to map every German, Italian, and Dutch test sentence to its English counterpart. Report the fraction of target test sentences whose English counterpart appears in the 3072-sentence English training split (or development split). Then re-run the main comparisons—at minimum BiW E,PE versus BiW E,PE,DE and Bi versus tree/P o/tree/Bi/tree—on only the target test sentences whose English counterpart is absent from both English train and dev. If the F1 differences shrink, reverse, or drop substantially, the reported zero-shot results are leakage artifacts.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim depends on a genuine zero-shot setting: the model must parse German, Italian, and Dutch sentences whose meanings were never seen in English training. The paper does not ensure this. Section 2.2 reports 4405 English sentences split into 3072 training, 663 development, and 670 test sentences, and states that 'all the sentences in other languages' are used as test. PMB is a parallel corpus of translations, so each non-English test sentence is a translation of an English sentence. Unless the English split deliberately excluded the English counterparts of those target test sentences, a large fraction (roughly 70% if the split is random) of German, Italian, and Dutch test sentences have their exact semantic content present in English training. This is not zero-shot semantic parsing; it is cross-lingual transfer of previously seen meanings. The leakage can inflate absolute F1 scores and can change the relative ordering of feature and architecture variants, so both parts of the abstract claim—that dependency features help and that tree encoders do not—are at risk. The paper provides no sentence-ID-based overlap analysis and no statement that overlap was avoided, so the burden is on the authors to rule out leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies zero-shot cross-lingual semantic parsing, training a coarse-to-fine DRS parser (Liu et al., 2018) on English and testing it on German, Italian, and Dutch using the Parallel Meaning Bank. The authors replace English-only lexical features with language-independent features: MUSE multilingual word embeddings, universal POS tags, and universal dependency relation embeddings, and compare a sequential BiLSTM encoder with child-sum tree-LSTM variants. They report that adding dependency relation features substantially improves F1 across all three target languages, while tree-structured encoders do not reliably outperform the BiLSTM baseline. The paper concludes that dependency relation labels are the key transferable feature for zero-shot cross-lingual semantic parsing, and that directly modelling the dependency tree is not worth the added complexity.","tokens_in":7910,"tokens_out":3164,"duration_ms":32197,"significance":"If the central empirical claim holds, the paper provides a practically useful result: for zero-shot cross-lingual semantic parsing, dependency relation labels transfer well across languages, while tree encoders add complexity without consistent gains. The comparison spans three typologically related but distinct target languages, and the error analysis by predicate type is informative. The paper also demonstrates a parameter-shared alternative to machine-translation-projection approaches that does not require parallel data for alignment. On the other hand, the zero-shot claim is only meaningful if the English training split does not overlap semantically with the target-language test sets; the paper does not establish this, and because PMB is a parallel corpus the risk of leakage is concrete. The absence of significance testing also weakens the negative claim about tree encoders. These issues are addressable, but they are load-bearing for the paper's central conclusions.","major_comments":[{"comment":"The zero-shot claim requires that the meanings of the German, Italian, and Dutch test sentences were never seen during English training. Because PMB is a parallel translation corpus, every non-English test sentence has an English counterpart, and the paper does not state that the English training split (3072/663/670 from 4405 sentences) excludes those counterparts. If the split is random, a large fraction of the target test sentences have their exact English translations in the training data, so the model has already seen the semantic content in English. The paper must either provide sentence-ID-based evidence of no overlap or re-run the experiments with a leakage-free split; this is essential for both the positive result on dependency features and the negative result on tree encoders.","section":"§2.2 (Data split)"},{"comment":"The abstract claims that UD features 'significantly boost' performance and that modelling the UD structure directly 'does not' help, but no significance tests, confidence intervals, or multiple runs are reported. The positive gap between BiWE,PE and BiWE,PE,DE is large in all three languages, but the negative claim about tree encoders rests on small differences (e.g., German F1: BiWE,PE,DE 0.6392 vs. treeWE,PE,DE 0.6405; BiWE,DE 0.6290 vs. treeWE,DE 0.6400) that could easily arise from random seed variation. Additionally, the Bi/treeDE condition is missing from both tables, so the conclusion that combining the BiLSTM and treeLSTM does not help is not directly tested. Please add multiple seeds or significance tests and complete the missing condition.","section":"§3, Tables 1 and 2"}],"minor_comments":[{"comment":"There are several language and typos issues, e.g., 'has lead' should be 'has led', and the sentence 'test it where in another where annotations are not available' is garbled and should be rephrased.","section":"Abstract and Introduction"},{"comment":"The caption contains a typo: 'parsi ng' should be 'parsing'.","section":"Table 1 caption"},{"comment":"The paper does not report the dimensions of the MUSE embeddings or the POS/dependency relation embeddings, nor training hyperparameters such as learning rate, batch size, and number of epochs; these details are needed for reproducibility.","section":"§2.3 (Cross-lingual features)"},{"comment":"The paper reports results for models with and without dependency relation features, but does not discuss the fact that using dependency relations alone (BiDE) yields F1 comparable to using word and POS embeddings alone in some settings; a sentence explaining this would help readers interpret the feature ablation.","section":"§3 (Results)"}],"recommendation":"major_revision","confidential_remarks":"The parallel-corpus leakage issue is the main risk to the paper's central claim. If the authors can show that the English training sentences are disjoint from the English counterparts of the target test sentences, or if they re-run with a proper split and the results hold, the paper would be a solid empirical contribution. Otherwise the headline claims about zero-shot transfer are not supported. The missing significance testing and the missing Bi/treeDE condition should also be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a compact empirical study that asks a practical question—what features let an English-trained DRS parser transfer to German, Italian, and Dutch. The answer it gives, dependency relation embeddings help a lot and tree encoders don't, is plausible and consistent across languages. But the paper doesn't rule out leakage from the parallel PMB split, and without significance tests the encoder comparison is fragile. So treat the headline as conditional.\n\nWhat's new: prior cross-lingual semantic parsing work relied on MT projection; here they use parameter sharing with language-independent features on PMB, which is a first for DRS parsing. The main positive result is large: adding dependency relation embeddings improves F1 by about 0.10–0.15 across all three languages (e.g., German BiWE,PE 0.48 vs. BiWE,PE,DE 0.64). That's a big, consistent effect and worth knowing. The monolingual English results and error analysis are also honest extras that help interpret the cross-lingual numbers.\n\nSoft spots: the stress-test concern is real. PMB is a parallel corpus, and Section 2.2 does not state that English counterparts of target test sentences were excluded from training. If any are present in the 3072 training sentences, the model has already seen the semantics in English, and \"zero-shot\" overstates the setting. This could inflate absolute scores and could also change the relative feature ranking, so both parts of the abstract claim are at risk. The authors need to report overlap statistics or re-split. Separately, there are no error bars or significance tests, so the null result on tree encoders is not robust—differences of 0.02 F1 are within noise. The missing Bi/treeDE row in both tables further weakens the claim that combining tree and BiLSTM doesn't help. No code or hyperparameters are provided, which limits reproducibility.\n\nIn proportion: the dependency-feature benefit is large enough that it may survive leakage checks, but we can't count on that without the authors showing it. The paper's design is otherwise sensible, and the writing is clear.\n\nWho it's for: researchers working on multilingual semantic parsing or transfer learning. They'll find useful baselines and a clear feature recommendation, but should treat the zero-shot claim with caution until the split is cleaned up.\n\nRecommendation: I would send it to peer review, with the expectation of major revision: either demonstrate no parallel overlap or soften the zero-shot framing, and add significance testing or at least variance estimates. The empirical question is important and the current results are suggestive enough to deserve referee time.","headline":"A useful empirical comparison of cross-lingual features for DRS parsing, but the zero-shot claim is compromised by an unstated parallel-corpus overlap and missing significance tests.","tokens_in":8453,"tokens_out":2343,"would_cite":false,"duration_ms":25783,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A semantic parser trained only on English can parse German, Italian, and Dutch when each word carries its universal dependency relation label; modeling the dependency tree itself does not help.","keywords":["zero-shot cross-lingual semantic parsing","universal dependencies","dependency relation embeddings","tree-LSTM","Discourse Representation Structure parsing","cross-lingual word embeddings","multilingual transfer","coarse-to-fine decoding"],"falsifier":"Check the multilingual graphbank's item identifiers to see whether any English training sentence is a translation of a German, Italian, or Dutch test sentence; if overlap exists, re-run the evaluation with those English sentences removed. If the dependency-feature boost disappears, the reported transfer is partly an artifact of leakage.","tokens_in":7519,"feed_emoji":"🌐","tokens_out":8964,"duration_ms":79645,"temperature":0.7,"pith_summary":"Zero-shot cross-lingual semantic parsing asks whether a parser trained on English alone can interpret sentences in another language. This paper answers yes for German, Italian, and Dutch, with one major qualification: adding universal dependency relation labels to each input word sharply improves accuracy, while encoders that model the dependency tree itself do not beat a sequential baseline. The finding matters because it offers a path to semantic parsers for languages with no annotated meaning representations, without relying on parallel corpora or machine-translation projection. The practical lesson is that a flat, easily predicted feature carries the transfer, so complex syntax-aware architectures are not necessary.","feed_headline":"Dependency labels, not syntax trees, power zero-shot parsing","feed_subtitle":"Training a semantic parser in English transfers to German, Italian, and Dutch once dependency relations are added.","key_machinery":"The central object is the encoder's token representation. Each input word is embedded as a concatenation of a multilingual word embedding, a universal POS-tag embedding, and a universal dependency-relation embedding, fed into either a sequential BiLSTM or a child-sum tree-LSTM operating over the dependency tree. The dependency-relation embedding is the load-bearing feature, and the comparison isolates flat relation labels from tree geometry. The decoder is a three-stage coarse-to-fine reconstructor for Discourse Representation Structures that first builds the box skeleton, then predicates, then referents, with a copy mechanism that transfers lexical predicates from the input when their lemmas align.","core_discovery":"The paper's central claim is that the cross-lingual transfer in zero-shot semantic parsing comes from universal dependency relation labels, not from the shape of the dependency tree. Across German, Italian, and Dutch, adding dependency-relation embeddings to a sequential BiLSTM encoder raises F1 by roughly 13 to 17 points over word embeddings with universal POS tags alone, and dependency relations alone outperform those two lexical features. Every tree-structured encoder tried—child-sum tree-LSTM, tree-LSTM with positional encodings, and a BiLSTM-initialized tree-LSTM—either underperforms or matches the sequential BiLSTM, both in zero-shot and monolingual settings. The authors conclude that dependency relation labels are the key transferable feature and that modeling the dependency structure directly when encoding the input does not help.","pith_inferences":["If dependency relation labels are the transferable signal, then automatically parsed UD labels in a new language may already be enough to extend the approach to typologically distant languages, even when the tree structure is unreliable.","The same feature decomposition could be tested on other meaning representations, such as Abstract Meaning Representation; the prediction would be that flat relation features transfer while tree encoders do not.","Because the multilingual word embeddings were kept fixed, the conclusion may be specific to static embeddings; fine-tuned contextual cross-lingual embeddings could shift the balance between lexical and syntactic features.","A data-efficiency experiment varying the amount of English training data would separate a genuine cross-lingual signal from simple overfitting to the source language."],"forward_implications":["Languages without any semantic-parsing annotations can be served by an English-trained parser once universal dependency relation labels are available, sidestepping parallel corpora and machine-translation projection.","The dependency-relation embedding is the main source of cross-lingual transfer: adding it to a sequential BiLSTM gives larger gains than multilingual word embeddings or universal POS tags.","Tree-structured encoders are not worth their added complexity for zero-shot transfer, since sequential BiLSTMs match or beat them on German, Italian, and Dutch.","The copy mechanism lets the decoder output correct lexical predicates even for tokens unseen during training, provided the predicate aligns to an input lemma.","Operators and binary predicates remain the weak points of cross-lingual DRS parsing, while lexical predicates transfer well."],"supporting_citations":[{"why":"Supplies the coarse-to-fine encoder-decoder parser that all experiments modify.","marker":"Liu et al., 2018"},{"why":"Provides the multilingual DRS graphbank with gold alignments used for training and evaluation.","marker":"Abzianidze et al., 2017"},{"why":"Supplies the fixed multilingual word embeddings used as a lexical feature.","marker":"Conneau et al., 2017"},{"why":"Produces the universal POS tags and dependency parses for all four languages.","marker":"Straka and Straková, 2017"},{"why":"Defines the universal POS tagset used as a feature.","marker":"Petrov et al., 2011"},{"why":"Provides the child-sum tree-LSTM encoder used to model dependency structure.","marker":"Tai et al., 2015"},{"why":"Supplies the positional encoding scheme used by the Po/tree model.","marker":"Vaswani et al., 2017"},{"why":"Provides the BiLSTM-initialized tree-LSTM combination used by the Bi/tree model.","marker":"Chen et al., 2017"},{"why":"Supplies the Counter evaluation metric that computes DRS precision, recall, and F1.","marker":"van Noord et al., 2018"}],"fun_headline_variants":["Labels, not trees, transfer parsing across languages","Zero-shot parsing: dependency labels alone win","Syntax trees don't help; dependency labels do","For cross-lingual parsing, skip tree structure"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that no English training sentence is a translation of a target-language test sentence; if such overlap exists in the parallel corpus, the model could copy semantic content it has already seen and the zero-shot scores would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Labels, not trees, transfer parsing across languages","Zero-shot parsing: dependency labels alone win","Syntax trees don't help; dependency labels do","For cross-lingual parsing, skip tree structure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1243,"prompt_tokens":840,"completion_tokens":403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":344}},"tokens_in":456,"tokens_out":403,"duration_ms":4611,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:43:08.519028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the multilingual graphbank's item identifiers to see whether any English training sentence is a translation of a German, Italian, or Dutch test sentence; if overlap exists, re-run the evaluation with those English sentences removed. If the dependency-feature boost disappears, the reported transfer is partly an artifact of leakage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the coarse-to-fine encoder-decoder parser that all experiments modify."},{"cited_title":"The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations","cited_arxiv_id":"1702.03964","evidence_quote":"Provides the multilingual DRS graphbank with gold alignments used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Produces the universal POS tags and dependency parses for all four languages."}],"review_version":1}