{"id":"5306d7e0-cce7-4585-9d72-04aacc0bc6ce","arxiv_id":"2412.06107","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Appending syntactic and semantic analyses to prompts improves text-to-SQL accuracy in four low-resource languages and speeds fine-tuning.","lead":"This paper shows that adding dependency-parse and semantic (AMR) information directly into prompts improves translation of natural-language questions to SQL in French, Spanish, Portuguese, and Chinese. The technique works with standard fine-tuned transformer models and, in two languages, surpasses previously published accuracy on the Spider benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never validates amrlib's AMR graphs for French/Spanish/Portuguese/Chinese, and its only shown AMR example is an English graph for a French query; without a semantic-content control, the reported gains may not be caused by semantic information.","rationale":"I read the paper as an empirical study arguing that appending dependency-tree and AMR serializations to prompts improves NL2SQL in low-resource languages. The pattern of improvements across languages, architectures, and epoch counts is consistent and worth taking seriously, which is why I do not move the reader's verdict to reject. However, the causal claim specifically names semantic information as the active ingredient, and the only semantic component is amrlib, an English-oriented AMR parser. The paper gives no evidence that amrlib's graphs are meaningful for French, Spanish, Portuguese, or Chinese, and the one displayed AMR example is in English for a French sentence, while the dependency prompt example also uses English column names. This is not merely a missing detail: it undermines the attribution of the gains to semantics. The reader's weakest assumption identified the same gap, and the proposed scrambled-AMR control directly tests whether semantic structure matters. Since the reader already conditioned acceptance on stronger evidence, my stress-test does not change the verdict; it sharpens the specific condition that should be met.","tokens_in":13049,"tokens_out":11432,"duration_ms":119772,"concrete_test":"Run one control experiment on Portuguese (or French) with T5-base at 32 and 128 epochs: replace the AMR linearization in the prompt with a randomized permutation of the same AMR tokens, preserving token count, brackets, and length but destroying graph structure, and compare EM/EX to the paper's AMR condition. If the randomized-prompt accuracy is statistically indistinguishable from the AMR-prompt accuracy, the semantic content is not the active ingredient and the paper's causal claim fails; if it is clearly worse, the AMR structure is doing real work and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states that 'Dependency trees and AMR graphs have been generated by the Spacy parser and amrlib, respectively,' while Section 1's central claim is that 'syntactic and semantic information lead to better scores.' amrlib's released models target English AMR, yet the paper provides no check that they produce meaningful AMR graphs for the four non-English languages evaluated. The only AMR example in Section 3 is an English PropBank graph—'(l / list-01 ... year ... create-01 ... department ...)'—for a French input sentence, and the final dependency prompt also switches from French lemmas to English column names ('year', 'name', 'budget', 'department'). This is load-bearing: if the actual prompts contain English or degenerate semantic strings, the consistent improvements in Tables 1 and 2 may be attributable to injected English cues, to format tokens, or to the model ignoring the extra text, rather than to the claimed semantic information. No control condition with scrambled or nonce AMR blocks is reported, and no cross-lingual AMR validation is described, so the causal attribution in the abstract and conclusion is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to improve NL2SQL for low-resource languages by appending serialized dependency trees and AMR graphs to the input prompts of sequence-to-sequence models. Using translated versions of Spider in French, Spanish, Portuguese, and Chinese, the authors fine-tune T5, BART, and mT5 with and without the infused linguistic information, reporting consistent improvements in exact-set match and execution accuracy, faster convergence (32 vs. 128 epochs), and new state-of-the-art results for Portuguese and French. The manuscript also includes an English control experiment and a manual evaluation of the Portuguese translation quality.","tokens_in":13273,"tokens_out":5743,"duration_ms":55634,"significance":"If the results hold, the contribution is a simple, architecture-agnostic prompt augmentation that improves low-resource semantic parsing and reduces training cost. The paper's strengths are its comparison across four languages, two base architectures, and two training budgets; the appended translation-quality evaluation; and the honest negative result on English. Its principal weakness is that the semantic component is not validated for the target languages: amrlib is English-oriented, and no control rules out the possibility that the gains come from added structured tokens or English cues rather than from meaningful AMR. The absence of error bars and single-run reporting also leaves the magnitude of the gains uncertain.","major_comments":[{"comment":"The AMR parser amrlib is trained on English AMR, and the paper provides no evidence that its graphs are meaningful for French, Spanish, Portuguese, or Chinese. Because AMR alone gives the largest gains in most rows of Tables 1 and 2, the central claim that semantic information causes the improvement is not established. Please add (a) a sample of AMR outputs for each language with human or intrinsic evaluation, and (b) a control experiment with scrambled or nonce AMR serializations (or a random graph baseline) to show that the linearized graph content, not the extra bracket/token structure, is responsible for the gains.","section":"Section 4, paragraph on dependency trees and AMR graphs; Tables 1-2"},{"comment":"The dependency-tree prompt shown for the French example uses English lexical items ('year', 'name', 'budget', 'department') even though the dependency tree displayed immediately above is in French ('année', 'nom', 'budget', 'département'). If this reflects the actual prompt construction, the 'with syntax' condition also changes the language mix relative to the 'without info' baseline, conflating structural information with English token injection. Please clarify how the dependency-to-text conversion maps tokens and, if English is introduced, ablate it separately.","section":"Section 3, running example and Figure 1"},{"comment":"All experiments appear to be single runs with no error bars, confidence intervals, or significance tests. Many reported differences are small (e.g., French T5 with syntax vs. without info at 32 epochs is a 1.1-point EM difference in Table 1 top), so the consistency of the improvement cannot be assessed. Please report multiple seeds with means and standard deviations, or paired tests, at least for the main comparisons.","section":"Tables 1-3 and Section 4"},{"comment":"The claim of surpassing the previous state of the art is based on a comparison with the authors' own prior system (Jose and Cozman 2023), for which no execution accuracy is available, and no details are given for the T5-3B plus NatSQL setup (e.g., whether it uses RESDSQL, which dev-set split, how many runs). Please specify the comparison protocol and, ideally, compare against an external published multilingual NL2SQL system on the same dev sets.","section":"Table 4 and Section 4"}],"minor_comments":[{"comment":"The figure labels the input side 'RoBERTa' while the generation models in the experiments are T5 or BART; clarify the role of RoBERTa in the RESDSQL schema-linking stage.","section":"Section 3, Figure 1"},{"comment":"The vEM and vEX columns are said to be relative variations but the formula is not given, and the captions do not state that the values are percentages; please add the definition and units.","section":"Section 4 and Tables 1-2"},{"comment":"The sentence 'Our code and resulting models will be made publicly available in case the present text is accepted' should be replaced with a concrete availability statement (repository, license, model links) in the accepted version.","section":"Section 4"},{"comment":"There are spacing artifacts in references (e.g., 'Y aghmazadeh', 'V Y') and a duplicated 'and' in the sentence describing Li et al.; please proofread the final text.","section":"References and Section 2"},{"comment":"The linearized AMR is written with spaces in 'list - 01' and 'year :time - of'; standard AMR notation has no such spaces, and the paper should ensure the shown linearization is exactly what is fed to the model.","section":"Section 3, AMR example"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the semantic effect is an artifact of applying an English-oriented AMR parser to non-English input; the authors should be asked for cross-lingual validation and a noise control before acceptance. The state-of-the-art claim is narrow and self-referential, so it should be framed more cautiously unless an external comparison is added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical study of a simple idea—append dependency and AMR serializations to NL2SQL prompts for four non-English languages—and the improvement pattern is consistent enough to take seriously. But the paper's central claim that syntax and semantics cause the gains is under-supported. The main missing controls are easy to run, so I would send it to review with a request for revision rather than desk reject.\n\nWhat is genuinely new: prompt-level infusion for NL2SQL across French, Spanish, Portuguese, and Chinese, tested with T5, BART, and mT5, plus the training-efficiency observation that 32 epochs with extra linguistic text roughly matches 128 epochs without it. The English negative control is a good idea, and the translation-quality appendix shows more care than most papers in this area. Credit where due: the design is straightforward, and the consistency of the pattern across languages and models is hard to write off as pure noise.\n\nNow the soft spots. First, amrlib is an English-oriented AMR pipeline, and the paper never checks that its graphs for French, Spanish, Portuguese, or Chinese are meaningful. The only shown AMR example is an English PropBank graph for a French sentence. That is not by itself fatal—AMR concepts are English words by design—but without validation or a sample of graphs per language, you cannot tell whether the model is reading real semantics or structured English tokens. Second, there is no control for non-linguistic prompt content. The dependency prompt contains English column names and relation tags; a control with the same tokens shuffled, or with matched English words but no dependency relations, would separate format effects from syntactic information. Third, the results are single runs, no error bars are reported, and code is promised only on acceptance. Fourth, the state-of-the-art comparison for Portuguese and French is against the authors' own earlier system with a different framework; it is disclosed, so not circular, but the SOTA claim should be read narrowly.\n\nNone of these sink the empirical pattern. The paper would be much stronger with: a small validation of AMR graphs per language, a scrambled-AMR or scrambled-syntax control, and released code and data. The English experiments already serve as a useful negative control. I would accept this for peer review. The result matters if it holds, and the causal claim needs one more round of evidence before I would trust the conclusion.","headline":"Plausible empirical result on prompt-infused syntax and AMR for multilingual NL2SQL, but the causal attribution to linguistic structure is not yet established because the cross-lingual parser output is unvalidated and no non-linguistic control is run.","tokens_in":13765,"tokens_out":2681,"would_cite":false,"duration_ms":29109,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Appending serialized dependency trees and AMR graphs to prompts improves multilingual text-to-SQL and surpasses prior state-of-the-art results on Portuguese and French.","keywords":["text-to-SQL","NL2SQL","prompt infusion","dependency parsing","Abstract Meaning Representation","low-resource languages","Spider benchmark","multilingual language models"],"falsifier":"A control experiment that replaces each AMR serialization with a scrambled or randomly generated bracketed string of the same length would settle the semantic claim: if scrambled graphs preserve the accuracy gains, the semantic content is not what is doing the work.","tokens_in":12873,"feed_emoji":"🗄️","tokens_out":9909,"duration_ms":84456,"temperature":0.7,"pith_summary":"This paper argues that appending cheap linguistic analyses to prompts can substantially improve how language models translate natural-language questions into SQL, especially for languages with fewer training resources than English. The authors concatenate a filtered dependency-tree serialization and a linearized Abstract Meaning Representation (AMR) graph to each input question, without altering the model architecture. On translated versions of the Spider benchmark, they report consistent gains in exact-set match and execution accuracy for French, Spanish, Portuguese, and Chinese, with AMR infusion usually the strongest single signal. Their best T5-based models surpass the previously reported state of the art for Portuguese and French. The same infusion gives little or no benefit on English, which the paper attributes to large models already absorbing the relevant linguistic information during pretraining.","feed_headline":"Grammar tags and AMR graphs lift text-to-SQL past previous best","feed_subtitle":"Gains on French, Portuguese, Spanish and Chinese text-to-SQL, with faster training at lower epochs.","key_machinery":"The mechanism is prompt-level infusion: instead of modifying attention or embeddings, the authors serialize the output of off-the-shelf parsers and concatenate it with the question before the transformer sees it. For syntax, they take dependency trees and keep only subject, object, and conjunction relations, rendered as '[row] head; relation [row] dependent; ...' using the serialization of Li et al. (2021). For semantics, they linearize Abstract Meaning Representation (AMR) graphs—directed acyclic graphs whose nodes are language-independent concepts and edges are semantic roles—into bracket notation such as '(l / list-01 :ARG1 ...)', tagging it with '[AMR]'. The resulting prompt is what trains and, optionally, prompts the model; no component of the underlying NL2SQL framework is changed.","core_discovery":"The discovery is that explicit syntactic and semantic structure, written into the prompt as plain text, helps sequence-to-sequence models do better at NL2SQL in low-resource languages. Dependency relations (subjects, objects, conjunctions) are serialized as '[row] head; dobj [row] dependent; ...' following the format of Li et al. (2021), and AMR graphs are flattened with '[AMR]' as a separator. The enlarged prompt is fed into T5 or BART models inside a schema-pruning, skeleton-decoding NL2SQL pipeline, which first selects relevant tables and columns and then decodes the SQL skeleton before filling in values. Across four non-English languages and two model families, every infused variant outperformed the corresponding raw-input model; AMR alone was often best, and syntax plus AMR was best for BART and Chinese. The paper also reports that infused models reach target accuracy with fewer epochs than uninfused models, and that a 3-billion-parameter T5 using the flattened SQL intermediate representation of Gan et al. (2021) exceeds the previous reported exact-set-match score for Portuguese (0.752 vs. 0.687) and French (0.749 vs. 0.698).","pith_inferences":["A testable extension is to apply the same serialized-parse and AMR prompt pattern to other relation-sensitive generation tasks, such as event extraction or structured summarization, since the pipeline does not depend on SQL.","The English results suggest a resource-volume crossover: as pretraining data for a language grows, explicit linguistic annotation matters less; locating that crossover would tell practitioners when parser costs are worth paying.","Because the semantic graphs come from an English-oriented parser, the reported AMR gains for Romance languages and Chinese could partly reflect added structure rather than faithful meaning; a study using gold multilingual AMR graphs would isolate the semantic contribution."],"forward_implications":["For French, Spanish, Portuguese, and Chinese, every infused model family (T5, BART, and mT5) outperformed its raw-input counterpart on exact-set match and execution accuracy.","AMR infusion alone is the strongest single signal in most comparisons, and combining syntax with AMR is best for BART and for Chinese mT5.","Models trained with infused prompts for 32 epochs reach accuracies close to those of uninfused models trained for 128 epochs, so the linguistic signal cuts training cost.","On English Spider the same infusion produces little or negative change, indicating the benefit concentrates in low-resource settings."],"supporting_citations":[{"why":"Supplies the Spider benchmark, the dataset underlying all translated corpora and the evaluation.","marker":"Y u et al. 2019"},{"why":"Provides the French, Portuguese, and Spanish Spider translations used in training and the previous state-of-the-art results that the paper surpasses.","marker":"Jose and Cozman 2023"},{"why":"Supplies the Chinese CSpider translation used in the Chinese experiments.","marker":"Min, Shi, and Zhang 2019"},{"why":"Defines the schema-pruning and skeleton-decoding NL2SQL pipeline that the prompt infusion wraps around.","marker":"Li et al. 2023"},{"why":"Defines the dependency-tree serialization format the paper adopts for syntax infusion.","marker":"Li et al. 2021"},{"why":"Defines PropBank notation used by the AMR graphs, anchoring the semantic representation.","marker":"Palmer, Gildea, and Kingsbury 2005"},{"why":"Defines the T5 architecture used as a base model in the main experiments.","marker":"Raffel et al. 2023"},{"why":"Defines the BART architecture used as a second base model family.","marker":"Lewis et al. 2019"},{"why":"Defines the flattened SQL intermediate representation used by the 3-billion-parameter T5 models that set the new state of the art.","marker":"Gan et al. 2021"},{"why":"Defines mT5, the multilingual model used for Chinese experiments where base T5 fails.","marker":"Xue et al. 2021"}],"fun_headline_variants":["Syntax and AMR boost low-resource text-to-SQL","Dependency tags and AMR graphs improve NL2SQL","Linguistic clues in prompts lift SQL translation","AMR graphs and grammar push past prior best","Grammar-infused prompts beat previous top for NL2SQL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's gains rest on the assumption that the English-oriented semantic parser it uses produces meaningful AMR graphs for French, Spanish, Portuguese, and Chinese input, and that any observed improvement comes from that semantic signal rather than from added tokens or noise.","fun_headline_variants_meta":{"raw":{"variants":["Syntax and AMR boost low-resource text-to-SQL","Dependency tags and AMR graphs improve NL2SQL","Linguistic clues in prompts lift SQL translation","AMR graphs and grammar push past prior best","Grammar-infused prompts beat previous top for NL2SQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1396,"prompt_tokens":870,"completion_tokens":526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":486,"tokens_out":526,"duration_ms":5658,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:59:45.346244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment that replaces each AMR serialization with a scrambled or randomly generated bracketed string of the same length would settle the semantic claim: if scrambled graphs preserve the accuracy gains, the semantic content is not what is doing the work.","supporting_citations":[{"cited_title":"A.; and Cozman, F","cited_arxiv_id":null,"evidence_quote":"Provides the French, Portuguese, and Spanish Spider translations used in training and the previous state-of-the-art results that the paper surpasses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Chinese CSpider translation used in the Chinese experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines PropBank notation used by the AMR graphs, anchoring the semantic representation."}],"review_version":1}