{"id":"acaddf0f-4b0f-4d34-9f44-9af7ab3133ae","arxiv_id":"2412.10434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-agent LLM framework with schema extraction, fine-tuned query generation, and execution-error feedback outperforms prior NL2GQL systems on both a new nGQL dataset and the existing SpCQL benchmark.","lead":"This paper presents a three-agent LLM pipeline that translates natural language questions into graph database queries, and a new financial-domain benchmark dataset for training and testing such systems. The framework improves accuracy on two NL2GQL benchmarks, and the dataset is a new public resource for a field that currently has few options.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Refiner agent's correctness trigger is unspecified; if it uses gold answers during inference, the reported SOTA comparison is not valid.","rationale":"The reader's conditional verdict is reasonable, and their focus on the Preprocessor's schema-extraction reliability is well supported: Table IV shows that replacing the golden schema with an erroneous schema drops EM from 91.46 to 20.34 on StockGQL, and the error analysis attributes 46% of failures to schema errors. However, the paper also claims that the Refiner can detect schema errors and trigger re-extraction, and it reports that regeneration contributes to the final result. The mechanism for detecting a semantically wrong but executable query is never specified. The Figure 3 flow and the Figure 7 prompt both suggest that the system may decide 'incorrect' using gold-comparison feedback. This would not be a flaw if the evaluation were framed as an upper bound, but the paper presents the numbers as a practical state-of-the-art system. The missing detail is therefore not cosmetic: it determines whether the reported improvements over baselines are legitimate. My proposed check targets this directly. Pending that check, I would keep the reader's conditional verdict, because the concern is substantial but not yet confirmed. If the oracle is confirmed, the correct outcome would be a rejection of the comparative claims; if only execution errors are used, the remaining weaknesses are the ones the reader already identified.","tokens_in":18188,"tokens_out":9690,"duration_ms":103701,"concrete_test":"Obtain the framework code or have the authors instrument the pipeline for Table II. Record for each test query whether the Refiner was invoked and what trigger condition fired: (a) database execution error, (b) empty result, (c) result compared against gold answer, or (d) other. Then re-run StockGQL and SpCQL with the Refiner/regeneration loop active only on signals available without ground truth (a and b). If EM/EX drops materially (e.g., to the 'Without Refiner' or 'Without Regeneration' levels), the reported SOTA is contingent on an oracle. Also report McNemar p-values for the headline gaps against the best baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing uncertainty is not just the Preprocessor's schema accuracy, but the feedback signal that drives the Refiner and the re-extraction loop. The flow diagram in Figure 3 contains a 'Right?' decision after GQL execution, and the Refiner prompt (Figure 7) instructs the model to fix queries whose 'returned output was empty or incorrect.' At inference time, a deployed system cannot know a returned result is 'incorrect' without a gold answer; the only inherent signals are database execution errors. The paper never states whether the Table II evaluation invokes the Refiner using gold-comparison feedback. If it does, the main experiments are not a fair comparison with the baselines, and the claimed SOTA depends on test labels. If it does not, the paper's claim that the Refiner detects related-schema errors and triggers regeneration (Section IV-C and the 'Without Regeneration' ablation) is unexplained, since a successfully executed query that returns the wrong answer yields no error information. This ambiguity directly affects the central claim: the gap over the best baseline is 2.45 EM on StockGQL and 4.93 EM on SpCQL, and the Refiner/regeneration steps are responsible for part of that gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NAT-NL2GQL, a multi-agent framework for translating natural language questions into graph query language (GQL). It consists of a Preprocessor agent (NER, entity alignment, linking completion, related-schema revision, and question rewriting), a Generator agent (a LoRA fine-tuned LLM), and a Refiner agent that iteratively rewrites GQL or re-runs preprocessing based on execution results. The authors also construct StockGQL, a Chinese financial-domain nGQL dataset, and evaluate the framework on StockGQL and SpCQL using exact-match (EM) and execution (EX) accuracy. They report state-of-the-art results on both datasets and ablation studies indicating each component contributes to performance.","tokens_in":18447,"tokens_out":6778,"duration_ms":65741,"significance":"If the results are robust, the paper makes a useful contribution: a new open dataset for nGQL, a modular multi-agent architecture, and strong empirical gains on two datasets against several strong baselines including ICL and fine-tuned LLMs. The ablation study and the schema-impact analysis are informative. However, the current manuscript does not establish the headline claim with sufficient rigor: the Refiner's feedback signal at inference is not specified, the main table contains an uninterpretable row, and no uncertainty quantification is provided. These issues need to be resolved before the claimed state-of-the-art can be accepted.","major_comments":[{"comment":"The trigger for the Refiner agent's intervention is not specified at inference time. Figure 3 shows a 'Right?' decision after GQL execution, and the prompt in Figure 7 instructs the Refiner to fix queries whose 'returned output was empty or incorrect.' A deployed system has access only to execution errors or empty outputs; it cannot determine that a non-empty output is 'incorrect' without a gold answer. The manuscript never states whether the Table II evaluation uses gold-comparison feedback to decide whether to invoke the Refiner. If it does, the comparison with baselines is not a fair head-to-head; if it does not, the mechanism behind the 'Without Regeneration' ablation (Table VI) and the claimed detection of related-schema errors is unexplained. Please specify the exact feedback signal used in the main experiments and, if gold labels are used, also report results with execution-only feedback.","section":"IV-C, Figure 3, Figure 7"},{"comment":"Table II contains a row labeled 'Others' approach' that reads 'SpCQL 1.4 1.8 2.3 2.6' with no method name, backbone, or citation; this row cannot be interpreted or reproduced. In addition, all results in the table appear to be single runs with no error bars or significance tests. Because LoRA fine-tuning and ChatGPT-4o prompting are stochastic, the reported improvements of 2.45 EM/2.12 EX on StockGQL and 4.93 EM/5.63 EX on SpCQL may be within run-to-run variance. Please correct or remove the malformed baseline row and report means and standard deviations over multiple runs for at least the main comparison.","section":"Table II"},{"comment":"The SpCQL experimental setup is underspecified in Section V-A. The paper does not state which train/dev/test split of SpCQL is used, the size of the fine-tuning set, the LoRA hyperparameters, or whether golden related schemas are used during SpCQL training. It also does not explain how the nGQL-oriented prompts and syntax conventions (e.g., '==' filters) are adapted to Cypher, beyond a brief mention of prompt adjustments. Without these details, the SpCQL results cannot be reproduced and the comparison across datasets is difficult to interpret.","section":"V-A"},{"comment":"Error analysis (Figure 9) shows that 46% of failures on StockGQL are related-schema errors, and Table IV shows that an erroneous related schema reduces EM to 20.34%. Since the Preprocessor agent uses an unfine-tuned ChatGPT-4o for schema revision (Section IV-A), the overall accuracy of the framework is heavily dependent on an unvalidated, stochastic component. The paper should discuss this dependency explicitly and, ideally, report related-schema extraction accuracy on SpCQL as well, to show that the conclusion generalizes beyond StockGQL.","section":"V-C, Table IV"}],"minor_comments":[{"comment":"Algorithm cross-references are inconsistent: the text refers to 'Algorithm 7' and 'algorithm shown in 15', but the paper contains Algorithms 1 and 2.","section":"III, IV-A"},{"comment":"In Section VI-A, the sentence 'the results on the two evaluation metrics, EX and EX, were not satisfactory' should read 'EM and EX'.","section":"VI-A"},{"comment":"The caption of Table II says the red number in parentheses indicates improvement, but the table shows arrows with numbers and no parentheses; Table VI uses parentheses. Unify the notation.","section":"Table II"},{"comment":"The abstract states that StockGQL 'will be released publicly', while Section III provides an active GitHub URL; make these statements consistent.","section":"Abstract, III"},{"comment":"Figure 10 reports per-hop EM/EX without error bars or counts; given the small number of 'Others' test items (9 in Table I), the 66.67% value should be interpreted cautiously and the figure would benefit from showing counts.","section":"Figure 10"}],"recommendation":"major_revision","confidential_remarks":"I see a publishable core once the feedback-signal question is resolved and the empirical claims are made reproducible. The authors' prior work appears as a baseline, which is acceptable as long as the comparison is clean; the main risks are the potential use of gold labels in the Refiner loop and the lack of variance estimates. I would not reject on the basis of disagreement with consensus; the multi-agent design is reasonable and the dataset is a concrete community resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nTwo things to know. First, this is a workable three-agent pipeline for natural-language-to-graph-query (NL2GQL) that comes with a new public nGQL dataset, StockGQL, built from a financial graph. Second, the reported SOTA results look plausible, but a key detail about the Refiner's feedback signal is underspecified, and that could change the numbers.\n\nWhat's new: the Refiner agent chooses between rewriting the GQL directly and sending the error context back to the Preprocessor for re-extraction of the schema. That two-path recovery loop is a clean idea and goes beyond the streamlined schemes in Align-NL2GQL and R3-NL2GQL. StockGQL is the first nGQL dataset, covers up to 6-hop paths, and includes colloquial entity variants, which is genuinely useful. The ablations show each agent contributes, and the error analysis is honest: schema extraction is right about 86% of the time on StockGQL and is still the largest failure source.\n\nThe soft spots are about evidence quality rather than architecture. The main one: Figure 3 has a 'Right?' check after query execution, and the Refiner prompt says it handles cases where the output was 'empty or incorrect.' At inference, 'incorrect' cannot be known without a gold answer. The paper never states whether the evaluation uses gold-comparison feedback to trigger refinement. If it does, the comparison with baselines is unfair; if it doesn't, the claim that the Refiner detects schema errors and triggers regeneration is unexplained. This must be clarified before the SOTA claim is credible.\n\nMinor issues: no error bars or repeated runs, which matters because LLM calls are stochastic; a malformed 'Others' approach' row in Table II; and the SpCQL fine-tuning setup is under-specified (which training data, what syntax adjustments). None of these are fatal individually.\n\nWho this is for: anyone working on NL2GQL or applying multi-agent LLM patterns to database tasks. It deserves a serious referee, because the dataset alone is a contribution and the framework is coherent. The Refiner clarification and variance reporting should be required before acceptance.\n\nRecommendation: engage with it via peer review, but make the gold-feedback question the first thing the reviewers ask.","headline":"A plausible multi-agent NL2GQL framework with a new nGQL dataset, but the Refiner's gold-feedback ambiguity and missing error bars mean the SOTA claim needs verification.","tokens_in":18956,"tokens_out":3316,"would_cite":true,"duration_ms":32084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting natural-language-to-graph-query translation into schema preprocessing, fine-tuned generation, and execution-error refinement yields new best results on two benchmarks: 85.44% exact-match on StockGQL and 59.99% on SpCQL.","keywords":["NL2GQL","graph query language","multi-agent framework","schema linking","nGQL","Cypher","graph database","StockGQL"],"falsifier":"A direct test is to rerun the framework on StockGQL with the related schema replaced by the oracle labels (golden schema) instead of the Preprocessor's predictions; the paper reports 91.46% versus 85.44% EM, so an independent replication finding a much smaller gap would indicate the claimed schema bottleneck is not the main driver. Conversely, injecting randomly corrupted schemas and observing little accuracy loss would contradict the paper's claim that related-schema accuracy controls end-to-end performance.","tokens_in":18029,"feed_emoji":"🤖","tokens_out":7471,"duration_ms":66470,"temperature":0.7,"pith_summary":"Translating a natural-language question into a graph query is hard because the system must first find the small piece of the database schema the question refers to; if that schema is wrong, every later step is likely wrong. The paper claims that a three-agent loop—a Preprocessor that extracts and cleans the related schema, a fine-tuned Generator that writes the query, and a Refiner that reacts to execution errors—breaks this error chain and beats existing NL2GQL methods. On the paper's new StockGQL benchmark the framework reaches 85.44% exact-match and 86.25% execution accuracy, and on the existing SpCQL benchmark it reaches 59.99% and 58.69%, both above the best prior baselines. The paper also contributes StockGQL, a multi-hop nGQL dataset built from a financial graph database.","feed_headline":"Three-agent framework tops graph-query translation benchmarks","feed_subtitle":"A three-way split of schema, generation, and refinement lifts exact-match accuracy to 85.4% and 60.0%.","key_machinery":"The load-bearing object is the related schema: the small subgraph of nodes, edges, and properties that a question actually needs. Its correctness is what lets a compact prompt reach a fine-tuned generator, and the paper's own measurements show the effect: with the golden related schema the generator hits 91.46% EM on StockGQL, with an erroneous related schema it falls to 20.34%, and with the full schema it sits in between at 82.67%. The Preprocessor produces this schema through LLM-based NER, locality-sensitive-hashing entity alignment, a linking-completion algorithm, and a ChatGPT-4o redundancy-filtering step; the Refiner's decision to return to the Preprocessor is specifically a mechanism for repairing a bad related schema.","core_discovery":"The central claim is that NL2GQL should be treated as a collaborative, iterative multi-agent task rather than a single model pass. The Preprocessor agent performs named-entity recognition, entity alignment, path linking, and related-schema revision; the Generator agent is a LoRA-fine-tuned LLM that produces GQL from the rewritten question plus the related schema; and the Refiner agent reads the execution error and either rewrites the query or sends the failing information back to the Preprocessor to restart, up to three times. This arrangement yields new best results on both datasets, and the ablation shows each agent contributes; swapping the fine-tuned Generator for in-context ChatGPT-4o is the most damaging change, dropping EM from 85.44% to 50.04% on StockGQL.","pith_inferences":["The three-agent split (context extraction, generation, execution-feedback repair) is a general recipe that likely transfers to other structured query tasks such as Text2SQL or SPARQL, but the paper does not test those settings.","Because schema errors account for 46% of failures, a fine-tuned or retrieval-based schema linker could plausibly raise end-to-end accuracy more than any improvement to the generator; this is a testable extension, not a paper claim.","The Refiner's 'Info Error' path effectively turns execution failure into a new training signal; one could measure its ceiling by giving the Refiner an oracle that always knows whether the schema is wrong.","The reported golden-schema ceiling of 91.46% EM suggests that even a perfect schema extractor would leave roughly 9% of errors, so future work will need question-comprehension improvements as well."],"forward_implications":["NL2GQL systems should budget their effort on schema linking, since related-schema accuracy is the largest single lever on final query accuracy.","A fine-tuned generator plus execution-feedback refinement can beat strong in-context LLM prompting, so training on domain GQL pairs still pays off.","Iterative refinement with a hard iteration cap gives measurable gains without unbounded cost; removing the regeneration path alone costs 1.23 EM points.","The StockGQL dataset gives the community a public multi-hop nGQL benchmark with 63% of queries involving more than two hops.","The same three-agent structure transfers across GQL dialects, since it improves results on both nGQL (StockGQL) and Cypher (SpCQL)."],"supporting_citations":[{"why":"Supplies the SpCQL dataset and the EM/EX evaluation metrics used for the general-domain comparison.","marker":"[6]"},{"why":"Provides the Align-NL2GQL baseline and the domain-specific dataset construction and schema-extraction approach this work builds on.","marker":"[7]"},{"why":"Provides the R3-NL2GQL baseline, a model-coordination method that is among the best competitors the framework must beat.","marker":"[8]"},{"why":"The self-instruct method used to generate the masked NL-GQL training pairs for StockGQL.","marker":"[19]"},{"why":"Locality-sensitive hashing, used by the Preprocessor agent to align query entities with graph database entities.","marker":"[23]"},{"why":"LoRA, the parameter-efficient fine-tuning method used to train the Generator agent on NL-GQL data.","marker":"[28]"}],"fun_headline_variants":["Three-agent pipeline sets new bar for NL-to-graph-query translation","Multi-agent framework beats baselines in graph query generation","Agent team outperforms single-model NL2GQL translation","Collaborative agents lift graph-query translation accuracy","Preprocess-generate-refine: multi-agent NL2GQL wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's accuracy rests on the Preprocessor agent extracting the correct related schema using an unfine-tuned ChatGPT-4o; the paper's error analysis reports that schema errors cause 46% of all failures, so if that extraction step goes wrong the Generator and Refiner cannot reliably recover.","fun_headline_variants_meta":{"raw":{"variants":["Three-agent pipeline sets new bar for NL-to-graph-query translation","Multi-agent framework beats baselines in graph query generation","Agent team outperforms single-model NL2GQL translation","Collaborative agents lift graph-query translation accuracy","Preprocess-generate-refine: multi-agent NL2GQL wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1506,"prompt_tokens":1056,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":672,"tokens_out":450,"duration_ms":4462,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:13:24.189240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to rerun the framework on StockGQL with the related schema replaced by the oracle labels (golden schema) instead of the Preprocessor's predictions; the paper reports 91.46% versus 85.44% EM, so an independent replication finding a much smaller gap would indicate the claimed schema bottleneck is not the main driver. Conversely, injecting randomly corrupted schemas and observing little accuracy loss would contradict the paper's claim that related-schema accuracy controls end-to-end performance.","supporting_citations":[{"cited_title":"Aligning large language models to a domain-specific graph database for nl2gql,","cited_arxiv_id":null,"evidence_quote":"Provides the Align-NL2GQL baseline and the domain-specific dataset construction and schema-extraction approach this work builds on."},{"cited_title":"r3-NL2GQL: A model coordination and knowledge graph alignment approach for NL2GQL,","cited_arxiv_id":null,"evidence_quote":"Provides the R3-NL2GQL baseline, a model-coordination method that is among the best competitors the framework must beat."},{"cited_title":"Locality-sensitive hashing scheme based on p-stable distributions,","cited_arxiv_id":null,"evidence_quote":"Locality-sensitive hashing, used by the Preprocessor agent to align query entities with graph database entities."}],"review_version":1}