REVIEW 2 major objections 6 minor 16 references
Multi-turn Natural Language to Graph Query Language Translation
T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper introduces MTGQL, the first multi-turn dataset for natural-language-to-graph-query translation, and reports that a dependency-aware baseline that rewrites the current question against dialogue history and refines by execution…
desk verdict First multi-turn NL2GQL dataset, but the benchmark numbers depend on unverified ground truth and a baseline that reuses the dataset's own construction machinery. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the dependency-aware Context Manager together with six expansion patterns. The Context Manager stores previous questions, their GQLs, execution answers, entities, and relations, and uses a weighted selection algorithm to choose one of six patterns for each new turn: attribute follow-up, temporal shift, relation extension, same-type entity comparison, aggregation calculation, and conditional filtering. This machinery forces follow-up questions to be interdependent, and it also drives the DA baseline, which reformulates the raw question into an explicit, de-referenced form and extracts only the relevant sub-schema before generating the query. The second mechanism is the GQL Validator and Optimizer: syntax is checked by executing the query against the graph database, and semantic correctness is checked by regenerating the question from the generated GQL and measuring vector-embedding similarity, with failed queries revised up to three times. Together these mechanisms are what make the dataset grounded in graph data, context-dependent across turns, and diverse in question type.
What would settle it
The claim that MTGQL is a reliable first multi-turn benchmark would be falsified by a full manual audit of a random sample of test dialogues that finds a substantial fraction of GQLs which execute successfully but clearly answer a different question than the annotated one — precisely the class of error the embedding-based reverse-generation check could systematically miss.
Extended reading notes
Core claim
The paper claims that multi-turn NL2GQL is a distinct task from single-turn NL2GQL and that the community lacked the data to study it. Its discovery is twofold: an automatic dependency-aware construction framework can produce a multi-turn dataset at scale, and the resulting MTGQL benchmark exposes a large gap between turn-level and dialogue-level accuracy. The construction pipeline chains a Context Manager, Question Generator, GQL Generator, GQL Validator and Optimizer, and Dataset Filter; every follow-up question is generated from one of six expansion patterns tied to entities and relations already in the dialogue, then validated by executing the GQL against the database and by reverse-generating the question from the GQL and comparing embedding similarity. Among the baselines, the dependency-aware method, which reformulates the current question against structured history and refines the GQL from execution feedback, is the most effective: turn-level exact match reaches 68.45% while dialogue-level overall exact match is only 40.60%. This is the sense in which MTGQL is claimed to be the first benchmark for multi-turn NL2GQL.
Load-bearing premise
The dataset's reliability rests on the automatic pipeline: the paper assumes that executing each generated GQL on the database and comparing reverse-generated questions by vector-embedding similarity catches semantic errors, even though only 600 of the 4,500 dialogues were checked by human raters.
Editorial extensions
If this is right
- MTGQL becomes a public, Chinese financial benchmark with 4,500 multi-turn dialogues, so single-turn methods can no longer claim to cover practical graph-database interaction without being tested on it.
- The reported scores imply that current LLMs handle early turns well (round-1 execution accuracy 82.88%) but drop to about 31% by round 5 or later, making dialogue-level context the main bottleneck.
- Fine-tuning on MTGQL transfers to the single-turn StockGQL dataset and improves joint training by roughly 4.5 percentage points, suggesting multi-turn data is a useful complement even for single-turn NL2GQL.
- The six expansion patterns and the DA baseline give future work a common template for generating multi-turn graph-query data and for comparing context-aware methods.
Reading between the lines
- The embedding-based semantic validator is the most fragile link: comparing vector embeddings of the original and reverse-generated question is a weak proxy for true semantic equivalence, so mismatches that survive filtering would inflate the benchmark's apparent quality; a full human audit of a held-out sample would test this directly.
- The six expansion patterns are a reusable typology — attribute, time, relation, comparison, aggregation, and filtering dependencies probably cover much of what users do in other graph domains, so the construction approach could transfer if the schema prompts and entity placeholders are adapted.
- Because AEM (40.60%) is much lower than EM (68.45%), a system that gets most individual turns right still fails most complete dialogues; future work should optimize for dialogue-level consistency rather than turn-level accuracy alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes an LLM-driven framework for automatically constructing multi-turn NL2GQL datasets, consisting of a context manager, question generator, GQL generator, validator/optimizer, and dataset filter. The authors apply it to a Chinese financial NebulaGraph database and build MTGQL, reportedly the first multi-turn NL2GQL dataset, with 4,500 dialogues, 29,196 nGQL statements, split 3,000/500/1,000. They define turn-level EM/EX and dialogue-level AEM/AEX metrics, evaluate four baselines (ICL-AS, RSE, FT-AS, DA) on three LLMs, and report DA with Qwen2.5-14B-Instruct as best (EM 68.45%, AEM 40.60%, EX 65.39%, AEX 38.30%). The appendix includes a human evaluation of 600 dialogues, pattern-wise results, an error analysis, and cross-dataset transfer experiments.
Significance. If the dataset is reliable, this is a useful community resource: it is the first Chinese multi-turn NL2GQL corpus in the financial domain, and the construction framework is a concrete recipe that others can adapt. The round-wise and pattern-wise analyses (Tables 6 and 7) provide evidence that context dependency is the main difficulty, and the cross-dataset transfer results in Appendix 9.6 are a genuine attempt to show utility beyond MTGQL. The manuscript is also honest about limitations, conceding that execution accuracy is not semantic accuracy and that the pipeline relies on heuristic rules. However, the central contribution is a benchmark, and the benchmark's value is only as good as its ground-truth GQL labels. The current evidence for label correctness is thin: automatic validation uses an embedding-similarity proxy, and human review covers only 600 of 4,500 dialogues with a protocol that does not check per-turn GQL correctness. The reported numbers and the 'first benchmark' claim are therefore conditional on an unverified assumption.
major comments (2)
- [Section 4.5, Section 5.2, Appendix 9.7] The gold-standard GQL labels in MTGQL are not sufficiently verified. Semantic Validation in Section 4.5 only checks the embedding similarity between the original question and a question reverse-generated from the candidate GQL using all-MiniLM-L6-v2; this is a weak equivalence proxy because it does not verify entity identities, relations, temporal constraints, or dropped predicates, and execution success only demonstrates that some result exists. The human evaluation in Section 5.2 and Appendix 9.7 covers 600 of 4,500 dialogues, and its 'Semantic Accuracy' dimension is defined as alignment of questions with the schema, not as correctness of the paired GQL with respect to the intended question. The Limitations section itself concedes that execution correctness may not capture semantic correctness. As a result, the reported EM/AEM/EX/AEX numbers in Table 5 and the dataset-quality claims are not yet supported. The authors should either human-verify the test set (or a substantially larger, statistically justified sample) for per-turn GQL correctness, report the measured error rate, and release the verified subset, or substantially weaken the benchmark claims.
- [Section 6.1, Appendix 9.8] The DA baseline is described as an adaptation of the same Dependency-aware Multi-turn Dataset Construction Framework used to build the dataset, with the same context-manager reformulation, sub-schema extraction, and GQL refiner components. This means the reported advantage of DA over ICL-AS, RSE, and FT-AS may partly reflect access to the construction pipeline's internal machinery, which future users of the benchmark will not have. The paper should specify exactly which components are shared, quantify the gain from each component (e.g., ablation removing reformulation, sub-schema extraction, or refinement), and clarify whether DA is meant as a practical baseline or as an upper-bound oracle. Without this, the conclusion that 'DA is the most effective method' is overstated.
minor comments (6)
- [Section 1 (Contributions)] The bullet list says 'Backmarch methods'; this should read 'Benchmark methods'.
- [Table 5 and Section 6.1] The second method is introduced as RSE in Section 6.1 but labeled 'RES' in Table 5; the labels should be made consistent.
- [Appendix 9.4.4, 9.4.5, Tables 8 and 9] The keyword-statistics and query-type analyses are described as being on 'StockGQL' and 'FinGQL', respectively, but the dataset under analysis is MTGQL; the headings, text, and captions should be corrected.
- [Table 2 and Section 5.2] Section 5.2 and Appendix 9.7 list four evaluation dimensions (coherence, question diversity, coverage, semantic accuracy), but Table 2 reports only three; the coverage scores are missing and should be added.
- [Appendix 9.4.2] The '1,000 multi-turn dialogue samples' experiment is under-specified; the authors should state the prompt, backbone model, filtering conditions, and whether these samples were generated with the same framework.
- [Section 4.4] The GQL Generator is said to be a fine-tuned LLM, but the paper does not state what data the LLM was fine-tuned on; if the fine-tuning data was itself produced by the framework, this would affect the independence of the validation loop and should be clarified.
Circularity Check
DA baseline is the inference-time half of the same dependency-aware framework that generated MTGQL, so its benchmark lead is partly self-consistency rather than independent generalization.
-
self definitional
[Section 6.1 (Dependency-aware method), with Sections 4.2, 4.4, 4.5 and Appendix 9.8]
"Section 6.1: 'We adapt the Dependency-aware Multi-turn Dataset Construction Framework with minor modifications ... The adapted method comprises three key modules: a Context Manager, a GQL Generator, and a GQL Refiner.' Section 4.4: 'responsible for generating the corresponding GQL based on the schema of G.' Appendix 9.8: the DA Context Manager 'reformulates the user question into a more explicit, context-independent version' and 'retrieves a relevant sub-schema.'"
MTGQL's labeled GQLs are outputs of the dependency-aware construction framework (Context Manager, GQL Generator, Validator/Optimizer), and DA is explicitly 'an adaptation' of that same framework, reusing the same three modules under slightly different names and fine-tuning its GQL Generator on MTGQL. The construction framework first rewrites the raw user question into a formal, disambiguated 'Complete Question' and supplies the relevant schema; DA's Context Manager performs the same reformulation and sub-schema retrieval at inference. DA therefore feeds its fine-tuned GQL Generator exactly the input representation from which the MTGQL labels were originally generated.
full rationale
The main circular step is the DA baseline reusing the dataset-construction machinery. The dataset's automatic semantic validation (Section 4.5, embedding-similarity reverse generation) and the fact that only 600 of 4,500 dialogues were human-reviewed (Section 5.2, Appendix 9.7) are genuine threats to ground-truth correctness, but they are correctness risks, not derivation circularity; they do not by themselves make the reported numbers circular. The self-citations to Liang et al. (2024a,b) supply the generator and validator recipes, and the paper also reports independent human evaluation and a cross-dataset transfer experiment (Appendix 9.6), so the dataset claim is not solely a self-citation chain. However, because the proposed best method is the same framework that created the labels, the central performance comparison in Table 5 is partly self-consistent by construction; I score this as partial circularity (6) rather than full circularity (8-10).
Assumptions & free parameters
free parameters (7)
- Embedding cosine similarity threshold =
0.6
- Masked GQL duplicate threshold =
>3 identical
- Dialogue turns range =
5 to 8
- Initial expansion pattern weight =
1/6, halved after use
- Entity re-reference weight increase =
+1/4
- ICL demonstrations K =
4
- Maximum GQL optimization retries =
3
assumptions (5)
- domain assumption LLMs, when prompted with a full graph schema, can generate syntactically valid and semantically appropriate GQL queries for generated questions.
- domain assumption GQL execution results on the graph database are a reliable oracle for query correctness.
- domain assumption Sentence-BERT embeddings (all-MiniLM-L6-v2) capture the semantic closeness of questions well enough for filtering and validation.
- ad hoc to paper The six hand-designed expansion patterns (P1-P6) adequately cover realistic multi-turn user intents.
- domain assumption The financial graph database schema is complete and accurate for grounding all generated questions.
Cite this review
Pith. "Pith review of Multi-turn Natural Language to Graph Query Language Translation." pith.science (2026). https://pith.science/paper/K2IAY4CX
@misc{pith2026250801871,
author = {Pith},
title = {Pith review of: Multi-turn Natural Language to Graph Query Language Translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/K2IAY4CX}},
note = {Machine review of arXiv:2508.01871}
}
read the original abstract
In recent years, research on transforming natural language into graph query language (NL2GQL) has been increasing. Most existing methods focus on single-turn transformation from NL to GQL. In practical applications, user interactions with graph databases are typically multi-turn, dynamic, and context-dependent. While single-turn methods can handle straightforward queries, more complex scenarios often require users to iteratively adjust their queries, investigate the connections between entities, or request additional details across multiple dialogue turns. Research focused on single-turn conversion fails to effectively address multi-turn dialogues and complex context dependencies. Additionally, the scarcity of high-quality multi-turn NL2GQL datasets further hinders the progress of this field. To address this challenge, we propose an automated method for constructing multi-turn NL2GQL datasets based on Large Language Models (LLMs) , and apply this method to develop the MTGQL dataset, which is constructed from a financial market graph database and will be publicly released for future research. Moreover, we propose three types of baseline methods to assess the effectiveness of multi-turn NL2GQL translation, thereby laying a solid foundation for future research.
Figures
Reference graph
Works this paper leans on
-
[1]
Generate a question based on the schema and dialogue context, ensuring it is contextually relevant and logically continues the conversation. The question should be conversational in style, incorporating ellipses, omissions, and vague expressions wherever appropriate
-
[2]
Use placeholders for entities, such as:[s] for stock,[c] for chairman,[h] for stockholder,[t] for trade, [p] for public offering fund,[f] for fund manager,[i] for industry,[d] for time, and[m] for numbers
-
[3]
If the dialogue history is empty, create an opening question. If there is existing dialogue, generate a follow-up question that aligns with the provided question expansion pattern
-
[4]
Generate the raw question in a conversational style, incorporating relevant references
-
[5]
Generate the formal question based on the raw question. The formal question should be a disam- biguated version of the raw question, clarified and free of placeholders or references. Input:
-
[9]
Schema Information: {SCHEMA}
-
[10]
Dialogue History: {DIALOGUE_HISTORY}
-
[11]
Question Expansion Pattern: {QUESTION_EXPANDING_PATTERN} Output: Provide the generated raw question after "Question" and the formal question after "Complete Question" directly. Question: Complete Question: Figure 3: The prompt for question generation. The results show that the execution accuracy (EX) for single-turn queries was only39.8%, while the overal...
work page 1923
Show all 16 references
-
[12]
This is especially problematic in ICL-AS and FT-AS, which must reason over the en- tire schema without contextual focus
Schema Selection Errors (26%)These errors arise when the model selects incorrect or incom- plete schema elements (i.e., node or edge types) for the current turn. This is especially problematic in ICL-AS and FT-AS, which must reason over the en- tire schema without contextual f...
-
[13]
What about its sub- sidiaries?
Contextual Understanding Failures (32%) These include failures where the model misun- derstands the dependencies between the current utterance and the previous turns. For instance, co-reference resolution (e.g., “What about its sub- sidiaries?”) or omitted subject/object refer...
-
[14]
Common examples include incorrect filtering conditions, missing relation con- straints, or reversed edges
Logical Form Generation Errors (18%) These involve syntactically valid but semantically incorrect GQL outputs. Common examples include incorrect filtering conditions, missing relation con- straints, or reversed edges. The RSE method par- ticularly struggles here when the relat...
-
[15]
How about their latest in- vestment?
Ambiguity and Underspecification (14%) These errors stem from under-specified questions, where even humans may interpret multiple valid GQLs. For example, “How about their latest in- vestment?” may refer to different temporal orders depending on context. Models often make arbi...
-
[16]
The DA method mitigates this partially using its GQL Refiner module, but residual issues persist due to imperfect execution feedback alignment
Execution-based Errors (10%)Some errors only become evident after query execution, such as returning empty results due to overly specific filters or semantic mismatches. The DA method mitigates this partially using its GQL Refiner module, but residual issues persist due to imp...
-
[191]
FRIEND_OF
Springer. Yue Zhou, Chenlu Guo, Xu Wang, Yi Chang, and Yuan Wu. 2024a. A survey on data augmentation in large model era.Preprint, arXiv:2401.15422. Yuhang Zhou, Yu He, Siyu Tian, Yuchen Ni, Zhangyue Yin, Xiang Liu, Chuanjun Ji, Sen Liu, Xipeng Qiu, Guangnan Ye, and Hongfeng Ch...
2024 arXiv
-
[2023]
InAus- tralasian Joint Conference on Artificial Intelligence, pages 120–132
Cyspider: A neural semantic parsing corpus with baseline models for property graphs. InAus- tralasian Joint Conference on Artificial Intelligence, pages 120–132. Springer. Ziyu Zhao, Michael Stewart, Wei Liu, Tim French, and Melinda Hodkiewicz. 2022b. Natural language query fo...
-
[2024]
Yuanyuan Liang, Keren Tan, Tingyu Xie, Wenbiao Tao, Siyuan Wang, Yunshi Lan, and Weining Qian
Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows.arXiv preprint arXiv:2411.07763. Yuanyuan Liang, Keren Tan, Tingyu Xie, Wenbiao Tao, Siyuan Wang, Yunshi Lan, and Weining Qian. 2024a. Aligning large language models to a domain-specific grap...
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.