{"id":"5ecc9b8c-1368-42d7-8d0c-960c9b9e127f","arxiv_id":"2606.27742","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"KG2Cypher generates validated synthetic Text-Cypher pairs from existing KGs, trains LoRA models with class-conditioned schema prompting, and reports execution-result F1 gains to 0.950 and 0.92 plus 95.2% exact match in Korean enterprise settings.","lead":"KG2Cypher builds synthetic Text-Cypher training pairs directly from an enterprise knowledge graph by first creating executable Cypher queries from observed facts and then using LLMs to generate matching natural-language questions. The approach could lower the cost of creating natural-language interfaces for private company graphs that are otherwise expensive to label.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"LLM-judge validation may introduce artifacts that LoRA training overfits rather than generalizes","rationale":"The reader's weakest_assumption directly identifies the same circularity risk in the data-generation loop; the abstract supplies no counter-evidence (e.g., judge statistics or external test set), so the concern remains load-bearing for the central performance claims.","tokens_in":1748,"tokens_out":357,"duration_ms":13966,"concrete_test":"Release the exact LLM-judge prompt, temperature, and per-example agreement table (LLM vs. two human annotators) on a 200-pair sample; then re-train and evaluate on a fresh 100-pair set annotated only by humans who never saw the judge output. If execution-result F1 drops >0.08 on the human-only set, the original gains are at least partly judge artifacts.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The pipeline generates Text-Cypher pairs by (1) synthesizing Cypher from observed facts, (2) using an LLM to produce the NL question, and (3) filtering with an LLM judge plus human validation before SFT. The reported gains (0.806→0.950 F1 on broadcast queries, 0.70→0.92 on company queries) and 0.964 execution-result F1 therefore rest on the assumption that the judge produces unbiased, high-quality supervision whose artifacts do not leak into the test distribution. No section quantifies judge prompt, decision threshold, or agreement rate with humans; if the judge accepts semantically loose paraphrases or the test split shares the same generation process, the LoRA improvement can be explained by consistency with the judge rather than improved Cypher generation.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper presents KG2Cypher, a data-centric pipeline that constructs executable Cypher queries from observed enterprise KG facts, generates corresponding natural-language questions via LLMs, validates the resulting Text-Cypher pairs with an LLM judge plus human review, and uses the filtered pairs for class-conditioned LoRA SFT. The trained model is deployed with schema prompting and entity retrieval. On Korean enterprise broadcast-program and company queries, LoRA SFT raises execution-result F1 from 0.806 to 0.950 and from 0.70 to 0.92 respectively; in an 11-class setting the system reports 95.2% exact match, 99.9% execution rate, and 0.964 execution-result F1.","tokens_in":1908,"tokens_out":673,"duration_ms":26383,"significance":"If the reported gains prove robust to the validation process, the work offers a practical, low-annotation route to text-to-Cypher systems for private KGs, directly addressing the cost barrier noted in the abstract. The emphasis on execution-result metrics rather than surface match and the concrete before-and-after numbers constitute a strength; the absence of error bars or ablations, however, leaves the magnitude of improvement difficult to interpret.","major_comments":[{"comment":"The validation subsection (described in the pipeline overview) provides no prompt template, decision threshold, or inter-annotator agreement statistics for the LLM judge. Because the central claim is that LoRA SFT on judge-filtered pairs produces the observed F1 lifts (0.806→0.950 and 0.70→0.92), the lack of these quantities makes it impossible to rule out that the model is fitting to judge-specific artifacts rather than improving Cypher generation.","section":"Pipeline description / validation step"},{"comment":"Experiments section reports aggregate F1 and exact-match figures but contains no ablation that isolates the contribution of the LLM-judge filter versus the human-validation step, nor any error bars or statistical significance tests on the before-and-after deltas. These omissions are load-bearing for the claim that the pipeline reliably improves execution accuracy.","section":"Experiments / results tables"},{"comment":"No analysis is given of whether the test queries share the same synthetic generation process as the training pairs; if they do, the high execution rate (99.9%) and F1 (0.964) could be explained by distributional overlap rather than generalization.","section":"Evaluation setup"}],"minor_comments":[{"comment":"The abstract and results tables use “execution-result F1” without an explicit definition or reference to the precise matching criterion (e.g., whether partial result overlap is credited).","section":"Abstract and §4"},{"comment":"Figure captions and table footnotes do not indicate the number of human validators or the exact protocol used for the final human-validation pass.","section":"Figures 2–3 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a systems contribution rather than a theoretical one; confirm that the target journal routinely accepts pipeline papers with limited ablation depth."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on the validation details, experimental reporting, and evaluation setup. We address each major comment below and indicate planned revisions to strengthen the manuscript.","responses":[{"response":"We will add the LLM judge prompt template and decision threshold to the revised manuscript (in the validation subsection or an appendix) to support reproducibility. The training data additionally underwent a human validation step after the LLM judge, which we will emphasize as a safeguard against potential judge-specific artifacts. We did not compute inter-annotator agreement for the LLM judge and will acknowledge this as a limitation.","revision_made":"yes","referee_comment":"[Pipeline description / validation step] The validation subsection (described in the pipeline overview) provides no prompt template, decision threshold, or inter-annotator agreement statistics for the LLM judge. Because the central claim is that LoRA SFT on judge-filtered pairs produces the observed F1 lifts (0.806→0.950 and 0.70→0.92), the lack of these quantities makes it impossible to rule out that the model is fitting to judge-specific artifacts rather than improving Cypher generation."},{"response":"We will add error bars by reporting means and standard deviations from multiple training runs with different random seeds. A dedicated ablation isolating the LLM-judge filter from the subsequent human-validation step is not feasible within the revision timeline due to computational constraints; we will explicitly note this limitation and the role of the combined validation process in the revised text.","revision_made":"partial","referee_comment":"[Experiments / results tables] Experiments section reports aggregate F1 and exact-match figures but contains no ablation that isolates the contribution of the LLM-judge filter versus the human-validation step, nor any error bars or statistical significance tests on the before-and-after deltas. These omissions are load-bearing for the claim that the pipeline reliably improves execution accuracy."},{"response":"The test queries were collected from real enterprise user logs and are independent of the synthetic generation process used to create the training pairs. We will add an explicit statement and brief description of the test-query collection process in the evaluation setup section to clarify the absence of distributional overlap.","revision_made":"yes","referee_comment":"[Evaluation setup] No analysis is given of whether the test queries share the same synthetic generation process as the training pairs; if they do, the high execution rate (99.9%) and F1 (0.964) could be explained by distributional overlap rather than generalization."}],"tokens_in":1538,"tokens_out":548,"duration_ms":52541,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core contribution is a data pipeline that starts from observed graph facts to build executable Cypher, then uses LLMs to generate matching natural-language questions, followed by LLM-judge plus human filtering and LoRA fine-tuning. It adds class-conditioned schema prompting at inference time and tests this on Korean enterprise graphs with short, paraphrase-heavy queries.\n\nIt does a few things solidly. The before-and-after numbers are concrete: execution-result F1 rises from 0.806 to 0.950 on broadcast-program queries and from 0.70 to 0.92 on company queries. The 11-class setting reports 95.2% exact match, 99.9% execution rate, and 0.964 F1. The approach is explicitly data-centric and tailored to private KGs where labeled pairs are scarce, which matches real deployment constraints.\n\nThe main soft spot is the validation stage. The stress-test concern holds: without reported judge prompt details, agreement rates with humans, or checks that the test split is independent of the generation process, the LoRA gains could partly come from the model learning the judge's acceptance patterns rather than better Cypher logic. No error bars, no ablation on the filtering step, and no breakdown of how many pairs were rejected appear in the abstract. That makes the central claim plausible but not fully pinned down.\n\nThis paper is for practitioners who need to stand up text-to-Cypher interfaces on their own enterprise graphs, especially in non-English settings. Applied researchers working on LLM data synthesis for structured queries will get usable details. It is coherent on its own terms and shows honest engagement with the practical problem, so it deserves a serious referee even if the evaluation needs tightening on the judge artifacts.","headline":"KG2Cypher delivers a practical pipeline for synthetic Text-Cypher data on enterprise KGs with clear metric lifts, but the LLM-judge step leaves open whether gains reflect real generalization or just consistency with the validator.","tokens_in":2408,"tokens_out":443,"would_cite":false,"duration_ms":22516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"KG2Cypher pipeline generates validated Text-Cypher pairs from knowledge graphs to train accurate text-to-Cypher models for enterprise use.","keywords":["text-to-cypher","knowledge graphs","enterprise search","LoRA fine-tuning","supervised fine-tuning","data generation","natural language interfaces","Cypher queries"],"falsifier":"Evaluating the trained model on a new enterprise graph with unseen query classes or different schema structures and measuring if execution-result F1 drops significantly below 0.9.","tokens_in":2658,"feed_emoji":"📊","tokens_out":672,"duration_ms":21222,"temperature":0.7,"pith_summary":"The paper introduces a pipeline that starts with existing enterprise knowledge graphs to create training data for text-to-Cypher conversion. It builds Cypher queries from observed facts, has LLMs generate corresponding natural language questions, validates them with an LLM judge and humans, then uses the pairs for supervised fine-tuning with LoRA. This approach improves execution accuracy on Korean enterprise queries, reaching over 95% exact match in multi-class settings. The method addresses the high cost of building natural language interfaces for private graphs by leveraging data-centric generation rather than manual annotation.","feed_headline":"Pipeline turns knowledge graphs into 95% accurate text-to-Cypher systems","feed_subtitle":"Validated LLM-generated pairs and LoRA training boost execution F1 to 0.95 on enterprise queries.","key_machinery":"The data-centric pipeline that generates Text-Cypher pairs by deriving Cypher from graph facts and reverse-generating questions with LLMs, followed by validation and LoRA SFT training.","core_discovery":"KG2Cypher constructs an executable Cypher query from observed graph facts and uses LLMs to generate its associated natural-language question. The resulting Text-Cypher pairs are validated with an LLM judge and human validation, and are converted into candidate-aware SFT data. The trained generator is served with class-conditioned schema prompting, entity retrieval, and LoRA-based inference, achieving 95.2% exact match, 99.9% execution rate, and 0.964 execution-result F1 in an 11-class setting.","pith_inferences":["This could lower the barrier for companies to deploy natural language query interfaces on their internal graphs.","The validation step might be adaptable to other structured query languages like SQL.","Scaling the pipeline could enable zero-shot or few-shot adaptations to new graph schemas."],"forward_implications":["LoRA SFT raises execution-result F1 from 0.806 to 0.950 on broadcast-program queries.","Execution-result F1 improves from 0.70 to 0.92 on company queries.","In 11-class setting, achieves 95.2% exact match and 99.9% execution rate.","The system handles short search-style queries and schema paraphrases in Korean enterprise settings."],"fun_headline_variants":["Pipeline constructs Cypher from graph facts for text-to-Cypher","LLMs generate questions from validated Cypher query pairs","SFT data from candidate-aware pairs for LoRA inference","95.2% exact match on enterprise text-to-Cypher tasks","Class-conditioned prompting for 99.9% execution rate"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The LLM judge combined with human validation produces sufficiently unbiased Text-Cypher pairs that training improves real execution performance rather than fitting to validation artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Pipeline constructs Cypher from graph facts for text-to-Cypher","LLMs generate questions from validated Cypher query pairs","SFT data from candidate-aware pairs for LoRA inference","95.2% exact match on enterprise text-to-Cypher tasks","Class-conditioned prompting for 99.9% execution rate"]},"model":"grok-4.3","cost_usd":0.008317,"raw_usage":{"total_tokens":3701,"prompt_tokens":695,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":83165500,"prompt_tokens_details":{"text_tokens":695,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2923,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":695,"tokens_out":83,"duration_ms":22903,"temperature":1.0,"reasoning_tokens":2923,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T04:44:05.942912+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Evaluating the trained model on a new enterprise graph with unseen query classes or different schema structures and measuring if execution-result F1 drops significantly below 0.9.","supporting_citations":[],"review_version":1}