{"id":"19e75ad6-882e-48e7-8dcf-6821c6e08223","arxiv_id":"2607.28082","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selective Gate-triggered correction of LLM SPARQL outputs lifts MCQA query accuracy from 90.23% to 98.33% while cutting correct-all inference cost by ~45%.","lead":"A three-stage LLM pipeline (generate, gate, correct) raises Text-to-SPARQL query accuracy on a movie KG from 90% to 98% by fixing only high-risk queries. It matters because executable-but-wrong SPARQL is common and blindly rewriting every query both wastes compute and can break good answers.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"MCQA gains are real but hinge on Gate/Corrector fit to one Generator’s errors; sole external evidence (SciQA) is internally inconsistent, so generality of the reliability claim is not yet secured.","rationale":"The reader’s strongest claim matches the paper’s actual evidence on MCQA, and the weakest assumption (Gate/Corrector tied to one Generator’s error distribution; limited transfer evidence) is the right load-bearing soft spot. I do not find an internal contradiction that overturns the MCQA tables: selective beating correct-all is explained by low Gate FPR plus Corrector over-rewrite on untouched correct queries, consistent with Case C.2 and the correct-all gap. Novelty and baseline gaps are real but secondary to soundness of the stated mechanism. The SciQA table/text mismatch is the concrete place where the paper’s own external support fails a basic consistency check and should be fixed before any stronger than CONDITIONAL accept. No change to CONDITIONAL is warranted; confidence stays moderate without artifacts or a clean second-Generator/second-KG run.","tokens_in":17604,"tokens_out":676,"duration_ms":67105,"concrete_test":"Reconcile SciQA accounting (include vs exclude 43 non-executable gold queries; align Table 13b with §D.2 text) and re-report Generator-only vs GGC. In parallel, freeze the published Gate/Corrector and swap in a second MCQA Generator (same Llama-3.2-3B recipe, different seed or train split); if Gate F1 falls materially or selective query acc drops toward correct-all/Generator-only, the end-to-end claim is Generator-coupled rather than a portable reliability layer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central MCQA result (90.23% → 98.33% selective vs 92.34% correct-all, ~45% less inference than correct-all) is internally coherent: semantic errors dominate Generator failures (Table 3: 77.70%), Gate FPR is ~0.31% with stable thresholds (Tables 4–5a), and over-correction is demonstrated (Case C.2). What the headline reliability claim still rests on is that a Gate and Corrector labeled from one fine-tuned Llama-3.2-3B Generator on MCQA’s label-based movie SPARQL remain the right risk detector and repairer more broadly. Limitations already flags Generator-error coupling and narrow scope. The only non-MCQA support is SciQA Appendix D.2, but it contradicts itself: Table 13b reports 88.09% → 94.47% (+6.38) query acc and 98.23 item F1, while the body text cites 80.70% → 91.62% (+10.92) and 98.31 F1. Until that external number is reconciled and at least one shifted Generator/KG setting is clean, the leap from “works on this Generator–MCQA pair” to “selective correction for reliable Text-to-SPARQL” is the soft joint in the argument—not the MCQA arithmetic itself.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Generator–Gate–Corrector (GGC), a three-stage pipeline for LLM-based Text-to-SPARQL: a fine-tuned Generator produces an initial SPARQL query, a binary Gate decides whether correction is needed, and a Corrector is invoked only on high-risk queries. The central empirical claim is that on MCQA this selective strategy raises query-level accuracy from 90.23% (Generator-only) to 98.33%, outperforming correct-all (92.34%) while cutting test inference time by about 45% relative to correct-all. The motivation is that most Generator failures are executable but semantically wrong (Table 3: 77.70% semantic vs 21.07% syntax), so execution-failure triggers miss the dominant error mode. Ablations cover Gate backbones, threshold sensitivity, KG execution feedback, Corrector training mix (wrong vs correct samples), subset-level Gate–Corrector coordination, case studies of over-correction, and a preliminary SciQA transfer experiment.","tokens_in":17864,"tokens_out":1493,"duration_ms":42100,"significance":"If the MCQA results hold under broader evaluation, the paper makes a useful systems contribution: selective post-generation correction is a practical reliability lever for structured query generation, not only a stronger generator. The work is concrete about the dominant failure mode (semantic, executable queries), documents over-correction risk with cases, and shows that a low-FPR Gate can both save compute and protect already-correct queries. The ablations on Corrector mix and threshold stability are informative for practitioners. Strengths include clear pipeline design, honest Limitations on Generator-error coupling and domain scope, and efficiency reporting alongside accuracy. The significance is currently more engineering/empirical than conceptual novelty (detect-then-correct is familiar from Text-to-SQL), and the generality of the reliability claim still rests heavily on one Generator–MCQA pairing plus a thin external check.","major_comments":[{"comment":"Appendix D.2 contradicts itself on the only non-MCQA evidence. Table 13b reports Generator-only 88.09% → GGC 94.47% query accuracy (+6.38) and item F1 98.23, while the surrounding prose states 80.70% → 91.62% (+10.92) and item F1 98.31. Until these numbers are reconciled (and gold-SPARQL-error filtering is stated once, consistently), SciQA cannot support the transfer/generality claim made in the abstract, introduction, and conclusion.","section":"Appendix D.2, Table 13b"},{"comment":"The headline framing—“reliable LLM-based Text-to-SPARQL” via selective correction—overreaches the experimental base. Main results use one fine-tuned Llama-3.2-3B Generator on MCQA’s label-based movie SPARQL; Gate and Corrector are trained on that Generator’s error distribution (Limitations already notes this coupling). No shifted Generator backbone, ID-based SPARQL setting, or second full-scale KG is reported. Either add at least one clean transfer setting (different Generator and/or KG/SPARQL dialect) or narrow abstract/conclusion claims to “on MCQA, with a Gate/Corrector matched to this Generator.”","section":"Abstract; §1; §5; Limitations"},{"comment":"Baselines for the Text-to-SPARQL claim are thin. Table 2 compares mainly Generator-only, correct-all, and two traditional KGQA systems (NSMh, QGG) that do not emit SPARQL and are not query-level comparable. Related work cites FIRESPARQL and other LLM SPARQL generators, but none appear as controlled baselines. Without at least one recent generation/correction baseline under matched data and metrics, it is hard to judge whether GGC’s gains are specific to selective gating or would be matched by stronger single-pass generation or always-on correction with a better model.","section":"§4.1 Compared Settings; Table 2; §2.2"},{"comment":"Query-level accuracy is defined as answer-set match between executing the predicted SPARQL and the gold SPARQL (§4.1), not string match to gold SPARQL. That is reasonable, but the paper should quantify how often distinct SPARQL strings yield identical answer sets, and whether Gate labels (correct/incorrect) use the same criterion as the reported metric. Any mismatch between labeling criterion and evaluation criterion would inflate Gate quality and end-to-end gains. Please state the exact equality test used for both labeling and scoring.","section":"§4.1 Evaluation Metrics; §3.6 Gate labels"}],"minor_comments":[{"comment":"Table 5b’s default without-feedback pipeline query accuracy is 98.27%, while Table 2’s main selective result is 98.33% (aligned with Table 5c W1.0+C0.2). A short note tying the main row to the exact ablation configuration would remove ambiguity.","section":"Table 2; Table 5b–c"},{"comment":"Item-level precision for Generator-only is 37.96% versus 90.23% query accuracy (Table 2). Briefly explain this gap in the main text (over-broad answer sets on failures), since readers may otherwise distrust one of the two metrics.","section":"§4.2–4.3; Table 2"},{"comment":"Figure 1 is referenced but not visually described in text beyond the equations in §3.2; ensure the camera-ready figure clearly marks the Gate decision and the selective Corrector path.","section":"§3.2; Figure 1"},{"comment":"Minor wording/typos: “Genera-tor–Gate–Corrector” line break in Limitations; inconsistent “WikiData” vs “Wikidata”; “Infer . Time” spacing in Table 2 header.","section":"Limitations; Appendix B; Table 2"},{"comment":"Prompt templates (Appendix B) hard-code Wikidata prefixes and “Do not use wdt syntax” while SciQA uses ORKG—clarify how prompts were adapted for SciQA.","section":"Appendix B; Appendix D.2"}],"recommendation":"major_revision","confidential_remarks":"The MCQA arithmetic and ablations look internally coherent; I would not reject on novelty alone. The SciQA number clash is the clearest fix-or-explain issue and should not reach print as-is. If the authors only patch SciQA text and lightly hedge claims without any extra transfer experiment, minor_revision could become acceptable on a resubmission; as written, major_revision is the proportionate bar."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: on MCQA this is a coherent detect-then-correct stack. Fine-tuned Llama-3.2-3B Generator hits 90.23% query accuracy; always-correct only gets to 92.34% at ~2× inference; selective GGC reaches 98.33% at ~33h vs ~60h. The useful diagnosis is that most Generator failures are executable but semantically wrong (77.7%), so execution-failure triggers miss the real problem. The Gate (RoBERTa, FPR ~0.31%, stable from τ=0.25–0.75) is what makes the cost/accuracy tradeoff work, and the over-correction case study shows why correct-all hurts.\n\nWhat is actually new is not “post-correct SPARQL”—Firesparql and Text-to-SQL repair already exist—but the learned binary risk gate plus the ablations that justify it: threshold sweep, feedback on/off, Corrector mix of wrong+correct samples, and subset coordination. Those pieces are done carefully and the main table hangs together. Limitations are honest about Generator-error coupling and narrow scope.\n\nSoft spots, in proportion: (1) no serious LLM Text-to-SPARQL baselines in the main comparison—NSMh/QGG are answer-retrieval refs, not query generators; (2) SciQA appendix contradicts itself (table 88.09→94.47 vs text 80.70→91.62), so the only external support is not yet trustworthy; (3) Gate/Corrector are fit to one Generator’s mistakes on label-based movie SPARQL—transfer is asserted more than shown. None of that breaks the MCQA arithmetic; it caps how far you can push “reliable Text-to-SPARQL” as a general method.\n\nMath/data/citations look fine for an applied NLP systems paper: external answer-set metrics, timed inference, standard SFT/LoRA setup, related work is on-point. No code/artifacts, so reproducibility is medium.\n\nWho it’s for: people building KGQA or structured-query LLM pipelines who care about when to spend a second pass. I’d send it to peer review with a clear ask to fix SciQA numbers, add at least one shifted Generator or KG setting, and tighten baseline claims. Worth engaging if you work in this lane; not a must-read outside it.","headline":"Clean MCQA systems result: a learned Gate makes selective SPARQL correction beat both Generator-only and correct-all on accuracy and cost; generality beyond that pair is still thin.","tokens_in":18602,"tokens_out":597,"would_cite":false,"duration_ms":18550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Selective correction of only high-risk SPARQL queries raises accuracy from 90% to 98% while cutting rewrite cost nearly in half.","keywords":["Text-to-SPARQL","knowledge graph question answering","large language models","selective correction","query reliability","semantic parsing","SPARQL generation"],"falsifier":"Train or swap in a different generator (or move to ID-based SPARQL / another KG), keep the original gate and corrector frozen, and check whether selective correction still beats both generator-only and correct-all on query-level accuracy and inference cost; collapse of the gap would falsify the claim as a general reliability method.","tokens_in":18344,"feed_emoji":"🔧","tokens_out":903,"duration_ms":18367,"temperature":0.7,"pith_summary":"LLM generators can produce SPARQL that runs on a knowledge graph yet still answers the wrong question. This paper argues that the reliability problem is not only how to generate queries but when to rewrite them. It introduces Generator–Gate–Corrector: a fine-tuned generator proposes a query, a binary gate scores whether that query is risky, and a corrector rewrites only the queries the gate flags. On the MCQA movie benchmark the pipeline lifts query-level accuracy from 90.23% to 98.33%, beats a correct-everything baseline (92.34%), and uses about 45% less inference time than always correcting. The authors show most generator failures are semantic, not syntactic, so execution failure alone cannot decide when to correct; a learned gate that reads the question–query pair is needed.","feed_headline":"Only fix the risky SPARQL: 90% to 98% accuracy","feed_subtitle":"A gate decides when to rewrite LLM queries, beating fix-everything at half the cost","key_machinery":"Generator–Gate–Corrector (GGC): a three-stage pipeline where a supervised generator emits s(0), a binary gate on the question–query pair decides g ∈ {0,1}, and a corrector rewrites only when g = 1, leaving low-risk queries untouched.","core_discovery":"In LLM-based Text-to-SPARQL, selective post-generation correction—triggering a corrector only on queries a learned gate marks as high-risk—raises end-to-end query accuracy well above both generator-only and correct-all pipelines, because most remaining errors are executable but semantically wrong and because unnecessary rewriting can damage already-correct queries.","pith_inferences":["The same detect-then-correct pattern likely transfers to Text-to-SQL and other structured-query settings where fluent but unfaithful outputs are common.","If gate labels must be regenerated whenever the base generator changes, selective correction is a co-trained stack, not a drop-in reliability layer.","A natural next measurement is whether a weaker or frozen generator plus a strong gate–corrector can match a stronger generator-only system at lower total cost.","Case studies of logical composition failures that neither generator nor corrector fix point to needing structural or constraint-level repair, not only label rewriting."],"forward_implications":["Execution-failure triggers alone will systematically miss most Text-to-SPARQL errors if semantic failures dominate as on MCQA.","Always-on correctors can lower net accuracy by overwriting good queries; a low false-positive gate is required for safe rewrite.","Mixing some originally correct samples into corrector training stabilizes preservation without erasing repair skill.","KG execution feedback can help the gate slightly but is not required when question–query mismatch signals are strong.","Reliable Text-to-SPARQL systems should budget compute for detection-then-correct rather than uniform post-editing."],"fun_headline_variants":["Gate flags risky SPARQL only: 90% to 98% accuracy","Selective correction lifts Text-to-SPARQL to 98%","Fix high-risk queries alone: 98% accuracy, 45% less cost","Skip rewrites on good SPARQL; accuracy climbs to 98%","Gate-then-correct beats fix-all on SPARQL reliability"],"cache_read_input_tokens":128,"weakest_assumption_plain":"A gate and corrector trained on one generator’s errors on movie-domain label-based SPARQL will still be the right risk detector and repairer when the generator, SPARQL style, or knowledge graph changes.","fun_headline_variants_meta":{"raw":{"variants":["Gate flags risky SPARQL only: 90% to 98% accuracy","Selective correction lifts Text-to-SPARQL to 98%","Fix high-risk queries alone: 98% accuracy, 45% less cost","Skip rewrites on good SPARQL; accuracy climbs to 98%","Gate-then-correct beats fix-all on SPARQL reliability"]},"model":"grok-4.5","effort":"low","cost_usd":0.003705,"raw_usage":{"total_tokens":1204,"prompt_tokens":771,"num_sources_used":0,"completion_tokens":82,"cost_in_usd_ticks":37048000,"prompt_tokens_details":{"text_tokens":771,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":351,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":771,"tokens_out":82,"duration_ms":6887,"temperature":1.0,"reasoning_tokens":351,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T18:20:27.878106+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train or swap in a different generator (or move to ID-based SPARQL / another KG), keep the original gate and corrector frozen, and check whether selective correction still beats both generator-only and correct-all on query-level accuracy and inference cost; collapse of the gap would falsify the claim as a general reliability method.","supporting_citations":[],"review_version":1}