{"id":"22a5b2e8-aeb5-4e36-89b6-efee2c76d317","arxiv_id":"2508.10467","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"A fine-tuned LLM with optional RAG and a correction layer reaches 0.90 ROUGE-L and 0.85 RelaxedEM for SPARQL generation over scholarly knowledge graphs on SciQA.","lead":"FIRESPARQL translates natural language questions about scholarly knowledge graphs into SPARQL queries by combining fine-tuned language models, retrieval augmentation, and a query correction layer. On the SciQA benchmark, it reports 0.90 ROUGE-L for query text and 0.85 RelaxedEM for returned results, with fine-tuning alone performing best.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set contamination risk in reported fine-tuning metrics; requires disclosure of validation split and model selection protocol.","rationale":"The reader's weakest assumption matches the primary risk: the evaluation assumes a clean train/test split and no test-set influence on model selection. Since the full text is unavailable, I cannot identify a definite internal flaw, but the abstract's reported numbers hinge on this unverified premise. My suggested concrete test would provide the necessary evidence. The verdict remains UNVERDICTED because the absence of the full text prevents confirming or rejecting the concern; no adjustment is warranted beyond what the reader already concluded.","tokens_in":890,"tokens_out":1540,"duration_ms":18883,"concrete_test":"Inspect the SciQA official splits and any released code. Verify: (1) no SciQA test question appears in the fine-tuning/validation sets (e.g., exact-match and fuzzy-match on normalized text); (2) training logs and hyperparameter search scripts reference only a validation split, not test labels; (3) run a memorization check by prompting the fine-tuned model with test questions and measuring whether output queries resemble gold queries beyond chance, comparing to a held-out set of fresh questions; (4) if code is available, re-run fine-tuning with the reported hyperparameters on the official train split and evaluate on the official test split to reproduce 0.90 ROUGE-L and 0.85 RelaxedEM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the reported fine-tuning performance on SciQA (0.90 ROUGE-L, 0.85 RelaxedEM). This claim is load-bearing because it supports the paper's conclusion that fine-tuning outperforms zero-shot, one-shot, and RAG baselines. The abstract does not state how hyperparameters, early stopping, RAG context selection, or correction-layer rules were chosen. If any of these choices were made using the test set, the reported metrics are optimistic. Moreover, SciQA is a public benchmark; the base LLM's pretraining may include these questions, which would inflate lexical overlap and result-set accuracy regardless of the framework's contribution. The paper may be internally consistent, but its external validity rests on an unverified clean split and contamination-free pretraining. Without full-text disclosure of the experimental protocol, the numbers cannot be trusted as evidence of generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FIRESPARQL, a modular framework for generating SPARQL queries from natural-language questions over scholarly knowledge graphs. The framework combines a fine-tuned LLM core with optional retrieval-augmented generation (RAG) and a SPARQL query correction layer. The abstract reports evaluations on the SciQA benchmark across zero-shot, one-shot, RAG-augmented, and fine-tuned configurations, with the fine-tuned configuration reaching 0.90 ROUGE-L for query accuracy and 0.85 RelaxedEM for result accuracy on the test set, and claims improvements over baselines and state-of-the-art approaches.","tokens_in":1089,"tokens_out":2121,"duration_ms":25790,"significance":"If the reported performance is obtained under a clean and reproducible evaluation protocol, FIRESPARQL is a practical contribution to question answering over scholarly knowledge graphs. The modular combination of fine-tuning, RAG, and a correction layer is sensible, and the use of both lexical similarity (BLEU/ROUGE) and result-level accuracy (RelaxedEM) is appropriate for SPARQL generation. However, this review is based only on the abstract, which omits baseline values, variance information, and experimental-protocol details. The empirical strength of the central claim therefore cannot currently be confirmed.","major_comments":[{"comment":"The load-bearing claim is that fine-tuning achieves 0.90 ROUGE-L and 0.85 RelaxedEM on the SciQA test set. The abstract does not state how the train/validation/test split was created, how hyperparameters were selected, or whether early stopping, RAG context size, or correction-layer rules were chosen with reference to the test set. If any of these choices involved test-set information, the reported metrics are optimistic. The full manuscript must disclose the model-selection protocol, including validation-split construction and any hyperparameter search.","section":"Abstract, experimental results paragraph"},{"comment":"No baseline numbers, confidence intervals, error bars, or significance tests are reported. The sentence claiming that FIRESPARQL 'outperforms' baselines and state-of-the-art approaches cannot be assessed from the abstract alone. A complete comparison table with all named baselines and configurations, plus variance or significance measures, is needed to support this comparative claim.","section":"Abstract, experimental results paragraph"},{"comment":"SciQA is a public benchmark, and the base LLM's pretraining data may contain these questions and gold SPARQL queries. The abstract does not address possible test-set contamination. Because lexical overlap and result-set match could be inflated by memorization, the paper should report contamination checks, such as n-gram overlap analysis, or otherwise justify that the reported metrics reflect generalization rather than leakage.","section":"Abstract, experimental results paragraph"}],"minor_comments":[{"comment":"Typographical issue: 'relaxed exact match(RelaxedEM)' should read 'relaxed exact match (RelaxedEM)'.","section":"Abstract, first sentence of evaluation description"},{"comment":"The abstract does not mention code or data availability. If the framework and evaluation scripts are publicly released, stating this would strengthen reproducibility.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This review was performed on the abstract only, as the full text was not available. The uncertainty in my recommendation reflects insufficient evidence rather than a judgment that the work is flawed. If the full manuscript contains the missing experimental protocol, baseline tables, and contamination analysis, I would be able to make a more definitive recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked for a quick read on this one. It's abstract-only, so everything is provisional, but here's my take. The paper proposes FIRESPARQL, a modular pipeline for turning natural-language questions into SPARQL over scholarly knowledge graphs: fine-tuned LLM core, optional RAG context, and a correction layer. That's a sensible modular decomposition, and the reported test numbers are strong—0.90 ROUGE-L and 0.85 RelaxedEM on SciQA, with fine-tuning beating zero-shot, one-shot, and RAG variants. The abstract is coherent and the claim is concrete. Good. What's genuinely new is the integration and the explicit diagnosis of two error types (structural inconsistencies vs. semantic inaccuracies). That's a useful framing, though not a deep theoretical contribution. The paper does what an applied systems paper should: name a problem, build a framework, evaluate on a public benchmark, compare against baselines. I credit that. Now the soft spots. First, the abstract gives no baseline numbers, no error bars, no significance tests. So we can't tell how much of the 0.90/0.85 is real headroom over existing systems. Second, the contamination concern the stress-test raises is legitimate, but it's a question, not a demonstrated flaw. The authors don't disclose their validation split or model-selection procedure in the abstract. If the test set influenced early stopping or correction-layer rules, the metrics are optimistic. Also, SciQA is public, so pretraining contamination is possible for the base LLM, which would inflate lexical overlap regardless of the framework. But nothing here proves leakage. I'd want the full paper to check for a held-out validation split and any leakage-avoidance discussion. The bigger issue is that the full text is invisible to us. No code, no data, no detailed protocol. So I can't verify the central claim. That's not a flaw in the paper, just a limit on what I can say. Who gets value from this? People working on KG question answering, especially scholarly KG access. It's an applied systems result, not a paradigm shift. If the full paper is solid, it's a useful data point. I'd send it to peer review because the claim, if reproducible, is worth referee time. My own verdict is unproven until I see the experimental section. If the protocol holds up, I'd cite it. Recommendation: read the full text when it's available; ask the authors about split hygiene and whether the base model was tuned on SciQA. Not a desk reject, but not a pass yet either.","headline":"Abstract-only: plausible modular SPARQL-generation framework with a strong reported score, but the evaluation protocol is unverifiable from what's shown and the contamination question is fair to ask.","tokens_in":694,"tokens_out":731,"would_cite":false,"duration_ms":17748,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuning a language model inside a modular framework—with optional retrieval and a SPARQL correction layer—achieves 0.90 ROUGE-L and 0.85 RelaxedEM on the SciQA benchmark, the best of the configurations tested.","keywords":["SPARQL query generation","scholarly knowledge graphs","large language models","fine-tuning","retrieval-augmented generation","question answering","SciQA benchmark","query correction"],"falsifier":"Re-run the fine-tuned configuration on the SciQA test set after choosing all hyperparameters and correction rules using only a validation split, then compare ROUGE-L and RelaxedEM to the reported values; a large drop would indicate selection on the test set. Also compute exact result-set match alongside RelaxedEM: if exact match is far below 0.85, the generated queries are lexically close but not fully semantically equivalent to the gold queries.","tokens_in":812,"feed_emoji":"🤖","tokens_out":3762,"duration_ms":38756,"temperature":0.7,"pith_summary":"This paper claims that LLM-generated SPARQL queries over scholarly knowledge graphs fail in two identifiable ways: structural mistakes such as missing or redundant triples, and semantic mistakes where the query is well formed but names the wrong entity or property. To counter this, the authors propose FIRESPARQL, a modular framework whose core is a fine-tuned LLM, with optional retrieval-augmented generation (RAG) context and a rule-based SPARQL query correction layer. Evaluated on the SciQA benchmark, the fine-tuned configuration reaches 0.90 ROUGE-L for query text accuracy and 0.85 RelaxedEM for result accuracy, outperforming zero-shot, one-shot, and RAG-augmented configurations. The claim matters because reliable natural-language-to-SPARQL translation would let researchers query scholarly knowledge graphs without learning SPARQL or the underlying schema.","feed_headline":"Fine-tuned LLMs generate SPARQL at 0.90 ROUGE-L on SciQA","feed_subtitle":"Modular framework with RAG and correction beats zero- and one-shot baselines; fine-tuning alone leads.","key_machinery":"FIRESPARQL, a modular framework combining a fine-tuned LLM as the core component, optional retrieval-augmented generation context, and a SPARQL query correction layer. The error taxonomy of structural inconsistencies versus semantic inaccuracies is the organizing principle: the correction layer targets structural errors, while fine-tuning and RAG are meant to improve semantic grounding of entities and properties.","core_discovery":"The central claim is that fine-tuning an LLM on scholarly knowledge graph question-answer data is the dominant factor in producing correct SPARQL queries, and that the remaining errors can be categorized and addressed by optional retrieval context and a correction layer. FIRESPARQL's fine-tuned configuration achieves 0.90 ROUGE-L for query accuracy and 0.85 RelaxedEM for result accuracy on the SciQA test set, surpassing the zero-shot, zero-shot with RAG, one-shot, and fine-tuning-with-RAG configurations as well as baseline and state-of-the-art approaches compared in the paper. The paper also offers a two-part error taxonomy: structural inconsistencies, such as missing or redundant triples, a","pith_inferences":["A likely reason fine-tuning beats RAG in this setup is that RAG supplies relevant text fragments but does not teach the model the graph schema's shape, whereas fine-tuning encodes that shape directly into the model weights; this explanation goes beyond what the paper states.","Because BLEU and ROUGE measure lexical overlap while RelaxedEM measures result equivalence, a natural follow-up is to report exact result-set match and an ablation that isolates how much the correction layer contributes to each metric.","The framework's claims should transfer to other scholarly knowledge graphs only if their schemas and query patterns resemble SciQA; a cross-graph evaluation would test that transfer."],"forward_implications":["Fine-tuning alone outperforms zero-shot, one-shot, and RAG-augmented configurations on SciQA, suggesting that domain-specific weights matter more than added context for SPARQL generation over scholarly knowledge graphs.","The two-part error taxonomy gives future systems a concrete target: structural errors can be repaired by rules, while semantic errors require better entity and property grounding.","Because the framework is modular, the RAG and correction components can be removed or swapped independently, making it a reusable testbed for LLM-to-SPARQL methods on other knowledge graphs.","At 0.85 RelaxedEM, a meaningful fraction of test questions still do not return fully matching result sets, so the problem is substantially improved but not closed."],"supporting_citations":[],"fun_headline_variants":["Fine-tuning beats RAG for SPARQL queries on scholarly graphs","SPARQL generation hits 0.90 ROUGE-L with fine-tuned LLMs","Why fine-tuning wins for scholarly SPARQL queries","LLM fine-tuning tops RAG for SciQA SPARQL accuracy","Error taxonomy guides LLM SPARQL correction layer"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The reported scores assume the SciQA train/test split is clean, meaning model selection, hyperparameters, RAG settings, and correction-layer rules were chosen without looking at the test set; if any of those choices leaked test information, the reported 0.90 ROUGE-L and 0.85 RelaxedEM are optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning beats RAG for SPARQL queries on scholarly graphs","SPARQL generation hits 0.90 ROUGE-L with fine-tuned LLMs","Why fine-tuning wins for scholarly SPARQL queries","LLM fine-tuning tops RAG for SciQA SPARQL accuracy","Error taxonomy guides LLM SPARQL correction layer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1228,"prompt_tokens":853,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":283}},"tokens_in":597,"tokens_out":375,"duration_ms":3922,"temperature":1.0,"reasoning_tokens":283,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:24:27.042139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the fine-tuned configuration on the SciQA test set after choosing all hyperparameters and correction rules using only a validation split, then compare ROUGE-L and RelaxedEM to the reported values; a large drop would indicate selection on the test set. Also compute exact result-set match alongside RelaxedEM: if exact match is far below 0.85, the generated queries are lexically close but not fully semantically equivalent to the gold queries.","supporting_citations":[],"review_version":1}