{"id":"83d158a4-949a-4f94-a284-86d283f40d2f","arxiv_id":"2507.11954","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-stage Wikidata KGQA pipeline using small fine-tuned models generates and executes SPARQL queries, beating GPT-4o on multi-hop and temporal benchmarks.","lead":"This paper builds a modular question-answering system that turns natural language questions into database queries over Wikidata, using small language models for each step. It reports that the system outperforms ChatGPT on complex, multi-hop, and time-sensitive questions, while staying cheaper and more controllable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The direct-QA comparison is scored on mismatched metrics: GPT-4o text answers are exact-matched while the pipeline is scored by entity-set F1.","rationale":"The reader identified the same evaluation-metric mismatch as the most load-bearing concern, and I agree. The central claim depends on a head-to-head comparison that uses different scoring protocols for the two systems. The concern is concrete, testable, and can be settled by remapping GPT-4o outputs to entity IDs. Other weaknesses (missing baselines, no error bars, BM25 tuning) are secondary and do not directly threaten the central claim. The paper's own pipeline numbers are plausible and the system design is reasonable, so the verdict should remain conditional pending the requested check.","tokens_in":10907,"tokens_out":1308,"duration_ms":15989,"concrete_test":"Re-run the GPT-4o direct-QA evaluation on LC-QuAD 2.0 and PAT, mapping each predicted text answer to Wikidata Q-IDs via exact label matching, alias matching, and a standard entity linker (e.g., ReFinED or the paper's RetReason). Then recompute F1 and Acc@1 using the same entity-set evaluation as the pipeline and compare with the pipeline results in Table 4.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline claim (Section 5, Table 4) compares pipeline F1 on entity sets with GPT-4o direct-QA performance scored by exact text match. No normalization of GPT-4o's free-form text answers to Wikidata Q-IDs is described, so answers expressed via aliases or indirect wording are counted as wrong. On LC-QuAD 2.0 and PAT, where gold answers are entity IDs, this scoring asymmetry could substantially understate GPT-4o's true entity-level accuracy. The reported margins are large, so a mapping would not necessarily erase the lead, but the central claim as stated is not verified under commensurate scoring.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-stage pipeline for Wikidata knowledge graph question answering (KGQA). The pipeline consists of entity linking (ReFinED or the proposed RetReason BM25+CoT approach), predicate matching (same RetReason), a fine-tuned Qwen2.5 text-to-SPARQL model (0.5B–7B), and an entity-predicate compatibility filter. The system is evaluated on LC-QuAD 2.0, RuBQ 2.0, QALD-10, and PAT-Questions, against GPT-4o direct QA, few-shot GPT-4o with pipeline components for SPARQL generation, and published baselines. The authors report that the pipeline significantly outperforms direct GPT-4o and baselines on LC-QuAD 2.0 and PAT, which are dominated by multi-hop and temporal questions. They also present a cross-dataset generalization study and a rejection study for filtering invalid queries.","tokens_in":11067,"tokens_out":8623,"duration_ms":88400,"significance":"If the results hold, the paper provides a valuable empirical data point that structured pipelines based on small fine-tuned models can outperform proprietary LLM direct answering on complex and temporal Wikidata QA, with public code and reproducibility materials. The use of four standard benchmarks and the inclusion of a few-shot GPT-4o SPARQL baseline are strengths. The few-shot GPT-4o + ReFineD baseline is scored with the same entity-set F1 as the pipeline, and the pipeline wins on LC-QuAD 2.0 and PAT by substantial margins. However, the headline comparison against GPT-4o direct QA is currently not commensurate because of different scoring schemes, and the per-dataset BM25 tuning raises a leakage risk; both issues need to be addressed before the central claim can be accepted as stated.","major_comments":[{"comment":"The comparison between GPT-4o direct QA and the pipeline is scored on mismatched metrics. Direct answers are evaluated by exact text match against gold labels, while the pipeline and the few-shot GPT-4o + ReFineD row are evaluated with F1 over executed answer entity sets. Free-form text answers that mention the correct entity via an alias or an indirect phrasing are counted as wrong, while the pipeline receives partial credit. The paper does not describe any normalization of GPT-4o outputs to Wikidata Q-IDs. This is a load-bearing issue because the abstract and Section 5 use this comparison to claim that the pipeline 'significantly outperforms' ChatGPT on LC-QuAD 2.0 and PAT. Please re-score GPT-4o direct answers under the same entity-set F1 metric (e.g., by mapping predicted text to Wikidata entities before scoring) or remove/qualify the claim. Note that the few-shot GPT-4o + ReFineD baseline already provides a fair entity-set comparison and the pipeline still wins on LC-QuAD 2.0 (31.0 vs 24.4) and PAT (39.0 vs 17.7), so the central finding is likely recoverable; the current presentation, however, asserts it on the basis of an invalid comparison.","section":"§5, Table 4"},{"comment":"The BM25 hyperparameters (k1, b) and the candidate retrieval depth are reported as optimized 'for each dataset' in Appendix B, but the manuscript does not state on which split this optimization is performed. If the test split is used to select k1, b, and the entity-index popularity threshold, then all reported end-to-end numbers are optimistically biased, and the cross-dataset generalization study in Section 6 is compromised because the target test set would have influenced retrieval parameters. Please state explicitly whether these values are chosen on training/validation splits only; if they are, provide the split definitions and the selected values for each dataset. If they are chosen on test, the experiments should be re-run with a fixed configuration or a proper nested validation.","section":"§4.1, §6, Appendix B"},{"comment":"The generalization experiment shows a large drop on PAT: the full system trained on the other three datasets reaches 12.5 F1, compared with 38.9 F1 when fine-tuned on PAT (Qwen-1.5B + ReFineD in Table 4), a gap of 26.4 points. The text claims that 'the performance gap remains relatively small' and that in certain cases the model exhibits 'comparable generalization,' but the PAT result is neither small nor comparable. Since PAT is one of the two datasets highlighted in the headline claim, this discrepancy needs to be discussed explicitly; the abstract's wording that the system is 'robust' across multi-hop and temporal datasets is stronger than the evidence in Table 5.","section":"§6, Table 5"}],"minor_comments":[{"comment":"The row labeled 'Few-shot GPT-4o + ReFineD' does not match the description in the text, which states that the prompt includes entities from ReFineD and predicates from RetReason. Please rename the row to reflect the actual configuration.","section":"§5, Table 4"},{"comment":"The paper alternates between 'ChatGPT' and 'GPT-4o' for the same model family (e.g., abstract vs Section 5). Use one consistent name, and specify the exact model version and access date for reproducibility.","section":"Throughout"},{"comment":"Acc@1 is defined as a full set match between the predicted and gold entity sets; this is not the standard top-1 accuracy. Consider renaming it to 'exact set match' or clarifying the definition to avoid confusion.","section":"§3, Eq. (1)"},{"comment":"The rejection study reports the 'proportion of identified incorrect SPARQL queries' without defining how the set of incorrect queries is determined (e.g., queries whose execution differs from the gold answer set) or what the denominator is. Please provide the evaluation protocol and report rejection precision and recall as well.","section":"§7, Table 6"},{"comment":"For the published baselines (SPINACH, Konstruktor, Text2Graph, Review-Then-Refine), please specify whether the scores are copied from the original papers or re-computed with the same evaluation script; different protocols for entity-set comparison could affect the comparison.","section":"§5, Table 4"},{"comment":"Minor typographical issues include 'T able' in Table 1, 'P A T' in Table 1, 'an multi-stage' in Section 2, and 'ReFined' in Section 5. The appendix cross-references are otherwise fine.","section":"§4.1, §6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable systems paper, and the fair few-shot comparison suggests the core claim is likely correct. The main risks are the metric mismatch and the unspecified BM25 tuning split; both are fixable in revision. I would not recommend rejection given the public code and the reproducible evaluation setup, but the authors should be required to re-score or qualify the direct-GPT comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a workmanlike extension of the known text-to-SPARQL pipeline recipe: retrieve entity and predicate candidates with BM25, disambiguate with an LLM, feed them to a small fine-tuned Qwen model, execute, and filter. What is actually new is the RetReason CoT disambiguation variant (though ReFinED still does better), a clear joint rejection mechanism that combines entity-predicate compatibility filtering with execution checks, and the first systematic cross-dataset generalization and rejection studies for this setup on Wikidata. Those studies are the most useful part; they are reproducible, the code is public, and they give a realistic picture of where error propagation hurts.\n\nThe main experimental claim is that the pipeline beats GPT-4o direct QA and existing baselines on LC-QuAD 2.0 and PAT. The baseline part largely holds: the pipeline's best F1 on LC-QuAD (31.0) beats Konstruktor and Text2Graph, and on PAT (38.9) beats Review-Then-Refine. It also beats few-shot GPT-4o + ReFineD on the same entity-set F1 metric. That is a fair, same-metric comparison and it supports the core story.\n\nThe soft spot is the GPT-4o direct QA row. Section 5 says direct answers are scored by exact text match, while the pipeline is scored by entity-set F1. Table 4 reports both in the same table without flagging the asymmetry. On PAT, where gold answers are entity IDs, a correct answer phrased as an alias or indirect wording would count as wrong for GPT-4o. The margin there is large (18.3 vs 38.9), so mapping text to Q-IDs probably wouldn't erase the lead entirely, but the claim \"outperforms ChatGPT's direct question answering\" is not verified under commensurate scoring. This should be fixed by either mapping GPT-4o outputs to Wikidata entities or clearly labeling the direct-QA comparison as a secondary, different-metric result.\n\nSecond soft spot: BM25 hyperparameters are tuned per dataset with no validation split described. If that tuning is on the test split, the retrieval numbers are optimistic. Please check Appendix B and the repo. Also, no error bars or seed variance are reported; the training seed is fixed, so we have no idea how stable the 31.0 vs 30.3 differences are. Minor, but worth adding.\n\nWho is this for: KGQA researchers who care about building pragmatic small-model pipelines and measuring rejection behavior. It is not a breakthrough, but the generalization and rejection experiments are a genuine contribution. It deserves serious peer review, and I'd accept it with revisions rather than desk reject. The scoring mismatch must be addressed before publication.","headline":"A solid multi-stage Wikidata KGQA system with a genuinely useful rejection study, but the headline direct-vs-pipeline comparison is scored on mismatched metrics and needs fixing.","tokens_in":11582,"tokens_out":2988,"would_cite":true,"duration_ms":32710,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modular KGQA pipeline that turns questions into SPARQL queries beats direct GPT-4o answering on multi-hop and temporal Wikidata benchmarks.","keywords":["question answering","knowledge graphs","SPARQL","Wikidata","entity linking","temporal reasoning","multi-hop reasoning","chain-of-thought"],"falsifier":"Run GPT-4o direct QA on LC-QuAD 2.0 and PAT, map each generated answer text to Wikidata entity IDs, and compute entity-set F1 using the same gold labels and scoring function as the pipeline; if the direct-model F1 rises to or above the reported pipeline values, the central advantage claim fails.","tokens_in":10728,"feed_emoji":"🔗","tokens_out":4425,"duration_ms":47483,"temperature":0.7,"pith_summary":"The paper claims that a query-based KGQA system, built from small fine-tuned language models and explicit retrieval stages, outperforms direct large-language-model question answering on benchmarks dominated by multi-hop and temporal questions. On LC-QuAD 2.0 and PAT-Questions, the full pipeline achieves higher F1 than GPT-4o prompted to answer directly, and it also beats a few-shot GPT-4o text-to-SPARQL setup. The authors interpret this as evidence that structured, modular pipelines remain a viable path for complex structured QA, not merely a fallback when proprietary APIs are unavailable. The paper additionally introduces a retrieval-plus-chain-of-thought entity linking and predicate matching method, and a rejection mechanism that filters invalid queries before they reach the user.","feed_headline":"Structured queries beat ChatGPT on hard Wikidata QA","feed_subtitle":"A small-model KGQA pipeline outperforms direct GPT-4o answering on temporal and multi-hop benchmarks.","key_machinery":"The load-bearing components are: a BM25 index over Wikidata entities and predicates that proposes candidates; a chain-of-thought disambiguation step that asks a reasoning LLM to select the correct entity or predicate from the candidates; a fine-tuned Qwen2.5 text-to-SPARQL generator that turns the question plus linked candidates into an executable query; and two rejection filters—an entity-to-predicate compatibility check using the knowledge graph's actual relation structure, and execution-based filtering that drops queries that error out or return empty. The central object is the full pipeline treated as one system from natural language to entity-set answers, with the text-to-SPARQL model as the core and the linking stages as guardrails.","core_discovery":"On the paper's own terms, the central discovery is that an explicit pipeline—BM25 candidate retrieval, LLM-based disambiguation of entities and predicates, a fine-tuned text-to-SPARQL model, and ontology-aware filtering—achieves strong results on complex Wikidata QA, and that its advantage is largest precisely on datasets requiring multiple hops or time-sensitive answers. The best reported setup (Qwen-7B with ReFinED) reaches F1 of 31.0 on LC-QuAD 2.0 and 39.0 on PAT, versus 21.1 and 18.3 for GPT-4o direct QA, under the paper's evaluation protocols. The paper also shows that the text-to-SPARQL component generalizes across datasets with only a modest drop, and that combining entity-to-predicate compatibility checks with query execution filtering rejects a large share of incorrect query generations.","pith_inferences":["If the reported advantage over GPT-4o is robust, an even stronger test would be to map direct GPT-4o answers to Wikidata entity IDs and score them with the same entity-set F1 used for the pipeline; any shrinkage of the gap would indicate part of the lead is an artifact of scoring protocol rather than true QA ability.","The rejection mechanism could be repurposed as a confidence signal for deciding when to fall back to direct LLM answering, yielding a hybrid that keeps the pipeline's precision on complex questions and the LLM's strength on simple ones.","The BM25-plus-chain-of-thought retrieval pattern may transfer to other structured generation tasks, such as text-to-SQL over schemas with many relations, where candidate generation followed by reasoning-based selection has the same shape.","For temporal questions in PAT, a future test would re-run the pipeline on updated versions of the benchmark to see whether the auto-updating answers remain correct without retraining."],"forward_implications":["On datasets dominated by multi-hop and temporal questions, query-based KGQA composed of small fine-tuned models can outperform direct proprietary LLM answering, which is the paper's headline claim from Table 4.","The proposed RetReason entity and predicate linking via BM25 plus chain-of-thought is competitive with a strong fine-tuned linker like ReFinED and can be used through an LLM API without additional training, per Tables 2 and 3.","The rejection mechanism combining ontology-based entity-predicate filtering with query execution detects substantially more incorrect queries than LLM self-rejection alone, as reported in Table 6.","Cross-dataset validation shows the text-to-SPARQL model generalizes to unseen datasets with a modest performance gap relative to in-dataset fine-tuning, as shown in Figure 3 and Table 5.","Error propagation across successive components is acknowledged as a limiting factor, yet system-level gains on LC-QuAD and PAT remain the paper's main evidence."],"supporting_citations":[{"why":"Provides the ReFinED entity linker used as the primary fine-tuned entity linking baseline and as a component in end-to-end system variants.","marker":"[3]"},{"why":"Defines the LC-QuAD 2.0 benchmark over Wikidata, one of the two datasets on which the pipeline's main advantage is claimed.","marker":"[12]"},{"why":"Defines the PAT-Questions benchmark for present-anchored temporal QA, the other dataset central to the paper's main claim.","marker":"[26]"},{"why":"Supplies the motivation that small fine-tuned language models can handle text-to-SPARQL tasks, which the paper builds on and extends.","marker":"[8]"},{"why":"SPINACH serves as a strong query-based KGQA baseline that the system is compared against on QALD-10.","marker":"[23]"},{"why":"Konstruktor acts as a baseline for simple KGQA and is used as a comparison point on LC-QuAD 2.0 and RuBQ 2.0.","marker":"[25]"}],"fun_headline_variants":["Small-model KGQA pipeline beats GPT-4o on temporal QA","Explicit query pipeline outperforms direct LLM on Wikidata QA","Modular KGQA beats GPT-4o on multi-hop and temporal Q&A","Query-based KGQA wins on temporal and multi-hop datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison with GPT-4o direct QA assumes that exact-match text scoring is on the same footing as entity-set F1 for the pipeline; if direct answers were first mapped to Wikidata entity IDs and scored the same way, the reported lead on LC-QuAD and PAT could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Small-model KGQA pipeline beats GPT-4o on temporal QA","Explicit query pipeline outperforms direct LLM on Wikidata QA","Modular KGQA beats GPT-4o on multi-hop and temporal Q&A","Query-based KGQA wins on temporal and multi-hop datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000956,"raw_usage":{"total_tokens":4026,"prompt_tokens":845,"completion_tokens":3181,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":3106}},"tokens_in":461,"tokens_out":3181,"duration_ms":23731,"temperature":1.0,"reasoning_tokens":3106,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:57:34.645489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GPT-4o direct QA on LC-QuAD 2.0 and PAT, map each generated answer text to Wikidata entity IDs, and compute entity-set F1 using the same gold labels and scoring function as the pipeline; if the direct-model F1 rises to or above the reported pipeline values, the central advantage claim fails.","supporting_citations":[{"cited_title":"In: Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","cited_arxiv_id":null,"evidence_quote":"Provides the ReFinED entity linker used as the primary fine-tuned entity linking baseline and as a component in end-to-end system variants."},{"cited_title":"In: Findings of the Association for Computational Linguistics ACL 2024","cited_arxiv_id":null,"evidence_quote":"Defines the PAT-Questions benchmark for present-anchored temporal QA, the other dataset central to the paper's main claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the motivation that small fine-tuned language models can handle text-to-SPARQL tasks, which the paper builds on and extends."},{"cited_title":"In: Findings of the Association for Computational Linguistics: EMNLP 2024","cited_arxiv_id":null,"evidence_quote":"SPINACH serves as a strong query-based KGQA baseline that the system is compared against on QALD-10."},{"cited_title":"In: International Confer- ence on Applications of Natural Language to Information Systems","cited_arxiv_id":null,"evidence_quote":"Konstruktor acts as a baseline for simple KGQA and is used as a comparison point on LC-QuAD 2.0 and RuBQ 2.0."}],"review_version":1}