{"id":"eea07ae2-10fe-4685-8264-6ff0e126e9b0","arxiv_id":"2508.01815","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"SchemaForge improves text-to-SPARQL over heterogeneous knowledge graphs by selecting a schema slice before generating the query, raising execution accuracy by 11.5 points on average.","lead":"This paper introduces SchemaForge, a system that answers natural-language questions by first picking a knowledge graph schema that can support the query, then generating a SPARQL query for that schema. The authors report large accuracy gains over existing methods on four question-answering benchmarks, including an 11.50 percentage point average improvement in execution accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 11.50 pp average improvement cannot be verified: the supplied full text is corrupted and contains a line from an unrelated arXiv paper, leaving no readable method, baseline, or results details.","rationale":"The reader's verdict is UNVERDICTED due to unreadable text. I agree with that verdict. The reader's weakest_assumption pointed to schema-alignment noise as the key technical risk; my concern is even more basic: the supplied text cannot be used to assess any technical risk because it is corrupted and contains a different arXiv paper's header. The central claim is an empirical comparison, and without the evaluation details it is impossible to judge whether the improvement is real. I am not claiming the result is false, only that the evidence is absent. A concrete recovery-and-verification step would settle the concern. Therefore the verdict should remain UNCHANGED (UNVERDICTED), since no technical objection can be validated from the available material.","tokens_in":13727,"tokens_out":4755,"duration_ms":52090,"concrete_test":"Obtain the original PDF from arXiv and inspect the evaluation section. Concretely, verify that (1) per-benchmark execution accuracy and standard deviations are reported for SchemaForge and every baseline; (2) the 'strongest matched agent baseline' is defined with identical LLM, prompting, and inference budget; and (3) an oracle-ablation quantifies the ceiling of schema-slice alignment. Recompute the average improvement from the per-benchmark tables. If the recovered numbers match the abstract and the baseline is matched, the claim stands; otherwise the 11.50 pp average is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on execution-accuracy comparisons across four benchmarks. For the claim to hold, the evaluation must be fair: the baseline must be genuinely strong, matching compute and prompting; the graph-allocation and schema-slice steps must be error-prone in the way the paper describes; and the reported numbers must be reproducible. None of this can be checked because the full text supplied is unreadable and includes the line 'arXiv:2508.01811v2 [math.AP] 6 Jul 2026', which belongs to a different paper. No experimental setup, hyperparameters, ablation studies, or per-benchmark breakdowns are available. This is a missing-support issue: the abstract's concrete numbers are unsupported by any readable evidence, so the central claim is unverified rather than demonstrated. In particular, we cannot rule out that the improvement is an artifact of an under-tuned baseline or of evaluation leakage, and we cannot assess the robustness of the schema-slice alignment that the method depends on.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SchemaForge, an agentic framework for text-to-SPARQL over heterogeneous knowledge graph collections. The core idea is question-conditioned schema-slice alignment: the system first uses weak graph evidence to select plausible graphs, then uses stronger schema evidence to select a local schema slice that supports the question's predicates, types, joins, and filters, before generating and verifying SPARQL queries. The abstract reports execution-accuracy improvements over a matched agent baseline on LC-QuAD 2.0, QALD-9 Plus, QALD-10, and Spider4SPARQL, including a claimed average gain of 11.50 percentage points and a Spider4SPARQL improvement from 54.86% to 64.18%. However, the supplied full text is largely unreadable due to character corruption and contains a line from an unrelated arXiv paper, so the method, experimental setup, baselines, and per-benchmark results cannot be inspected. The central empirical claim is therefore unverified in the submitted manuscript.","tokens_in":13875,"tokens_out":2248,"duration_ms":26274,"significance":"If the reported results are reproducible, the paper addresses a genuine and under-explored problem: text-to-SPARQL when the target graph is not fixed in advance and multiple heterogeneous KGs with partial schema alignments are available. The focus on schema-slice realization, rather than syntax-only generation, is a plausible and potentially useful direction, and the use of four public benchmarks (LC-QuAD 2.0, QALD-9 Plus, QALD-10, Spider4SPARQL) is appropriate for demonstrating the claim. The abstract's concrete quantitative claims are falsifiable and would be strengthened by sharing code and instructions for reproduction. However, because the full text cannot be read, none of these potential strengths can be confirmed: the method is not formally specified, the baseline strength and matching protocol are unknown, and no error bars or per-benchmark breakdowns are available. The significance is thus conditional on a verifiable manuscript, which is not what was provided.","major_comments":[{"comment":"The supplied full text is unreadable: most characters are replaced by placeholder symbols, and the text includes the line 'arXiv:2508.01811v2 [math.AP] 6 Jul 2026', which belongs to a different paper. As a result, the proposed schema-slice alignment algorithm, the SPARQL generation and verification modules, the baselines, the experimental protocol, and the per-benchmark results cannot be inspected. This is a load-bearing issue because the central claim of an 11.50 percentage point average improvement over the 'strongest matched agent baseline' rests entirely on the experimental details, none of which are available.","section":"Full Text"},{"comment":"The abstract reports aggregate execution accuracy improvements without per-benchmark numbers, standard deviations, the number of evaluation runs, or a named baseline. Specifically, the claim 'improves execution accuracy over the strongest matched agent baseline by 11.50 percentage points on average' cannot be verified or reproduced from the given material, and the Spider4SPARQL numbers (54.86% to 64.18%, plus 73.0% Top-1 and 97.0% Top-3 graph allocation accuracy) are presented without definitions or confidence intervals.","section":"Abstract (experimental claims)"}],"minor_comments":[{"comment":"The terms 'Top-1 graph allocation accuracy' and 'Top-3 graph allocation accuracy' are used but not defined; please provide formal definitions in the abstract or in the experiments section.","section":"Abstract"},{"comment":"The phrase 'strongest matched agent baseline' does not name the baseline; the identity of this baseline and the matching criteria (e.g., same LLM, same prompting budget) should be stated explicitly.","section":"Abstract"},{"comment":"The full text contains a line from arXiv:2508.01811, indicating a corrupted PDF or a compilation error; the authors should ensure the submission contains the intended content and no foreign text.","section":"Full Text"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the supplied PDF appears to be corrupted, as the full text is almost entirely unreadable and includes a header from an unrelated mathematics paper. It may be worth asking the authors to resubmit a clean version before any substantive review. Even reading the abstract alone, however, the experimental reporting is too thin to support the central claim. If a clean full text becomes available, the review should focus on baseline strength, the schema-slice alignment failure modes, and whether test-set hyperparameters were tuned. The topic is within the journal's scope, so a resubmission can be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe abstract for 2508.01815 is the only readable part of the submission, and it makes a claim worth checking: SchemaForge improves execution accuracy by 11.5 percentage points on average over the strongest matched agent baseline across four public text-to-SPARQL benchmarks. The core idea—question-conditioned schema-slice alignment, where weak evidence selects candidate graphs and stronger schema evidence selects a local slice that constrains query generation, plus counterfactual answer-set validation—is a plausible and genuinely new combination. I cannot verify any of it.\n\nThe supplied full text is mangled to the point of being unreadable; it even contains a header from an unrelated arXiv paper in math.AP. So I have zero access to method details, baseline definitions, hyperparameters, ablations, or per-benchmark results. The stress-test concern is exactly right: the reported numbers could be real, or they could come from an under-tuned baseline or evaluation leakage. Nothing in the readable material lets me distinguish. This is a missing-support problem, not a proven flaw. The abstract is carefully written, and the reduction to the single-KG case is a nice touch that suggests the authors thought about the scope of their contribution.\n\nWhat the paper does well: it targets a real problem—practical KGQA over heterogeneous graph collections where the target graph is not fixed in advance—and it evaluates on four public benchmarks rather than a single controlled setting. The graph-allocation accuracy numbers on Spider4SPARQL (73% Top-1, 97% Top-3) are concrete and falsifiable, if the rest of the evaluation holds up.\n\nThe soft spot is entirely the unreadable full text. I cannot judge the soundness of the experiments, the strength of the baselines, or the robustness of the schema-slice alignment. I also cannot rule out that the 11.5-point average is driven by one outlier benchmark. These are not accusations; they are exactly what a referee would need to check, and the current submission does not allow it.\n\nWho is this for? Researchers working on text-to-SPARQL, schema grounding, or KGQA over multiple graphs. If the actual arXiv PDF is readable and matches this abstract, it deserves a serious referee. I would not desk-reject on the strength of the abstract alone, but I would not trust the numbers until the full text is available.\n\nRecommendation: send to peer review, conditional on receiving a readable version. Ask the authors to release code and data, since the claim is empirical and the baseline strength is the main threat to validity.","headline":"The abstract promises a meaningful gain in heterogeneous text-to-SPARQL, but the unreadable full text makes the central claim unverifiable.","tokens_in":14392,"tokens_out":3521,"would_cite":false,"duration_ms":34888,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The decisive step in heterogeneous text-to-SPARQL is schema realization: finding the local schema slice that can support the question's predicates, joins, and filters before generating the query.","keywords":["text-to-SPARQL","knowledge graph question answering","heterogeneous knowledge graphs","schema grounding","graph retrieval","counterfactual validation","schema-slice alignment","agentic framework"],"falsifier":"A direct ablation on Spider4SPARQL would settle the claim: replace the question-conditioned schema-slice alignment with a lexical-only selector over the same candidate graphs and keep query generation and counterfactual checks fixed. If execution accuracy stays near 64.18%, schema realization is not the mechanism driving the gain; if it drops toward the 54.86% baseline, the central claim is supported. A second check is to build a heterogeneous benchmark whose candidate graphs share near-identical schemas; the 73.0% Top-1 graph allocation should not collapse if the aligner is using genuine schema evidence rather than surface word matches.","tokens_in":13532,"feed_emoji":"🧩","tokens_out":5551,"duration_ms":59620,"temperature":0.7,"pith_summary":"The paper sets out to establish that, when a question must be answered over a collection of knowledge graphs with different schemas, the binding constraint is not graph retrieval alone but schema realization: finding a local schema slice that can support the question's predicates, entity types, joins, filters, and constraints. Its system, SchemaForge, first uses weak graph evidence to nominate plausible graphs, then uses stronger schema evidence to select a schema slice, and only then generates SPARQL constrained by that slice, followed by counterfactual answer-set checks. Across LC-QuAD 2.0, QALD-9 Plus, QALD-10, and Spider4SPARQL, it reports an average execution-accuracy gain of 11.50 percentage points over the strongest matched agent baseline, and on Spider4SPARQL improves from 54.86% to 64.18% with 73.0% Top-1 graph allocation. If correct, the result reframes heterogeneous KGQA: query generation should be constrained by a verified schema commitment rather than left to free-form decoding.","feed_headline":"Schema selection before SPARQL lifts accuracy by 11.5 points","feed_subtitle":"Four KGQA benchmarks: schema-realization constraints plus counterfactual checks improve executable SPARQL.","key_machinery":"The load-bearing object is the question-conditioned schema slice: a subset of a knowledge graph's schema, comprising predicates, entity types, join paths, and filters, that is sufficient to support one question. The framework's central mechanism is two-stage alignment, in which weak graph evidence first selects plausible graphs and stronger schema evidence then decides whether a candidate slice can realize the intended query; the chosen slice constrains SPARQL generation, and a counterfactual answer-set validation stage rejects queries whose answers do not behave as the question implies under controlled perturbations. In the single-graph case the same pipeline reduces to schema-grounded text-to-SPARQL, which lets the reported gains be attributed to schema realization rather than to graph retrieval alone.","core_discovery":"On its own terms, the central discovery is that over heterogeneous knowledge graphs, text-to-SPARQL should be treated as a schema-realization problem rather than a graph-retrieval-then-generation problem. The system first uses weak graph evidence to identify plausible graphs, then lets stronger schema evidence decide whether a local schema slice can realize the predicates, entity types, joins, filters, and constraints in the question. That selected slice then constrains query generation and verification before execution, and counterfactual answer-set checks filter out queries that would execute but answer the wrong question. The reported results are an 11.50 percentage-point average improvement in execution accuracy over the strongest matched agent baseline across four benchmarks, and on Spider4SPARQL an improvement from 54.86% to 64.18% with 73.0% Top-1 and 97.0% Top-3 graph allocation accuracy.","pith_inferences":["A testable extension is that the same schema-slice-constrained decoding could transfer to text-to-SQL over heterogeneous relational schemas, where the load-bearing step is likewise matching question predicates and joins to a schema before generating the query.","The reported gap between Top-1 and Top-3 graph allocation suggests that a reranking stage trained on counterfactual answer-set outcomes could recover additional accuracy without changing the generator; this is not tested in the paper.","If schema evidence extraction is the bottleneck, then synthetic question-to-schema alignment data should transfer directly to execution accuracy, a prediction the paper's account implies but does not itself demonstrate."],"forward_implications":["Heterogeneous KGQA systems should budget effort for schema realization, not just graph retrieval, because the schema slice that can support the question is what constrains whether any generated SPARQL can execute correctly.","When only one graph is available, the formulation reduces to schema-grounded text-to-SPARQL, so schema-slice selection is a candidate improvement for single-graph systems as well.","Graph allocation accuracy becomes a diagnosable intermediate metric: the 73.0% Top-1 versus 97.0% Top-3 gap on Spider4SPARQL shows most errors are rankable within the top three graphs, so reranking among candidates is a direct lever.","Counterfactual answer-set checks can suppress executable-but-wrong queries, a failure mode that execution accuracy alone would otherwise count as success."],"supporting_citations":[],"fun_headline_variants":["Schema-first SPARQL: 11.5-point accuracy gain","Schema realization before query generation lifts SPARQL accuracy","Counterfactual validation improves text-to-SPARQL by 11.5 points","For heterogeneous graphs, schema slicing beats graph guessing for SPARQL","From graph retrieval to schema realization: better SPARQL accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the predicates, entity types, joins, filters, and constraints needed by a question can be reliably extracted from natural language and matched to the correct schema slice of the correct knowledge graph; if that alignment is noisy, the gains over baselines would collapse because query generation is constrained by the selected slice.","fun_headline_variants_meta":{"raw":{"variants":["Schema-first SPARQL: 11.5-point accuracy gain","Schema realization before query generation lifts SPARQL accuracy","Counterfactual validation improves text-to-SPARQL by 11.5 points","For heterogeneous graphs, schema slicing beats graph guessing for SPARQL","From graph retrieval to schema realization: better SPARQL accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3174,"prompt_tokens":1046,"completion_tokens":2128,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":2038}},"tokens_in":662,"tokens_out":2128,"duration_ms":18231,"temperature":1.0,"reasoning_tokens":2038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:20:46.079814+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct ablation on Spider4SPARQL would settle the claim: replace the question-conditioned schema-slice alignment with a lexical-only selector over the same candidate graphs and keep query generation and counterfactual checks fixed. If execution accuracy stays near 64.18%, schema realization is not the mechanism driving the gain; if it drops toward the 54.86% baseline, the central claim is supported. A second check is to build a heterogeneous benchmark whose candidate graphs share near-identical schemas; the 73.0% Top-1 graph allocation should not collapse if the aligner is using genuine schema evidence rather than surface word matches.","supporting_citations":[],"review_version":1}