{"id":"024f2ca5-25da-45fe-a467-e4ddd70fddf1","arxiv_id":"2501.16350","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A decomposition-based Persian KGQA method and a new 5,600-question decomposition dataset improve F1 from 62.98% to 75.55% on PeCoQ.","lead":"Researchers built a Persian question answering pipeline that splits complex multi-hop questions into simpler pieces, converts each piece into a SPARQL query over the FarsBase knowledge graph, and runs the queries in sequence. On the PeCoQ benchmark it reports gains of about 12.6% in F1 and 12.1% in accuracy over the only prior Persian system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gain may rest on an undisclosed overlap between the MRDCPQ training data and the PeCoQ test set; the paper never states how its 80/10/10 split relates to the PeCoQ split used in Table 4.","rationale":"The reader's weakest assumption (semantic equivalence of decomposition) is real and important: with decomposition accuracy at 77.61%, the sequential-execution pipeline cannot be guaranteed to preserve answers on the remaining 22%. However, the assumption I would stress-test first is evaluation validity. The empirical gain is the only quantitative support for the method, and it depends entirely on PeCoQ test questions being unseen. Because MRDCPQ was created from PeCoQ and used to train every module, a missing disjointness statement is a concrete, checkable threat. If the splits overlap, the reported comparison is not a comparison. The reader already flags the lack of a 'defined, un-curated test protocol'; my concern sharpens this into a specific overlap test. I would keep the reader's CONDITIONAL verdict, with the added condition that the split mapping be disclosed before the headline claim is accepted.","tokens_in":12206,"tokens_out":5941,"duration_ms":61185,"concrete_test":"Ask the authors to release, or to report, the exact mapping between PeCoQ question IDs and MRDCPQ split assignments. Compute the intersection of the MRDCPQ training split with the PeCoQ test split used in Table 4, and also count whether Table 4 covers all PeCoQ test questions or only the 5,600 curated MRDCPQ questions. If the intersection is non-empty or the evaluation is restricted to the curated subset, the claimed 12.57% F1 improvement is not a valid comparison; if the intersection is empty and the test set is complete, the leakage concern is resolved and attention shifts to the semantic-equivalence assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (abstract; §4.4, Table 4) is that the four-component pipeline outperforms Etezadi et al. on the PeCoQ test set by 12.57% F1 and 12.06% accuracy. For this claim to hold, the PeCoQ test questions must be unseen by every trained component. The paper does not establish this. MRDCPQ is built 'based on the PeCoQ dataset' (§3.1.1), and the mT5 decomposer, ParsBERT NER, and mT5 SPARQL generator are all fine-tuned on MRDCPQ splits (80/10/10). The evaluation is then reported 'on the test dataset of PeCoQ' (§4.4) without saying whether those test questions are disjoint from the MRDCPQ training split, or whether Table 4 is computed on all PeCoQ test questions or only on the 5,600 'correctly decomposed' questions. If an overlap exists, the large deltas can be an artifact of memorization. The provenance note that 'approximately 15,000 multi-hop complex questions were extracted from PeCoQ' is also inconsistent with PeCoQ's stated 10,000 questions, so the curation step is underspecified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a four-component pipeline for answering multi-hop complex Persian questions over the FarsBase knowledge graph: (1) decomposition of the question into BREAK-style semantic segments using a fine-tuned mT5 model, (2) named entity recognition with ParsBERT and zero-shot entity linking to FarsBase, (3) generation of one SPARQL query per segment using a fine-tuned mT5 model, and (4) sequential execution of these queries to compose the final answer. The authors introduce MRDCPQ, a dataset of 5,600 Persian multi-hop questions with semantic decompositions, derived from the PeCoQ dataset. Component-level results include task decomposition accuracy of 77.61%, NER accuracy of 99.16%, and SPARQL generation F1 of 82.35%. The main empirical claim, reported in Section 4.4, is that the full pipeline outperforms the only prior Persian KGQA system of Etezadi et al. on the PeCoQ test set by 12.57% F1 and 12.06% accuracy.","tokens_in":12523,"tokens_out":3418,"duration_ms":36769,"significance":"If the empirical claims are validated, this is a useful contribution to Persian KGQA: it provides the first Persian dataset for semantic question decomposition, demonstrates that a decomposition-based pipeline can exceed the previous state of the art on PeCoQ, and reports per-component results that are informative for future work. The strengths are the creation of MRDCPQ, the use of a standard BREAK-style decomposition formalism, and the explicit evaluation of decomposition, NER, and SPARQL generation as separate stages. However, the significance is currently limited by evaluation weaknesses and unresolved data-provenance questions that could affect the validity of the headline improvement.","major_comments":[{"comment":"The central claim of a 12.57% F1 improvement on the PeCoQ test set is not supported without a clear statement that the PeCoQ test examples are disjoint from the MRDCPQ training split. MRDCPQ is built 'based on the PeCoQ dataset' and split 80/10/10 for training, testing, and validation, while Table 4 reports results 'on the test dataset of PeCoQ.' The paper never specifies whether the PeCoQ test questions used in Table 4 are the same as, overlapping with, or disjoint from the MRDCPQ training set. It also does not state whether Table 4 is computed on all PeCoQ test questions or only on the 5,600 'correctly decomposed' questions. This is load-bearing for the headline result, and the inconsistency that 'approximately 15,000 multi-hop complex questions were extracted from PeCoQ' despite PeCoQ being described as containing 10,000 questions makes the curation step underspecified.","section":"§3.1.1, §4.4, Table 4"},{"comment":"The end-to-end evaluation uses a single baseline, reports no variance or significance tests, and gives no details about the number of test questions or the exact evaluation protocol. With a 12.57 F1-point difference over one comparator, the result could be meaningful, but the absence of standard deviations over multiple fine-tuning seeds, confidence intervals, or a significance test makes it impossible to assess whether the gap is robust. The paper should report results over multiple runs with statistical testing, and should also report end-to-end accuracy conditioned on whether the decomposition was correct, since the decomposition TDA is only 77.61% and error propagation is a stated risk of the sequential execution architecture.","section":"§4.4, Table 4"},{"comment":"The method assumes that the semantic decomposition produced in Section 3.1 is exactly semantically equivalent to the original complex question, so that sequential execution of the per-step SPARQL queries yields the same answer as a single correct query. This assumption is never verified. The TDA metric is described qualitatively, with no formula or exact matching definition, and the paper does not report how decomposition failures affect downstream entity linking, SPARQL generation, and final answer accuracy. Since 77.61% TDA means a nontrivial fraction of test decompositions are incorrect, the paper needs an error analysis tracing those failures to end-to-end answers, and ideally a comparison against a non-decomposition baseline on the same test set.","section":"§3.1, §3.4, §4.1"},{"comment":"The entity linking component is a critical part of the pipeline, yet no quantitative evaluation of entity linking accuracy is provided. Section 3.2 claims that the proposed NER method outperforms existing Persian tools, and Figure 6 describes a zero-shot linking process, but Table 3 only reports NER accuracy and the final system evaluation does not isolate linking errors. Linking errors will propagate directly into SPARQL generation and final answers, so the paper should report entity linking accuracy on the MRDCPQ test set and, if possible, an ablation that shows the contribution of the linking step to the end-to-end F1.","section":"§3.2, §4.4"}],"minor_comments":[{"comment":"The dataset description states that 'approximately 15,000 multi-hop complex questions were extracted from PeCoQ,' but the same paper describes PeCoQ as containing 10,000 questions. This inconsistency should be resolved and the exact filtering rules for the 5,600 selected questions should be stated.","section":"§3.1.1"},{"comment":"Reference [25] is cited as FarsBase in Section 3.1.1, but [25] is the Freebase paper; FarsBase is correctly cited as [31] and [32] elsewhere. The incorrect citation should be fixed.","section":"§3, References"},{"comment":"The text in Section 3.2 describes entity linking using cosine similarity, while Figure 6 mentions a cross-encoder and a ranking step. These are two different procedures; the paper should describe one consistent linking method and explain what is actually implemented.","section":"§3.2, Figure 6"},{"comment":"Table 4 lacks information about the number of test questions, whether the same questions were used for both methods, and whether the reported numbers are averages over runs. Adding this information would make the comparison interpretable.","section":"§4.4, Table 4"},{"comment":"The English language needs another round of editing: examples include 'Figures 8 demonstrates' in Section 3.4, and several Persian-to-English translations such as 'جهانن' and 'کد منطقهای' appear inconsistent with the English glosses. Typos and grammatical errors do not affect the technical content but should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The data-overlap concern is the most serious issue and should be resolved before the paper is considered further. If the authors can demonstrate disjointness between the MRDCPQ training split and the PeCoQ test questions, and add statistical rigor, the paper could be a useful contribution to Persian KGQA. The current lack of dataset release and weak baseline comparison make external validation difficult."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution to a small field, and the main empirical claim is plausible but not yet established. The paper builds MRDCPQ, apparently the first Persian dataset for decomposing complex questions into semantic steps (5,600 questions), and evaluates a four-stage pipeline (decompose, NER+link, SPARQL generate, execute) on PeCoQ/FarsBase, reporting +12.57 F1 over Etezadi et al. The components are adaptations of known methods, but the combination for Persian KGQA is new, and the decomposition dataset alone is worth something if released.\n\nWhat's good: the task is underserved; the pipeline is clearly described; decomposing before SPARQL generation is a reasonable design; and reporting TDA, NER accuracy, SPARQL F1, and end-to-end metrics gives component-level visibility. The internal numbers (TDA 77.61%, SPARQL F1 82.35%, end-to-end 75.55/74.81) are mutually consistent with error propagation.\n\nThe soft spots are real. First, the evaluation may leak. MRDCPQ is built from PeCoQ, and the mT5 decomposer, ParsBERT NER, and SPARQL generator are all fine-tuned on MRDCPQ splits. The paper says Table 4 is on 'the test dataset of PeCoQ' but never states whether those PeCoQ test questions are disjoint from MRDCPQ training. If the MRDCPQ training split includes any PeCoQ test questions, the 12.57-point delta is partly memorization. The paper also says ~15,000 multi-hop questions were extracted from PeCoQ, but PeCoQ has 10,000; that discrepancy is unexplained and makes the curation step underspecified. Second, the evaluation uses one baseline, no variance or significance, and it is unclear whether Table 4 covers all PeCoQ test questions or only the 5,600 'correctly decomposed' ones. A curated subset would invalidate the comparison. Third, the semantic-equivalence assumption—that decomposed steps executed sequentially reproduce the original question's answer—is load-bearing, and with TDA at 77.61%, roughly a quarter of decompositions are wrong; there is no error analysis connecting those failures to final F1.\n\nThis is not a takedown. The architecture is sane, the dataset is a legitimate contribution, and the results are believable. But the paper currently does not give a reader enough to verify the headline. If the authors disclose the exact split overlap, report on the full PeCoQ test set, and add a significance check, it becomes a solid paper. As it stands, send to serious reviewers but expect a request for major revision.","headline":"Useful first Persian question-decomposition dataset, but the 12.57 F1 headline needs a clean test-split disclosure before I'd trust it.","tokens_in":13023,"tokens_out":2461,"would_cite":false,"duration_ms":23483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a decompose-then-execute pipeline for Persian multi-hop questions raises F1 to 75.55% and accuracy to 74.81% on the PeCoQ benchmark, beating the only prior Persian KGQA system by 12.57 and 12.06 points.","keywords":["knowledge graph question answering","multi-hop question answering","Persian question answering","question decomposition","SPARQL generation","FarsBase","PeCoQ dataset","named entity linking"],"falsifier":"Replace the predicted decompositions on the PeCoQ test set with the gold decompositions from MRDCPQ and rerun the rest of the pipeline unchanged. If the F1 score stays near 75.55%, decomposition errors are not the binding constraint; if it rises substantially, the claimed end-to-end gain depends on the 77.61%-accurate decomposer and would shrink for questions the decomposer gets wrong.","tokens_in":12017,"feed_emoji":"🧩","tokens_out":7097,"duration_ms":65400,"temperature":0.7,"pith_summary":"This paper tries to establish that Persian multi-hop questions over a knowledge graph can be answered by decomposition: split the question into a sequence of small semantic units, link the entities in each unit to the FarsBase knowledge graph, generate one SPARQL query per unit, and execute the queries in order. To do this the authors built MRDCPQ, a 5,600-question Persian dataset of complex questions paired with their step-by-step semantic decompositions, and fine-tuned multilingual and Persian transformer models for each stage. On the PeCoQ test set the complete system scores 75.55% F1 and 74.81% accuracy, beating the only prior Persian KGQA system by 12.57 and 12.06 points respectively. A reader should care because this supplies the first public Persian dataset for question decomposition and a complete recipe for a language that previously had a single comparable KGQA system.","feed_headline":"Decompose-then-query lifts Persian KGQA by 12.6 F1","feed_subtitle":"Step-by-step question decomposition turns Persian multi-hop queries into sequential SPARQL calls, lifting accuracy 12 points on PeCoQ.","key_machinery":"The load-bearing object is the Meaning Representation of Decomposed Complex Persian Question (MRDCPQ), a numbered list of short semantic units in which a complex question is rewritten as a sequence of entity and relation steps with back-references such as '#1' and '#2'. The paper's architecture has four components: a multilingual text-to-text decomposer fine-tuned on the 5,600-question MRDCPQ dataset, a Persian transformer named-entity recognizer and zero-shot entity linker that maps entities to FarsBase, a multilingual text-to-text unit-to-SPARQL generator, and an executor that chains the generated queries so the object of one hop becomes the subject of the next. The MRDCPQ representation is what makes sequential execution possible: it turns multi-hop reasoning into a fixed executable order rather than a single monolithic query.","core_discovery":"Stated on the paper's own terms, the discovery is that the bottleneck in Persian KGQA is the transformation of multi-hop questions into a semantically equivalent SPARQL query, and that this transformation becomes tractable when the question is first rewritten as a chain of meaning units. Each unit is either an entity or a relation, and later units refer back to earlier ones with numbered markers, so the chain is executable: the answer of one unit feeds the next. Trained on the MRDCPQ dataset, the decomposition model reaches 77.61% task-decomposition accuracy, the entity recognizer reaches 99.16% accuracy, and the unit-to-SPARQL generator reaches 82.35% F1; end-to-end, the system outperforms the only prior Persian KGQA approach on precision (84.36% vs 71.24%), recall (68.41% vs 56.45%), F1 (75.55% vs 62.98%), and accuracy (74.81% vs 62.75%).","pith_inferences":["The 77.61% decomposition accuracy is probably the effective ceiling on end-to-end accuracy: if a step drops or reorders a relation, every downstream hop inherits the error, so improving decomposition is the highest-leverage intervention even though the paper does not report this ablation.","Ablating the system with oracle decompositions from MRDCPQ would separate the contribution of the decomposer from the contributions of entity linking and SPARQL generation; the paper does not report this experiment.","The 12.57-point F1 gap is measured against a single prior system; LLM-based KGQA methods, which the paper reviews for English, have not yet been evaluated on Persian PeCoQ and would be a natural next comparison.","The MRDCPQ format mirrors the BREAK-style question-understanding benchmark, so the decomposition dataset could support other Persian tasks such as semantic parsing and explainable QA, not only KGQA."],"forward_implications":["Any Persian KGQA system can treat question decomposition as a separate trainable component, because MRDCPQ provides the first benchmark for it.","Because the units are human-readable, the system's reasoning path is inspectable: a wrong answer can be traced to the exact hop that produced it.","The per-component results suggest the NER and entity-linking stages are near-saturated for this benchmark, so future gains on PeCoQ will likely come from decomposition and SPARQL generation.","The same decompose-then-execute pipeline could be applied to other low-resource languages with a multilingual transformer, provided a knowledge graph and an entity-linking resource exist."],"supporting_citations":[{"why":"Supplies the PeCoQ complex questions, their SPARQL queries, and their relations; it is the benchmark and the source of the 5,600 MRDCPQ questions.","marker":"[34]"},{"why":"The only prior Persian KGQA system on PeCoQ and the baseline whose precision, recall, F1, and accuracy are compared.","marker":"[23]"},{"why":"Defines FarsBase, the Persian knowledge graph used for entity linking and SPARQL execution.","marker":"[4]"},{"why":"Provides the BREAK standard that the MRDCPQ dataset follows for semantic question decomposition.","marker":"[36]"},{"why":"Supplies the multilingual text-to-text transformer fine-tuned for question decomposition and for MRDCPQ-to-SPARQL generation.","marker":"[37]"},{"why":"Supplies the Persian transformer model fine-tuned for named entity recognition and used for dense entity representations in linking.","marker":"[38]"},{"why":"Provides the zero-shot dense entity-linking method adapted to retrieve and rank candidate entities from FarsBase.","marker":"[39]"}],"fun_headline_variants":["Persian KGQA: decompose multi-hop into chains, +12.6 F1","Decompose-then-query beats Persian KGQA baseline by 12.6","Chain decomposition turns Persian multi-hop into SPARQL, +12.6 F1","Persian KGQA: stepwise decomposition lifts F1 by 12.6 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's decomposition of a complex Persian question is semantically equivalent to the original question, so executing the decomposed steps in sequence returns the same answer as one correct SPARQL query; the paper's own 77.61% decomposition accuracy shows this equivalence fails on a nontrivial fraction of questions.","fun_headline_variants_meta":{"raw":{"variants":["Persian KGQA: decompose multi-hop into chains, +12.6 F1","Decompose-then-query beats Persian KGQA baseline by 12.6","Chain decomposition turns Persian multi-hop into SPARQL, +12.6 F1","Persian KGQA: stepwise decomposition lifts F1 by 12.6 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000569,"raw_usage":{"total_tokens":2704,"prompt_tokens":970,"completion_tokens":1734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1643}},"tokens_in":586,"tokens_out":1734,"duration_ms":12346,"temperature":1.0,"reasoning_tokens":1643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:56:00.872748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the predicted decompositions on the PeCoQ test set with the gold decompositions from MRDCPQ and rerun the rest of the pipeline unchanged. If the F1 score stays near 75.55%, decomposition errors are not the binding constraint; if it rises substantially, the claimed end-to-end gain depends on the 77.61%-accurate decomposer and would shrink for questions the decomposer gets wrong.","supporting_citations":[{"cited_title":"PeCoQ: A Dataset for Persian Complex Question Answering over Knowledge Graph,","cited_arxiv_id":null,"evidence_quote":"Supplies the PeCoQ complex questions, their SPARQL queries, and their relations; it is the benchmark and the source of the 5,600 MRDCPQ questions."},{"cited_title":"A Knowledge-based Approach for Answering Complex Questions in Persian","cited_arxiv_id":"2107.02040","evidence_quote":"The only prior Persian KGQA system on PeCoQ and the baseline whose precision, recall, F1, and accuracy are compared."},{"cited_title":"ParsBERT: Transformer-based Model for Persian Language Understanding","cited_arxiv_id":"2005.12515","evidence_quote":"Supplies the Persian transformer model fine-tuned for named entity recognition and used for dense entity representations in linking."}],"review_version":1}