{"id":"a448734a-509b-4bda-86fc-8e794674ae24","arxiv_id":"2509.04770","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Multi-hop question decomposition yields a marginal accuracy edge over direct answering on MQUAKE-T with LLAMA3, both before and after LoRA fine-tuning, but the gains are not statistically validated.","lead":"This paper tests whether breaking complex questions into smaller ones improves a language model's answers on the MQUAKE benchmark. The observed accuracy gains are tiny, not statistically tested, and the multi-hop condition may receive the answer path in its input.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multi-hop condition is not a test of decomposition: it gives the model the sub-question answer chain in the input, so the Table 2 gap may be answer leakage rather than improved reasoning.","rationale":"The reader's weakest_assumption correctly identifies the main flaw, and I agree. The paper's strongest claim is that decomposition improves complex-question answering before and after LoRA fine-tuning. For that claim to be true, the only difference between conditions should be whether the model is asked to reason through sub-questions; instead, Section 3.1 shows the multi-hop condition receives the intermediate question-answer chain itself. That is a load-bearing design error, because it directly predicts the observed direction: models are good at reading a provided chain and reproducing the last answer, so even a small consistent gap is expected without any improvement in multi-hop reasoning. The tiny effect sizes and absence of repeated trials or statistical testing make it worse, but they are secondary: the confound is sufficient to undermine the central claim. The proposed ablation—randomizing the supplied intermediate answers—settles the question empirically because it isolates whether the model's accuracy depends on the content of the chain. If it does, the 'multi-hop method' is not being tested. The verdict should remain REJECT: the abstract's 'significantly outperforms' and 'consistently maintained its superiority' are not established by the reported experiment.","tokens_in":5196,"tokens_out":5068,"duration_ms":47713,"concrete_test":"Take the exact converted multi-hop test items and run one ablation: keep the question text and sub-question text unchanged but replace every intermediate answer in INPUT/history with a wrong or random entity from the same knowledge graph, then re-run inference with the same untrained and LoRA-fine-tuned LLAMA3 checkpoints and measure accuracy. If the high accuracy (89–90%) collapses, the reported advantage depends on being given the answer path, confirming leakage. If accuracy is unchanged, the chain is not the source of the gap, and the concern would need revision; in that case also check whether the gold answer or its alias appears verbatim in the multi-hop inputs.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that multi-hop decomposition improves accuracy—is not supported because the experimental comparison is confounded at the input level. Section 3.1 describes the Alpaca-format conversion with fields INSTRUCTION, INPUT, output, and history, and states that the multi-hop dataset maps 'the original question and multi-hop chain' to INPUT and uses history to 'record the intermediate questions and answers in the multi-hop reasoning process.' The single-hop dataset is then made by 'removing multiple chains from the transformed data,' leaving only the direct complex question. Thus the multi-hop condition differs from the single-hop condition by more than decomposition: it also supplies the intermediate answers along the reasoning path. A model that reads a chain ending in or containing the target answer is doing answer extraction, not multi-hop inference, so the 4.67‰, 4.33‰, and 1.11‰ improvements in Table 2 cannot be attributed to decomposition skill. The problem is not just missing significance tests; even a large, statistically significant gap under this design would not establish the paper's claim. A proper test requires either withholding intermediate answers (letting the model generate them) or holding the information content of the two conditions equal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-hop question decomposition method for complex question answering, built on the MQUAKE-T dataset and tested with LLAMA3. The authors convert MQUAKE-T into two Alpaca-format datasets: a single-hop dataset that directly asks the complex question, and a multi-hop dataset that includes decomposition chains and intermediate question-answer pairs. They fine-tune LLAMA3 with LoRA on each format and report accuracy on the corresponding test sets, both before and after fine-tuning. The central claim is that multi-hop decomposition consistently outperforms direct answering, with accuracy improvements of 4.67‰, 4.33‰, and 1.11‰ in the three experimental configurations.","tokens_in":5382,"tokens_out":3839,"duration_ms":34921,"significance":"If the central claim were valid, the paper would provide useful evidence that explicit multi-hop decomposition improves LLM performance on knowledge-graph-based complex QA and that LoRA fine-tuning preserves this advantage. The study uses an external benchmark (MQUAKE-T) and an evaluation procedure with alias matching, and it does not fit parameters to the test data, so there is no circularity. However, the significance is severely limited because the experimental comparison is confounded: the multi-hop condition supplies the decomposition chain and intermediate answers directly in the input, while the single-hop condition does not. The reported accuracy gaps are also tiny and are not accompanied by statistical tests. As a result, the paper does not establish its central claim.","major_comments":[{"comment":"The comparison between the multi-hop and single-hop conditions is confounded at the input level. The text states that in the multi-hop dataset the original question and multi-hop chain are mapped to the INPUT field and that history is used to record the intermediate questions and answers in the multi-hop reasoning process, while the single-hop dataset is created by removing multiple chains from the transformed data. The multi-hop model therefore receives the sub-answers leading to the final answer, so the accuracy gap in Table 2 may reflect answer extraction from the provided chain rather than improved multi-hop reasoning. This is a load-bearing flaw: the abstract's claim that the multi-hop method 'significantly outperforms' direct answering is not supported. A valid test would either require the model to generate the intermediate steps itself or would need to match the information content of the two conditions.","section":"Section 3.1, Data set format transformation and slicing"},{"comment":"No statistical tests or confidence intervals are reported for the accuracy comparisons. The observed improvements are 0.46, 0.43, and 0.11 percentage points (4.67‰, 4.33‰, and 1.11‰), which are small relative to the variability one would expect from LLM inference across runs. The word 'significantly' in the abstract is therefore not justified by the reported evidence. The authors should report multiple runs, variance, and a significance test, and should also state the number of test examples used.","section":"Section 4, Table 2 and Section 3.3"},{"comment":"The references cited in the introduction do not match the reference list entries, which makes the claimed relation to prior work unverifiable. For example, reference [5] is cited as 'Chen Mengke introduced WH-CoT' but the listed [5] is a paper on ride-matching decomposition; reference [6] is cited as 'Xing Cao and Yun Liu' but the listed [6] is a distributed estimation paper; reference [7] is cited as the GPT-4 Technical Report by Achiam et al. but the listed [7] is a survey on complex question answering. Additionally, 'Mello's decomposition method' mentioned in Section 1 is not identified in the references. These citation errors undermine the novelty positioning and must be corrected.","section":"Section 1 and References"}],"minor_comments":[{"comment":"The sentence 'The training set is the part of the training set of the single-hop and multi-hop datasets cut in the previous section, and the two datasets are fed into the model separately.' is confusing and should be rewritten for clarity.","section":"Section 3.2"},{"comment":"The text refers to the 'sklearning library'; the correct name is scikit-learn. Also, the evaluation criterion says a word is correct if it agrees with the label or an alias, but the paper does not explain how a single word is extracted from the model's generation.","section":"Section 3.3"},{"comment":"The value '2 / 10' for num_train_epochs is ambiguous; the authors should clarify which value corresponds to the 'LoRA Generation 2 Training' and 'LoRA Generation 10 Training' configurations.","section":"Table 1"},{"comment":"Figure 1 is a low-resolution screenshot with cramped labels and no clear axis description; it should be redrawn as a vector graphic with legible labels and a legend.","section":"Figure 1"},{"comment":"The paper does not report the dataset size, the number of training and test examples, or the random seed used for the 70/30 split, which limits reproducibility.","section":"Section 3.1"},{"comment":"The model name 'LLAMA3' should be written as 'Llama 3' in line with the official naming convention.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The central experimental comparison is invalidated by the input-level confound described in Section 3.1, and the reported effects are too small to be meaningful without statistical testing. Although the flaw could in principle be repaired by redesigning the conditions, that would require new experiments and likely a different evaluation protocol; as written, the manuscript does not establish its main claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is not supported. Section 3.1 says the multi-hop dataset maps the original question plus the multi-hop chain into the INPUT field, while the single-hop dataset removes the chains. So the multi-hop condition receives the intermediate sub-questions and their answers, and the single-hop condition does not. A model that reads the chain is doing extraction, not multi-hop reasoning. The 0.46, 0.43, and 0.11 percentage-point gaps in Table 2 can be explained by that leakage. Even a large gap under this design would not establish the paper's claim.\n\nTo be fair, the paper is transparent about the transformation, and the evaluation with alias matching is reasonable. The conclusion also admits the gains are modest and that decomposition is manually designed. The data split is synchronized, which is good practice.\n\nThe soft spots are the usual ones. The effect sizes are tiny, there are no confidence intervals or significance tests, and the paper ships no code or data. The related work is sloppy: references [5] and [6] in the introduction are about ride-matching and distributed estimation, not question decomposition. That undermines confidence in the literature handling.\n\nWhat is new? Very little. Decomposing questions into sub-questions is a known technique, and applying it to MQUAKE-T with LoRA fine-tuning is a routine extension. The paper does not propose a new decomposition algorithm or reveal a new phenomenon. The one point of interest is that the decomposition benefit mostly disappears after enough fine-tuning (gap shrinks from 4.67‰ to 1.11‰), but the confound makes even that hard to interpret.\n\nThe stress-test note holds up. This is not a missing significance test; the comparison itself is not a fair test of decomposition. A proper test would either let the model generate the intermediate answers or give the single-hop condition the same information without the decomposition structure.\n\nI would not cite this, and I would not bring it to the reading group. A serious editor should desk reject. If the authors redo the experiment with a controlled input condition, provide code and data, and fix the citations, the result might become a usable empirical footnote, but as it stands the central claim is unproven.","headline":"The multi-hop condition receives the reasoning chain in the input, so the paper's central claim is confounded by answer leakage; the tiny gains and routine nature anyway make this a desk reject.","tokens_in":5860,"tokens_out":3349,"would_cite":false,"duration_ms":26438,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that dividing a complex question into a chain of sub-questions, and feeding that chain into the LLM, improves answer accuracy on the MQUAKE-T multi-hop benchmark both before and after LoRA fine-tuning.","keywords":["multi-hop reasoning","question decomposition","complex question answering","knowledge graph","MQUAKE","LLAMA3","LoRA fine-tuning","prompt structure"],"falsifier":"Train or test a third condition that gives the single-hop model the same intermediate sub-questions and answers as a flat context, not framed as a decomposition. If this condition matches the multi-hop format's accuracy, the reported advantage is due to added information rather than decomposition; if it does not, the decomposition framing itself is doing the work.","tokens_in":5020,"feed_emoji":"🧩","tokens_out":12075,"duration_ms":104240,"temperature":0.7,"pith_summary":"The paper tries to establish that breaking a complex question into a chain of simpler sub-questions, and showing that chain to an LLM before asking the final question, improves answer accuracy on multi-hop knowledge-graph questions. It tests this on the MQUAKE-T dataset with the LLAMA3 model, comparing direct answering against the decomposed format before and after LoRA fine-tuning. In every configuration the decomposed format scores higher, from 25.93% versus 25.47% on the untrained model to 90.44% versus 90.33% after ten training epochs. If the claim holds, explicit decomposition is a cheap and reliable way to help LLMs handle complex reasoning, and the benefit survives parameter-efficient fine-tuning.","feed_headline":"Splitting questions into hops beats direct answers in every test","feed_subtitle":"The advantage holds before and after fine-tuning, although the gap narrows with more training.","key_machinery":"The load-bearing object is the decomposition chain: the sequence of sub-questions and answers derived from the MQUAKE-T fact paths, placed in the model input ahead of the original question. This is the only material difference between the two compared formats. The MQUAKE-T dataset supplies multi-hop questions built from temporally updated knowledge-graph facts; LoRA fine-tuning updates a low-rank parameter slice of LLAMA3; and accuracy is measured by exact or alias match to the reference answer. The comparison between the two input formats is the mechanism that carries the argument.","core_discovery":"The paper reports a consistent accuracy ordering: multi-hop decomposition beats single-hop direct answering in all three experimental configurations. Before fine-tuning, the multi-hop format reaches 25.93% accuracy against 25.47% for direct answering; after two epochs of LoRA fine-tuning the numbers are 89.32% versus 88.89%; after ten epochs they are 90.44% versus 90.33%. The gap is small but consistent, and the authors conclude that structured decomposition enhances the model's logical structuring and generalization, and that the method retains its edge as the model becomes better tuned.","pith_inferences":["A natural next experiment is to hide the chain from the input and ask the model to generate its own sub-questions; that would separate the benefit of producing a decomposition from the benefit of reading one.","The decomposition chains here come from the dataset's fact paths; extending the method to automatically generated sub-questions would test whether the advantage survives without hand-built chains.","Applying the same two-format comparison to other multi-hop benchmarks and to models of different sizes would show whether the ordering generalizes beyond LLAMA3 and MQUAKE-T."],"forward_implications":["Without any fine-tuning, just showing the LLM a decomposition chain improves accuracy over direct answering, so prompt structure alone is a usable lever in zero-shot settings.","After LoRA fine-tuning, both formats improve and the decomposed format stays ahead, so the decomposition benefit combines with parameter-efficient training.","The advantage persists from epoch 2 to epoch 10, so it is not only an early-training effect.","Because MQUAKE-T questions involve temporally updated knowledge-graph facts, the result suggests decomposition can help LLMs use updated facts in multi-hop settings.","The gap narrows as training proceeds, so the marginal value of decomposition is largest when the model has seen little task-specific training."],"supporting_citations":[],"fun_headline_variants":["Multi-hop beats direct answering in every test setup","Splitting questions into hops consistently outperforms direct","Hop-by-hop question decomposition wins before and after training","Multi-hop reasoning edge persists even after fine-tuning","Decomposed questions yield higher LLM accuracy across all runs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the multi-hop chain is just a prompt-structure change, but because the chain includes the intermediate answers, the multi-hop model is also receiving extra information; if that information is what drives the accuracy gain, the decomposition explanation does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Multi-hop beats direct answering in every test setup","Splitting questions into hops consistently outperforms direct","Hop-by-hop question decomposition wins before and after training","Multi-hop reasoning edge persists even after fine-tuning","Decomposed questions yield higher LLM accuracy across all runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1744,"prompt_tokens":887,"completion_tokens":857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":781}},"tokens_in":503,"tokens_out":857,"duration_ms":7190,"temperature":1.0,"reasoning_tokens":781,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:26:47.826185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or test a third condition that gives the single-hop model the same intermediate sub-questions and answers as a flat context, not framed as a decomposition. If this condition matches the multi-hop format's accuracy, the reported advantage is due to added information rather than decomposition; if it does not, the decomposition framing itself is doing the work.","supporting_citations":[],"review_version":2}