{"id":"ce5247f1-6e76-4c34-af8b-c6447417cdc0","arxiv_id":"2607.20459","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-timescale reasoning controller with explicit verification and repair lifts multi-hop QA accuracy across multiple LLM backbones.","lead":"THIS PAPER INTRODUCES THOR, A FRAMEWORK FOR MULTI-HOP QUESTION ANSWERING THAT SEPARATES LONG-RANGE PLANNING FROM PER-HOP RETRIEVAL AND ADDS A VERIFICATION-AND-REPAIR CONTROL LOOP. IF THE REPORTED GAINS HOLD, IT OFFERS A PRACTICAL PLUG-AND-PLAY WRAPPER THAT IMPROVES MULTI-HOP QA ACCURACY ACROSS SEVERAL LARGE LANGUAGE MODELS.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's headline comparison is load-bearing but unverified: baselines are not shown to share THOR's backbone, retriever, split, or judge; BELLE's F1<EM flags the table's reliability. Matched re-runs are needed before claiming superiority over prior methods.","rationale":"The reader's weakest assumption is exactly the one I find most load-bearing: Table 1's baseline comparability is unverified. The central claim of the paper has two parts: (1) THOR improves over its raw backbone, and (2) THOR outperforms existing multi-hop QA methods. Part (1) is well-supported by matched backbone adaptation and ablation experiments, especially the controlled 'same modules, different control' experiment in Section 6.1 (Table 7), which is strong internal evidence that the framework's control protocol matters. Part (2), however, depends on Table 1, and the paper provides no evidence that the baseline numbers were produced under conditions comparable to THOR. The BELLE MuSiQue F1 < EM inconsistency is a concrete red flag that the table is not internally reliable. I also note a secondary limitation: Table 6 shows substantial degradations for some reasoning backbones (e.g., O1 on MuSiQue drops from 41.7 to 39.1 EM and 55.1 to 49.4 F1), which is not discussed in the paper and qualifies the 'generalization across different backbones' claim. That said, the main superiority claim over prior methods is the one whose evidence is weakest. Because the concern is addressable with matched re-runs and the reader already made the verdict conditional, I recommend keeping the CONDITIONAL verdict rather than moving to ACCEPT or REJECT. The concern is real but not a demonstrated fatal flaw; it is a missing reproducibility constraint.","tokens_in":25436,"tokens_out":5189,"duration_ms":46632,"concrete_test":"Re-run Table 1 under matched conditions for the strongest baseline per category (e.g., CoRAG, ComposeRAG, ReAgent, KAG): same GPT-3.5-turbo backbone, BGE-M3 + BGE-Reranker, same 3-topic + 3-context evidence budget, same MuSiQue/HotpotQA/2Wiki split and number of test instances, same answer normalization and EM/F1 script, no oracle evidence. Also recompute BELLE's MuSiQue row and audit how each Table 1 baseline was produced (original paper vs local run, backbone, retriever, subset). If head-to-head EM/F1 gaps shrink to noise or reverse, the 'outperforms prior methods' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that THOR 'outperforms the vast majority of methods' rests on Table 1. The paper never states whether any baseline was re-run under THOR's exact conditions: GPT-3.5-turbo backbone, BGE-M3/BGE-Reranker retrieval, the same test split and subset size, the same answer normalization, and the same EM/F1 judging script. Section 5.1 specifies only THOR's implementation; Appendix C.2 gives conceptual descriptions, not experimental conditions. Published baseline numbers typically use different backbones (often GPT-4 or specialized retrievers), different subsets (especially on MuSiQue), and different evaluation pipelines. Those differences alone can produce or erase the reported EM/F1 gaps. The internal inconsistency in Table 1 (BELLE MuSiQue F1 42.1 < EM 50.5, impossible for a single prediction set) is a concrete indicator that the table is not a clean, auditable artifact. The controlled add-on experiment in Section 6.1 (Table 7) does provide matched evidence that THOR's control protocol helps over a same-module planner–executor loop, so the internal mechanism may survive, but the claim of superiority over existing multi-hop QA methods specifically depends on unmatched external numbers. This is an addressable comparability and reproducibility gap, not a theoretical flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"THOR is a closed-loop, brain-inspired reasoning wrapper for multi-hop QA. It alternates between a global 'theta' planner (iPFC), a local 'gamma' executor/retriever (iHPC), and a verifier (iACC), with a slot-schema memory and a finite-state controller that decides between Continue, Retrieve, Repair, and Replan. The authors claim that this design mitigates attention decay and error accumulation, and they report large EM/F1 gains over the plain GPT-3.5-turbo backbone on HotpotQA, 2WikiMultiHopQA, and MuSiQue, plus a backbone-adaptation study across eight regular and five reasoning LLMs. They also report FSR/ASR 'attention decay' metrics, an adversarial document-injection experiment, retrieval recall comparisons, and an accuracy-cost frontier. The main results are accompanied by ablations and a controlled add-on experiment, but the headline comparison to prior systems rests on baseline numbers whose experimental conditions are not documented.","tokens_in":25818,"tokens_out":3682,"duration_ms":34614,"significance":"If the central empirical claims hold, THOR would be a practically useful, largely model-agnostic reasoning wrapper: its controlled add-on experiment (Section 6.1, Table 7) isolates the contribution of finite-state control plus typed slot-schema memory, and the external EM/F1 and adversarial-drop benchmarks are appropriate targets. The paper also ships code, reports a human-reliability check for FSR (Table 10), and includes a broad backbone sweep — these are genuine strengths. The main reservation is that the strongest headline claim ('outperforms the vast majority of methods') depends on a baseline table whose provenance and comparability are not established. The mechanism-specific FSR/ASR evidence is also, in part, circular because the metrics are defined in terms of the same anchor/type/support checks the verifier is designed to optimize. These are addressable but load-bearing issues.","major_comments":[{"comment":"The head-to-head comparison against 27 prior methods is not auditable. The paper never states whether any baseline was re-run under THOR's exact conditions: backbone (GPT-3.5-turbo), retriever (BGE-M3/BGE-Reranker), evaluation split and subset size (especially MuSiQue), answer normalization, and the same EM/F1 judging script. Section 5.1 describes only THOR's implementation, and Appendix C.2 gives conceptual descriptions of each method without experimental conditions. An internal red flag is the BELLE row for MuSiQue: F1=42.1 with EM=50.5, which is impossible for a single prediction set and suggests the table mixes numbers from heterogeneous sources. Please either rerun all baselines under matched conditions, or clearly source each number and provide a comparability table. Without this, the superiority claim over prior methods is not supported.","section":"§5.2, Table 1; Appendix C.2"},{"comment":"The FSR/ASR evidence for 'attention decay mitigation' is partially circular with THOR's own verification checks. ASR is essentially the negation of C_ei from Algorithm 2 ('anchor mention in evidence'), and FSR is judged by a GPT-4o rubric for sub-question alignment with gold decomposition, which is closely related to the type/constraint checks C_tau and C_r. Since iACC explicitly rejects anchor-missing evidence and drives repair, low ASR is to some extent guaranteed by the construction. The authors report human agreement only for FSR (Table 10), not for ASR. To make the mechanistic claim non-circular, please (i) report FSR/ASR with human-validated rubrics on both metrics, (ii) show that FSR/ASR improvements predict EM/F1 gains beyond the effect of retrieval quality, or (iii) rely on the external adversarial and EM/F1 results as the primary evidence and present FSR/ASR as descriptive diag","section":"§5.1, §5.2.2, Algorithm 2"},{"comment":"No variance or statistical testing is reported anywhere. In Tables 2, 3, 6, and 7, every number is a single run. Several comparisons are within 1–2 EM/F1 points (e.g., Table 6: Qwen-3-Thinking 2Wiki F1 72.9→73.0; Table 7: FSR 13.1 vs 12.1), which is well within typical LLM stochasticity unless explicitly controlled. Please report means and standard deviations over at least 3–5 seeds (or fixed seeds plus sensitivity analysis), and run paired significance tests for the main backbone and the add-on comparison. At minimum, the ablation and backbone claims should not rely on differences that are smaller than expected run-to-run variance.","section":"§5.2, Tables 2, 3, 6, 7"},{"comment":"The claim that THOR 'consistently yields large gains' and 'generalizes' is contradicted by several cells in the authors' own table. Concretely: DeepSeek-R1 2Wiki F1 drops from 70.7 to 69.2; Gemini-2.5-Pro 2Wiki F1 drops from 82.9 to 82.3; O1 MuSiQue drops from 41.7 to 39.1 EM and from 55.1 to 49.4 F1. There are also unexplained numeric differences with Table 2 for the same GPT-3.5-turbo baseline (EM 31.9 vs 31.2). These patterns weaken the 'plug-and-play wrapper' story. Please report per-backbone wins/losses, discuss cases where THOR hurts, and consider whether the generalization claim should be softened to 'most backbones'.","section":"§5.2.5, Table 6"}],"minor_comments":[{"comment":"The THOR row for 'Drop' is printed as '12.79.3', which should be '12.7 9.3'. Please format the two columns clearly.","section":"Table 4"},{"comment":"The symbol 'I' is used for an indicator function but is not defined. Also 'δ_i ← I(Insuff(E_topic))' is unclear: what exactly does iACC return, and how is 'insufficient' operationalized? Please define these variables and the threshold.","section":"Algorithm 1"},{"comment":"'fixedLLM judge' has a missing space; please proofread formulas and prose for spacing errors (e.g., '0.7 9.8' in Table 3, 'THOR-Iter@273.1' in Table 5 caption).","section":"§5.1"},{"comment":"The accuracy-cost discussion would be clearer with the exact token/call/latency numbers from Table 11 referenced in the main text, and with a statement on whether the cost includes the verification calls or only generation tokens.","section":"§5.2.6 / Figure 4"},{"comment":"The FSR judge uses the gold decomposition and evaluates only the first h predicted hops. This is reasonable, but please state explicitly in the main text that FSR/ASR are computed on MuSiQue only, since the main text's 'all three benchmarks' phrasing in §5.2.2 is ambiguous.","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The controlled add-on experiment (Table 7) gives the paper a credible core: when the planner/retriever are held fixed, the finite-state control plus typed memory helps. My main concern is that the paper's strongest advertised result — superiority over the field in Table 1 — is built on unmatched baseline numbers, with at least one impossible cell (BELLE MuSiQue). If the authors cannot reproduce the baselines under matched conditions, I would advise the editor to require that the paper be reframed around the controlled comparisons and the backbone-adaptation study, and that the 'vast majority of methods' claim be removed or explicitly qualified as 'when numbers are taken from original papers under possibly different conditions.' The FSR/ASR circularity is secondary but should be fixed to avoid over-interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the actual mechanism is better supported than the marketing. THOR is a two-timescale controller for multi-hop QA: a slow theta loop maintains a typed slot-schema frame, a fast gamma loop does retrieval and verification, and a discrete state machine (continue/retrieve/repair/replan) decides when to escalate. None of these pieces is new individually, but the integration is clean and the ablations are honestly designed. The strongest evidence is the controlled add-on experiment in Table 7, which holds planner and retriever fixed and varies only the control protocol and the memory representation. That isolates the contribution of the controller and the typed frame, and it shows both help. The backbone adaptation table is also a real asset: consistent EM/F1 lifts over plain backbones across thirteen models, which supports the plug-and-play wrapper claim.\n\nThe soft spot is the headline comparison in Table 1. The paper never says whether any baseline was re-run under THOR's conditions (same backbone, retriever, split, judging script). The stress-test note is right: the BELLE row has F1 < EM, which is impossible for a single prediction set and strongly suggests the table was assembled from heterogeneous sources with different evaluation pipelines. That is a load-bearing flaw for the specific claim that THOR outperforms prior multi-hop QA methods. It is not fatal to the framework, because the controlled experiment and backbone adaptation stand on their own, but the authors need to rerun or clearly identify matched baselines before claiming superiority.\n\nThe FSR/ASR metrics also have a circularity problem: their definitions (anchor mention, type alignment, evidence support) mirror the iACC verifier checks in Algorithm 2, so THOR will naturally win on them. That is not an independent validation of attention-decay mitigation; it is evidence that the framework is aligned with its own verifier, which is useful but not the same thing. The human reliability check in Appendix D validates the FSR judge against two annotators, which is good, but it does not validate the construct.\n\nMinor issues: no variance or significance testing anywhere, and the main results are single runs. The cost-latency appendix has oddities (THOR Imax=5 F1 lower than Imax=3) but those are plausible.\n\nOverall: the paper is worth engaging with. It is a well-thought-out system with reproducible code and a clear failure-analysis section. The central mechanism may hold, but the comparative claims need matched re-runs and statistical reporting. I would send it to peer review with the demand that Table 1 be audited. If the matched numbers hold, this is a solid systems contribution; if not, the controlled experiment alone is still worth publishing as a smaller but credible result.","headline":"A well-engineered agentic wrapper with solid internal ablation, but the headline comparison against prior work rests on unverified numbers; the core mechanism deserves closer study.","tokens_in":26298,"tokens_out":1995,"would_cite":true,"duration_ms":20878,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"THOR — a theta–gamma control loop with slot-schema memory and verifier-gated repair — more than doubles exact-match accuracy on MuSiQue and suppresses drift and compounding errors in multi-hop QA","keywords":["multi-hop question answering","theta-gamma oscillation","attention decay","error accumulation","verification and repair","slot-schema memory","retrieval-augmented generation","LLM reasoning"],"falsifier":"Re-run THOR and each baseline in the main comparison with one fixed backbone, the same retriever, the same evidence budget, and the same answer-matching judge. If THOR's MuSiQue exact-match advantage over the strongest agentic baseline nearly vanishes, the claim that the framework itself — rather than backbone or retriever differences — drives the improvement is falsified.","tokens_in":25337,"feed_emoji":"🧠","tokens_out":12141,"duration_ms":94675,"temperature":0.7,"pith_summary":"Multi-hop question answering fails in two recurring ways, this paper argues: the model's attention drifts as the reasoning chain grows, and small early mistakes get committed and compounded. THOR is a proposed fix that organizes reasoning as a two-timescale loop: a slow 'theta' planner maintains the global reasoning frame in a typed slot-schema memory, while a fast 'gamma' executor retrieves evidence, answers one hop, and runs three verification checks. The check results drive explicit state transitions — continue, retrieve, repair, replan — so errors are caught and corrected by a controller instead of silently accumulating. On HotpotQA, 2WikiMultiHopQA, and MuSiQue, THOR reports large gains over a GPT-3.5-turbo baseline, including exact match moving from 19.7 to 48.5 on MuSiQue, with lower frame-shift and anchor-shift rates that persist at four hops. If the central claim holds, multi-hop accuracy can be improved primarily through the control loop and structured memory, not just through a stronger underlying model.","feed_headline":"Theta–gamma loop doubles exact-match on hardest QA benchmark","feed_subtitle":"A slow-fast controller with verifier-gated repair lifts GPT-3.5-turbo from 19.7 to 48.5 exact-match on MuSiQue.","key_machinery":"The load-bearing mechanism is a two-timescale finite-state controller inspired by theta–gamma phase-amplitude coupling. The 'theta' outer loop is a planner (iPFC) that owns a slot-schema memory — a typed frame in which each hop binds named slots (sub-question, core entity, expected answer type, sub-answer, completion flag) to explicit values. The 'gamma' inner loop is an executor/verifier pair: iHPC retrieves evidence using topic-top-3 and context-top-3 fallback strategies plus replay-based query refinement, and iACC checks whether the candidate answer anchors on the intended entity, matches the expected type, and is supported by the evidence. The controller converts that three-bit check vec","core_discovery":"THOR's central claim: multi-hop failures are control failures — drift plus compounding — correctable by an explicit two-timescale loop. A slow theta planner (iPFC) maintains a slot-schema frame (sub-questions, entities, answer types); a fast gamma executor (iHPC) retrieves and proposes a sub-answer; a verifier (iACC) emits three binary checks (anchoring, type, support). These drive a finite-state controller: continue, retrieve, repair, or replan. With GPT-3.5-turbo on MuSiQue, THOR lifts exact match from 19.7 to 48.5, reduces 4-hop frame-shift to 15.3% and anchor-shift to 10.4%, and degrades less under adversarial injection. Ablations show every component matters; removing the planner costs","pith_inferences":["A testable extension: replace the LLM planner and verifier with a rule-based planner and a cheaper verifier; if the anti-drift effect survives, per-question token cost could drop well below the reported figures.","The controller is essentially a deterministic protocol over typed signals, so the same design could transfer to other sequential tasks with explicit constraints — multi-step tool use, database query composition, or theorem proving — where the failure mode is also silent drift rather than missing information.","A stress-test for any future multi-hop system falls out of the adversarial protocol: inject a document that preserves the surface answer pattern but breaks the entity bridge, and measure the relative drop in exact match; this directly probes error accumulation independent of retrieval quality.","Because the paper's frame-shift and anchor-shift metrics rely on LLM and lightweight judges, an extension would be to publish the judged annotations and measure how sensitive the reported drift reductions are to judge choice — the framework's robustness claim depends on those metrics being reproducible."],"forward_implications":["A given LLM can become a much stronger multi-hop solver without changing the model: THOR reports exact-match gains of roughly 2× on MuSiQue for GPT-3.5-turbo and consistent gains across eight regular and five reasoning backbones.","Ablation evidence implies the four framework components are complementary: removing the planner costs 20.1 exact-match points on MuSiQue, while removing the retriever, verifier, or slot-schema memory each costs several points on all three benchmarks.","The anti-drift benefit grows with chain depth: THOR's frame-shift and anchor-shift rates are lowest at 4 hops, where chain-of-thought, chain-of-retrieval, and reversible-agent baselines drift most.","Iterative retrieval refinement is a concrete source of the gains: with up to three per-hop retries, recall@15 on MuSiQue rises from 36.4 to 44.5, surpassing a chain-of-retrieval baseline.","The retry budget Imax provides an explicit accuracy–cost dial: Imax=3 gives the best accuracy, and Imax=1 already achieves a better accuracy–token trade-off than a reversible multi-agent baseline."],"fun_headline_variants":["Theta-gamma loop lifts GPT-3.5-turbo from 19.7 to 48.5 exact match on MuSiQue","Verifier-gated repair loop cuts multi-hop error accumulation in QA","Brain-inspired two-timescale reasoning beats attention decay in multi-hop QA","THOR: slow theta planning plus fast gamma retrieval lifts QA accuracy","Explicit control loop fixes compounding errors in multi-hop QA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline comparison assumes every baseline in the main table was evaluated under identical conditions — same backbone LLM, same retriever, same evidence budget, same evaluation split, and same answer-judging protocol — as THOR; if the baseline numbers were taken from their original papers under different conditions, the reported gains are not purely from the framework.","fun_headline_variants_meta":{"raw":{"variants":["Theta-gamma loop lifts GPT-3.5-turbo from 19.7 to 48.5 exact match on MuSiQue","Verifier-gated repair loop cuts multi-hop error accumulation in QA","Brain-inspired two-timescale reasoning beats attention decay in multi-hop QA","THOR: slow theta planning plus fast gamma retrieval lifts QA accuracy","Explicit control loop fixes compounding errors in multi-hop QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1475,"prompt_tokens":713,"completion_tokens":762,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":667}},"tokens_in":457,"tokens_out":762,"duration_ms":7063,"temperature":1.0,"reasoning_tokens":667,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:54:02.880300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run THOR and each baseline in the main comparison with one fixed backbone, the same retriever, the same evidence budget, and the same answer-matching judge. If THOR's MuSiQue exact-match advantage over the strongest agentic baseline nearly vanishes, the claim that the framework itself — rather than backbone or retriever differences — drives the improvement is falsified.","supporting_citations":[],"review_version":1}