{"id":"70979b6c-ea7c-4cb6-a1a9-dd74ccd07146","arxiv_id":"2506.07528","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A pair of reinforcement-learning-trained agents, one for reasoning and one for search, improves accuracy on multi-hop claim verification benchmarks.","lead":"HARIS is a two-agent system for checking multi-hop claims: a reasoning agent plans and asks questions, while a search agent retrieves and refines evidence, and both are trained with reinforcement learning. On three benchmarks it beats several larger GPT-4o-powered baselines, though the reported gains are modest and some tables contain errors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline superiority claim rests on single-run, 500-instance comparisons with no error bars or significance tests; several HARIS margins over the best baseline are within sampling noise, and the only same-backbone table contains an internally inconsistent copied FOLK row.","rationale":"The central claim is an empirical superiority claim, so its truth conditions are that the compared systems are measured fairly and that the measured gaps are not sampling artifacts. The second condition is the least secure: the paper gives point estimates only, on 500 items, and several margins are within a 95% binomial interval. The first condition is also weakened by the internally inconsistent FOLK row in Table 5, the only same-backbone comparison. I do not think the method is wrong or that the architecture is worthless; the ablations (single-agent versus HARIS, SFT versus HARIS) go in the right direction, and the human evaluation of the judge is genuine supporting evidence. However, those ablations are also single-run and small-sample, so they do not repair the main inferential gap. The reader's weakest assumption about pseudo-answer training signal is a legitimate secondary risk, but the empirical case fails first: if the reported numbers are within noise, we cannot tell whether training-signal issues matter. A paired bootstrap or multi-seed rerun is standard, cheap, and decisive. If intervals exclude zero on the headline HOVER settings, my concern is resolved and the conditional accept can stand; if they include zero, the paper should be revised to claim a point-estimate advantage rather than consistent superiority.","tokens_in":16114,"tokens_out":11804,"duration_ms":153815,"concrete_test":"Use the released code to obtain per-claim predictions of HARIS and the strongest non-HARIS method (e.g., QACheck) on the same 500-instance samples, then compute a paired bootstrap (10,000 resamples) of the accuracy and F1 difference for each hop setting. If any 95% bootstrap interval for a headline difference includes zero, the 'consistently outperforms' claim should be softened to a point-estimate advantage; independently, rerun FOLK with Qwen3-4B from the official repository to verify whether the Table 5 row was copied from HARIS.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim that HARIS consistently outperforms all baselines, the paper must show that the observed margins are not artifacts of a small evaluation sample or of uncontrolled baseline differences. Section 4.1 reports evaluation on 500 stratified test instances per setting, and no table reports variance, confidence intervals, multiple seeds, or significance tests. For a 500-sample binary accuracy near 0.6, the standard error of a difference between two methods is roughly 3.1 points, so a 95% interval spans about ±6 points; the headline HOVER margins over QACheck are +1.8, +2.2, and +2.0 accuracy points for 2-, 3-, and 4-hop, all within that noise. Consistency across the six reported benchmark settings is suggestive but is never tested as a joint claim. In addition, the sole same-backbone comparison (Table 5) lists FOLK Qwen3-4B results identical to HARIS for all five rows, which cannot be a genuine independent run; the controlled 4B comparison is therefore not usable as printed. These issues do not show the method is wrong, but they show that the central 'consistently outperforms' claim is stronger than the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HARIS, a hierarchical two-agent framework for multi-hop claim verification. A high-level reasoning agent constructs a verification chain and emits factual questions, while a low-level search agent performs iterative retrieval and reports evidence; both agents are trained with GRPO in an alternating, coordinated scheme, with the search agent rewarded by an LLM-as-a-Judge against GPT-4o-generated pseudo ground-truth answers. The authors evaluate on EX-FEVER, HOVER, and CHECKWHY, reporting that HARIS, using a Qwen3-4B backbone, outperforms RAG, decomposition, program-guided, FOL, and search-augmented baselines, including a 32B Search-o1 baseline, and they provide ablations on single-agent design, coordination rounds, and reward functions.","tokens_in":16373,"tokens_out":3075,"duration_ms":36414,"significance":"If the reported results are reliable, HARIS is a meaningful contribution: it explicitly models the interleaving of reasoning and search, trains both roles with outcome-based RL, and provides a human-evaluated LLM-as-a-Judge reward for search-agent training. The paper also releases code, which supports reproducibility. The conceptual separation of reasoning and search is well motivated, and the qualitative examples illustrate the intended behavior. However, the central 'consistently outperforms all baselines' claim is currently supported by single-run, 500-instance evaluations without statistical testing or error bars, and the sole same-backbone comparison contains a concrete table error. The contribution is therefore promising but not yet fully evidenced as presented.","major_comments":[{"comment":"In Table 5, the FOLK row reports F1 and accuracy values that are identical to the HARIS row for all five dataset/hop settings (e.g., HOVER 2-hops F1=69.31, Acc=69.40; EX-FEVER 2-hops F1=80.12, Acc=80.20). This cannot be a genuine FOLK run and appears to be a copy/paste error. Since Table 5 is the only direct same-backbone comparison and Section B.1 explicitly claims HARIS significantly outperforms baselines on this table, the controlled comparison is invalid as printed. The FOLK row must be corrected or removed, and the authors should verify that no other rows in Table 5 or elsewhere are affected by the same issue.","section":"Section B.1, Table 5"},{"comment":"The headline claim that HARIS 'consistently outperforms all baseline methods' is not supported by the reported statistics. Evaluation is on 500 stratified test instances per setting (Section 4.1), and no table reports variance, confidence intervals, multiple seeds, or significance tests. For a binary accuracy near 0.6, the standard error of the difference between two methods on the same 500 instances is roughly 3.1 percentage points, so a 95% interval spans about ±6 points. Several headline margins are within this range: HOVER accuracy over QACheck is +1.8, +2.2, and +2.0 for 2-, 3-, and 4-hop, and HOVER 2-hop accuracy over Search-o1 is only +0.4. The consistency across settings is suggestive, but the paper needs either error bars or significance tests across repeated runs/seeds to substantiate 'consistently outperforms'.","section":"Section 5.1, Table 1"},{"comment":"The text states that increasing coordination rounds N 'generally leads to stronger performance,' but Table 3 shows decreases from N=1 to N=3 for HOVER 2-hops (F1 69.37 to 69.31; Acc 69.40 to 69.40) and HOVER 4-hops (F1 60.87 to 59.84; Acc 61.80 to 61.00). The claim is therefore only partially supported, and the table as presented does not demonstrate a monotonic or consistent benefit of more coordination rounds. The authors should either revise the claim to acknowledge the mixed results, provide additional N values with error bars, or analyze why some settings degrade with additional coordination rounds.","section":"Section 5.2, Table 3"},{"comment":"The search-agent training signal depends on GPT-4o pseudo ground-truth answers paired with questions sampled from the reasoning agent's rollouts. The paper does not evaluate the quality of these pseudo-answers; the human evaluation in Appendix D assesses only the LLM-as-a-Judge reward, not the pseudo ground-truth generation. Since noisy pseudo-answers could cause the search agent to retrieve the wrong kind of information and alternately propagate errors into the reasoning agent, the authors should either provide a human evaluation of the synthesized QA pairs, report agreement between GPT-4o pseudo-answers and gold evidence, or include a robustness experiment using gold-derived answers where available.","section":"Section 3.3.1, Appendix A.4"}],"minor_comments":[{"comment":"Sections C.2 and C.3 have identical headings and content: both are titled 'Synthesized Training Data' and present the same Table 11. One of these sections should be removed.","section":"Appendix C"},{"comment":"The caption reads 'Comparision of calls to search agent'; this should be corrected to 'Comparison'.","section":"Figure 4 caption"},{"comment":"The text in Section 6 refers to 'another example (Figure 7)' as part of the case study, but Figure 7 appears in the appendix and is labeled as a different rollout example. The figure numbering and cross-references should be cleaned up so the reader can follow which example is being discussed.","section":"Section 6 and Figure 7"},{"comment":"The sentence 'On average it improves over 3% performance' is vague; specify the exact metric (F1, accuracy, or both) and the settings averaged over Table 4.","section":"Section 5.2, LLM-as-a-Judge vs F1"},{"comment":"The paper states that all baselines except Search-o1 use GPT-4o, while Search-o1 uses QwQ-32B-preview, and HARIS uses Qwen3-4B. This is potentially a strength, but it should be acknowledged in the main text that API-based baselines with GPT-4o are not frozen-seed deterministic, which further motivates the need for repeated evaluation runs.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The Table 5 FOLK-row duplication is a red flag for data handling; before resubmission the authors should audit all reported tables for similar copy/paste issues. The overall approach is plausible and the ablations are informative, but the statistical support for the main superiority claim is currently thin. I would not reject on the basis of the current evidence, but the revision must address the table error and provide either error bars/significance tests or a careful rephrasing of the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: HARIS is a genuine new design—two specialized agents, one for reasoning and one for search, trained jointly with GRPO—and the paper is worth engaging with. The ablation story is coherent, the code is out, the human eval of the judge is a plus, and evaluation is on external benchmarks with held-out test subsets. But the 'consistently outperforms' claim is weaker than the prose. The HOVER margins over the best baseline are a couple of F1 points, which is inside the sampling noise for 500-instance sets, and the one same-backbone table (Table 5) has a FOLK row identical to HARIS, which looks like a copy-paste error. The paper deserves a serious referee, but the headline needs more evidence before it's convincing.\n\nNew: I don't know of a prior claim-verification paper that splits reasoning and search into two agents and trains both with outcome-based RL in an alternating schedule. The ablations support the split (single-agent vs. HARIS; LLM-as-judge vs. F1 reward), the coordination-round study is informative even if the result is not monotonic, and the related work covers the recent search-RL literature (Search-o1, Search-R1, R1-Searcher). The citation pattern looks appropriate.\n\nSoft spots. Table 5 is the main problem: as printed, the direct 4B-vs-4B comparison is unusable because the FOLK row is identical to HARIS on all five settings. That is a concrete error, not an interpretation issue. The main tables have no confidence intervals, no multiple seeds, and no significance tests. On a 500-sample binary accuracy near 0.6, the SE of a difference between two methods is about 3.1 points, so a 95% interval spans roughly ±6 points; the HOVER gains over QACheck are +1.8, +2.2, and +2.0 accuracy points, all within that noise. The consistency across the six settings is suggestive but isn't a tested joint claim. Also, the claim that increasing coordination rounds N 'generally leads to stronger performance' is not fully supported by Table 3: N=3 hurts HOVER-4hop and ties HOVER-2hop. The GPT-4o pseudo-answer pipeline for search-agent training is a reasonable way to bootstrap a training signal, but it is a self-referential loop; the human eval covers the judge, not the pseudo-answers.\n\nNet: the architecture is a solid contribution to the fact-checking and agentic-RAG subfields, and the paper should go to peer review. I'd condition acceptance on fixing Table 5 and adding variance-aware reporting. The size of the improvement over strong baselines is plausible but not yet demonstrated.","headline":"A real two-agent RL design for claim verification, but the empirical headline is not yet supported by the reported numbers.","tokens_in":16908,"tokens_out":5193,"would_cite":true,"duration_ms":55248,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two cooperating agents—one reasoning, one searching—beat single-model pipelines on multi-hop claim verification.","keywords":["multi-hop claim verification","hierarchical agents","reinforcement learning","GRPO","retrieval-augmented reasoning","fact-checking","LLM agents","coordinated training"],"falsifier":"Take HARIS's trained reasoning agent and swap in a search agent trained on human-annotated question-answer pairs instead of GPT-4o pseudo-answers, keeping everything else fixed; if verification accuracy does not drop, the synthesized training signal is not what makes the method work. Alternatively, run the same coordinated training with the LLM-as-a-Judge reward replaced by exact-match F1 on gold answers: the paper's own comparison predicts a roughly 3-point average loss, so a smaller or absent drop would falsify the claim that the judge reward drives search quality.","tokens_in":15914,"feed_emoji":"🤖","tokens_out":4759,"duration_ms":50053,"temperature":0.7,"pith_summary":"HARIS claims that multi-hop claim verification is best done by splitting the job between a high-level reasoning agent and a low-level search agent that trade questions and evidence back and forth, rather than having one LLM do all retrieval and reasoning in a single pass. The paper argues that this division lets each agent specialize, with the reasoning agent deciding what is missing and the search agent iteratively refining queries through a Wikipedia corpus. Both agents are trained together with GRPO, an outcome-reward reinforcement learning method, so the collaboration itself is optimized rather than hand-coded. If the claim holds, a relatively small 4-billion-parameter model can beat much larger or more elaborate verification pipelines on the EX-FEVER, HOVER, and CHECKWHY benchmarks, especially on the hardest 3-hop and 4-hop claims.","feed_headline":"Two-agent claim checker out-verifies single-model rivals","feed_subtitle":"Separate reasoning and search agents lift F1 on 3- and 4-hop benchmarks, even with a 4B model.","key_machinery":"The load-bearing object is the two-agent coordination loop trained by Group Relative Policy Optimization (GRPO), an RL variant that normalizes rewards within groups and uses a KL penalty to keep the policy stable. Specialized tags (<question>, <search>, <result>, <report>, <verification>) create a structured protocol between the two agents, and retrieved tokens are masked from the policy-gradient loss so the model learns from its own generation. The alternating training schedule—foundation stage followed by segment-wise alternation—is what keeps the search agent aligned with the questions the reasoning agent actually asks.","core_discovery":"The central discovery is that the interleaving of reasoning and retrieval—reasoning-guided search and search-informed reasoning—can be explicitly modeled as a two-level agent system and trained end-to-end. The high-level reasoning agent constructs the verification chain and emits factual questions in <question> tags; the low-level search agent executes queries, reads top-3 retrieved passages, and reports evidence in <report> tags. Training alternates: questions sampled from the reasoning agent's rollouts, paired with GPT-4o pseudo-answers derived from dataset evidence, train the search agent, and the updated search agent then supports another round of reasoning-agent training. Across five hop settings on HOVER and EX-FEVER plus a transfer test on CHECKWHY, HARIS reports the highest F1 and accuracy among RAG, decompose-then-verify, program-guided, FOL, and search-augmented baselines, with its largest margins on 3-hop and 4-hop verification.","pith_inferences":["Beyond the paper: the same alternating question-synthesis loop should transfer to open-domain QA and multi-hop retrieval tasks, where the reward would need to be answer correctness rather than claim verdict.","Beyond the paper: because the search agent is trained on questions the reasoning agent asks, the framework should be sensitive to distribution drift in claims; a test set with different styles of claims would reveal how much of the gain is task-specific.","Beyond the paper: one could test the separation hypothesis directly by replacing the learned search agent with a fixed, non-trained retrieval policy (for example, retrieving the claim's noun phrases); if HARIS still wins, the search-agent training is not the active ingredient.","Beyond the paper: the reported Cohen's kappa of 0.81 validates the LLM judge on 150 sampled questions; scaling that human check to several hundred across adversarial, partially-supported claims would show whether the reward remains aligned on the cases the system gets wrong."],"forward_implications":["Harder multi-hop claims (3-hop and 4-hop) stand to gain the most from role separation, since HARIS's largest margins over baselines appear at those depths.","A 4B-parameter model trained this way can outperform a 32B reasoning model (Search-o1's QwQ-32B-preview) on the same verification task, suggesting that architecture and training signal matter more than raw scale.","LLM-as-a-Judge rewards for the search agent produce more informative evidence-gathering than exact-match rewards, leading to fewer search calls and higher final verification reward.","Because the interaction uses explicit question and report tags, the verification process is inspectable step by step, making the system's evidence chain interpretable.","With the same 4B base, HARIS beats supervised finetuning on every dataset and hop setting, implying that the RL coordination, not just imitation of correct answers, carries the improvement."],"supporting_citations":[{"why":"Supplies GRPO, the outcome-reward RL algorithm used to train both agents.","marker":"Shao et al., 2024"},{"why":"Provides the EX-FEVER benchmark and its human-annotated explanations used for evidence pairing.","marker":"Ma et al., 2024"},{"why":"Provides the HOVER benchmark covering 2-, 3-, and 4-hop claims.","marker":"Jiang et al., 2020"},{"why":"Search-o1 is the strongest search-augmented baseline and the comparison target showing the 32B model is beaten.","marker":"Li et al., 2025"},{"why":"Prior work supplying the question-tag rollout pattern and retrieved-token loss masking.","marker":"Chen et al., 2025"},{"why":"Search-R1 is the prior RL search-training approach that HARIS extends to cooperative agents.","marker":"Jin et al., 2025"},{"why":"ProgramFC is the program-guided baseline that HARIS outperforms.","marker":"Pan et al., 2023b"},{"why":"FOLK is the FOL-guided baseline and the source of the stratified 500-sample evaluation protocol.","marker":"Wang and Shu, 2023"}],"fun_headline_variants":["Coordinated reasoning and search agents beat single models","HARIS coordinates two agents to outperform in multi-hop claim checks","Two-agent interleaving of reasoning and search lifts claim verification","Interleaved reasoning and search agents outscore single models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training loop depends on the belief that questions sampled from the reasoning agent's own early rollouts, answered by GPT-4o pseudo-ground-truth built from dataset evidence, are good enough to teach the search agent what to retrieve; if the sampled questions are off-target or the pseudo-answers are noisy, both agents learn the wrong collaboration.","fun_headline_variants_meta":{"raw":{"variants":["Coordinated reasoning and search agents beat single models","HARIS coordinates two agents to outperform in multi-hop claim checks","Two-agent interleaving of reasoning and search lifts claim verification","Interleaved reasoning and search agents outscore single models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002314,"raw_usage":{"total_tokens":8906,"prompt_tokens":910,"completion_tokens":7996,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":7928}},"tokens_in":526,"tokens_out":7996,"duration_ms":62746,"temperature":1.0,"reasoning_tokens":7928,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:30:51.295817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take HARIS's trained reasoning agent and swap in a search agent trained on human-annotated question-answer pairs instead of GPT-4o pseudo-answers, keeping everything else fixed; if verification accuracy does not drop, the synthesized training signal is not what makes the method work. Alternatively, run the same coordinated training with the LLM-as-a-Judge reward replaced by exact-match F1 on gold answers: the paper's own comparison predicts a roughly 3-point average loss, so a smaller or absent drop would falsify the claim that the judge reward drives search quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HOVER benchmark covering 2-, 3-, and 4-hop claims."}],"review_version":1}