{"id":"e675c0d6-bea3-43f5-a885-40e2f35d5a83","arxiv_id":"2608.12877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A self-reflective agent pipeline with evidence-drift and reasoning-reflection checks reports new state-of-the-art Macro-F1 on HOVER and EX-FEVER multi-hop fact verification.","lead":"ReflectFact adds two self-check steps to a three-stage agent pipeline for multi-hop fact verification: it compares answers given with and without evidence, and it re-verifies each reasoning step. On the HOVER and EX-FEVER benchmarks it reports state-of-the-art Macro-F1, beating the strongest compared baselines by 3.32 and 2.78 points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central mechanism rests on an unmeasured claim that the same LLM reliably verifies its own intermediate outputs; no verifier-accuracy or correction-success data are reported.","rationale":"The reader's weakest assumption identifies exactly the most load-bearing concern: the method's effectiveness hinges on the LLM's ability to verify its own intermediate reasoning, and the paper provides no direct measurement of that ability. My reading of the full text confirms that the central claim is the SOTA gain from self-reflective verification, and that RRV, not EDV, carries most of the ablated benefit. Since no verifier-accuracy statistics, correction-success rates, or code are provided, the mechanism behind the headline numbers remains empirically unsupported. I do not see an internal inconsistency that would falsify the method, and the ablation and backbone-generalization results are plausibly consistent with the stated mechanism, so the appropriate response is not rejection but a conditional acceptance requiring the verifier-reliability probe above and artifact release. This does not change the reader's conditional verdict, hence UNCHANGED.","tokens_in":12596,"tokens_out":4694,"duration_ms":54094,"concrete_test":"Build a labeled probe set of 200-300 intermediate (x, o) pairs sampled from ReflectFact on HOVER and EX-FEVER, with each o independently annotated correct or incorrect against the provided gold evidence. Run the verification prompt V(x, o) on these pairs and compute verifier precision and recall for detecting incorrect steps. Then take the flagged-inconsistent pairs, regenerate o, and measure the fraction of regenerated outputs that become correct. If verifier recall on incorrect steps is materially below 80% or the correction yield is below 50%, the reported gains cannot be attributed to reliable self-reflection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that Reasoning Reflection Verification (RRV) reliably detects flawed reasoning steps. Equation (1) accepts an output o whenever V(x, o) = 'consistent' and otherwise regenerates it, so any systematic failure of V to flag incorrect steps would let errors propagate and erase the reported gains. The paper's own ablation attributes the largest performance contribution to RRV, yet the only support for the verifier's reliability is the sentence 'LLMs exhibit stronger verification than generation capabilities,' which is stated without citation or measurement. In particular, the paper never reports verifier precision or recall on labeled intermediate outputs, and no experiment measures how often a flagged 'inconsistent' step is actually corrected after one regeneration. Since decoding is greedy with temperature 0, the regeneration mechanism also depends entirely on the added inconsistency signal changing the model's output; if that signal is not interpretable or not sufficiently informative, the step may be regenerated incorrectly or identically. The error analysis compounds the concern: 82.5% of the remaining errors are classified as factual hallucination, suggesting that verification, even when present, misses many errors. If verifier accuracy is near chance, the Table 2 gains could stem from an unrelated effect such as the additional prompt surface or a small number of successful corrections, rather than from the self-verification mechanism that the paper claims is the primary source of robustness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReflectFact, a self-reflective agent framework for multi-hop fact verification. It constructs an explicit reasoning path through implicit entity resolution, semantic decomposition, and integrative logical reasoning, and augments it with two post-verification mechanisms: Evidence-Drift Verification (EDV) and Reasoning Reflection Verification (RRV). EDV flags cases where an evidence-grounded answer matches the model's parametric answer and forces a re-answer with explicit evidence quotation; RRV checks each reasoning step with the same LLM and regenerates the step if it is flagged inconsistent. Experiments on HOVER and EX-FEVER report Macro-F1 gains over ten baselines, with ablations attributing the largest improvement to RRV, cross-backbone generalization tests on GPT-4o-mini and Qwen3-8B, and an interpretability analysis using ROUGE scores. The paper also includes an error analysis categorizing remaining errors.","tokens_in":12801,"tokens_out":5199,"duration_ms":49416,"significance":"If the results are robust, ReflectFact offers a practical, model-agnostic recipe for improving multi-hop fact verification by adding reflective checks on intermediate reasoning. The framework is transparent, the hop-wise reporting is informative, and the cross-backbone experiments support a generality claim. The paper's main contribution is the two verification mechanisms and the demonstration that they improve accuracy. However, the central mechanism (RRV) rests on an unmeasured assumption about LLM self-verification reliability, and the strongest baseline comparison on EX-FEVER is not backbone-controlled. The contribution is promising but not yet fully established at the journal level.","major_comments":[{"comment":"The claim that LLMs can reliably verify their own intermediate outputs is load-bearing but unsupported. The paper states 'LLMs exhibit stronger verification than generation capabilities' without citation or measurement, and no verifier precision, recall, or correction-success statistics are reported. The ablation in Table 2 shows that removing RRV hurts, but this does not distinguish reliable verification from a beneficial extra regeneration pass. Please report an evaluation of V(x,o) on a labeled sample of intermediate steps: the rate at which the verifier flags genuinely incorrect steps, the rate at which it flags correct steps, and the fraction of regenerated outputs that become correct after flagging.","section":"Reasoning Reflection Verification, Eq. (1)"},{"comment":"The claim of outperforming the strongest baseline by 2.78% on EX-FEVER is against ProgramFC (80.98), yet ProgramFC is run with its original backbone (codex and Flan-T5) rather than GPT-4o-mini. The paper's fairness statement only replaces the backbone for agent-based baselines. Since ReflectFact uses GPT-4o-mini, the comparison confounds method with model capacity. Please either re-run ProgramFC with a matched GPT-4o-mini backbone, or report the discrepancy and temper the SOTA claim accordingly.","section":"Table 1, EX-FEVER"},{"comment":"All results are single-run measurements with no variance estimates or significance tests. Given that the claimed gains over the strongest baselines are 3.32 and 2.78 Macro-F1 points, and that LLM-prompting pipelines can be sensitive to prompt ordering and random seeds, the reader cannot assess whether the improvements are robust. Please add multiple runs (at least three) with standard deviations, or bootstrap confidence intervals, and state the number of runs explicitly.","section":"Tables 1-2, Figures 3-4"},{"comment":"The analysis on 40 sampled errors reports that 82.5% of remaining errors are 'factual hallucination'. This appears to undercut the central claim that RRV catches and corrects reasoning flaws before they propagate: if verification is effective, one would expect many hallucinations to be flagged at intermediate steps. Please clarify whether these hallucinations occur in the final verdict, passed through RRV as 'consistent', or occur in steps not covered by verification. This is essential for interpreting the mechanism and for assessing the reliability of the verifier.","section":"Error Type Analysis, Figure 4"},{"comment":"The criterion for flagging evidence drift is a≡a_p (grounded answer equals parametric answer). The paper does not report how often EDV triggers, how often the re-answer changes the output, or the precision of this heuristic. Without these measurements, the contribution of EDV is under-specified. Please add a trigger-rate and change-rate analysis, ideally with examples of the drift cases it actually corrects.","section":"Evidence-Drift Verification"}],"minor_comments":[{"comment":"The evaluation uses the validation set of HOVER (4,000 claims) rather than the test set; this should be stated explicitly and justified, since HOVER provides a test set.","section":"Experiments - Datasets"},{"comment":"Figure 2 shows 'k=1 to K' in the Reasoning Reflection Verification loop, but the paper never defines K or the iteration procedure in the main text. Please specify the budget and the stopping criterion.","section":"Methodology, Figure 2"},{"comment":"The Replace operation mentions 'dynamic sample selection' for demonstrations, but the selection procedure is not described.","section":"Methodology - Implicit Entity Resolution"},{"comment":"The Related Work section contains a duplicated sentence: 'These methods decompose fact verification into subtasks and execute them through agent workflows' appears twice consecutively.","section":"Related Work"},{"comment":"A citation is needed for the claim that LLMs are better verifiers than generators; the current text presents a contested assertion as a fact, and related self-verification work (e.g., SelfCheckGPT, CRITIC, Self-Refine) is not cited.","section":"Reasoning Reflection Verification"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable incremental contribution to agent-based fact verification, but the central mechanism is under-evidenced and the SOTA claim is not fully controlled. The missing verifier evaluation is the most serious issue; without it, the reported gains could plausibly come from the extra regeneration pass rather than from reliable self-verification. Also, the single-run results and the unmatched ProgramFC baseline need to be addressed before the paper can support a journal-level SOTA claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid incremental paper with a genuinely new check (Evidence-Drift Verification) and a sensible three-stage agent pipeline. The reported SOTA margins on HOVER and EX-FEVER are plausible but currently unverifiable: no code, no variance, single runs.\n\nWhat it does well: The two failure modes it targets (objective conflicts, knowledge conflicts) are real and clearly illustrated. EDV, which compares an evidence-grounded answer to a parametric-only answer and forces a quoted-span re-answer, is the freshest idea. RRV is standard self-reflection but applied systematically to intermediate reasoning steps, and the ablation shows it contributes more than EDV. The evaluation is above the norm for this area: hop-wise Macro-F1, two backbones (GPT-4o-mini and Qwen3-8B), a ROUGE interpretability analysis, and a manual error taxonomy. Replacing agent-baseline backbones with GPT-4o-mini is the right call for fairness.\n\nWhere it gets soft: The load-bearing premise of RRV is that the LLM reliably verifies its own intermediate outputs, and the paper supports that only with an unsupported sentence (\"LLMs exhibit stronger verification than generation capabilities\"). No verifier precision/recall, no correction-success rate. The stress-test concern about this is fair; if the consistency checker is near chance, the gains could come from prompt surface rather than genuine verification. That said, the ablation showing a large drop without RRV is real evidence the mechanism is doing work, so it is a gap in support, not a demonstrated failure. The paper should provide verifier diagnostics in revision. Also, ProgramFC, the strongest EX-FEVER baseline, is in the inference-augmented category and not controlled for backbone; the comparison may not be apples-to-apples. Minor issues: single runs without significance tests, no code release, and a small (n=40) error analysis.\n\nBottom line: this deserves a serious referee. The empirical case is strong enough to engage with despite the soft spot on the self-verification mechanism. A good revision would add code, confidence intervals, verifier accuracy, and a controlled ProgramFC comparison. I would bring it to a reading group for the EDV idea.","headline":"Solid incremental agent framework for multi-hop fact verification; the genuinely new EDV check is worth attention, but the RRV mechanism's reliability is asserted rather than measured.","tokens_in":13365,"tokens_out":3395,"would_cite":true,"duration_ms":32124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReflectFact reports state-of-the-art Macro-F1 of 78.51 on HOVER and 83.76 on EX-FEVER by adding self-reflection checks that catch evidence drift and reasoning flaws before they reach the verdict.","keywords":["multi-hop fact verification","self-reflective agents","evidence grounding","reasoning verification","fact-checking","HOVER","EX-FEVER","LLM reasoning"],"falsifier":"Take a set of multi-hop claims from either dataset, corrupt individual reasoning steps with known bias types such as replacing an entity with a co-referring but inconsistent one or shifting a date or location, and run only the verification prompt of Reasoning Reflection Verification on the corrupted steps; if the verifier accepts a large share of corrupted steps as 'consistent,' the reflection mechanism cannot be what drives the reported gains. Alternatively, ablate the regeneration branch by keeping the verification verdict but discarding the regenerated output; if Macro-F1 does not drop, the mechanism's value lies elsewhere.","tokens_in":12386,"feed_emoji":"🔍","tokens_out":5413,"duration_ms":48675,"temperature":0.7,"pith_summary":"The paper argues that two failure modes—agents losing sight of the global verification goal, and agents silently substituting parametric memory for the supplied evidence—can be caught by adding a reflective verification step after each subtask. It proposes ReflectFact, a pipeline that first constructs an explicit reasoning path by resolving implicit entities, decomposing the claim into sub-questions, and composing the answers into a verdict, then runs two checks. Evidence-Drift Verification asks the model to re-answer from a quoted evidence span whenever its grounded answer coincides with its evidence-free answer, and Reasoning Reflection Verification re-examines each reasoning step and regenerates it if it is judged inconsistent with the global fact-checking objective. A sympathetic reader would care because, if this works, a generic LLM can be made substantially more reliable at multi-hop verification without fine-tuning, simply by using the model as a critic of its own intermediate outputs.","feed_headline":"Self-reflection lifts multi-hop fact-checking to new bests","feed_subtitle":"Two checks, one on evidence drift and one on reasoning steps, raise Macro-F1 on HOVER and EX-FEVER.","key_machinery":"The central object is the self-reflective verification loop defined by Eq. (1): for each instruction-driven reasoning step, the pair of input and output is presented to the LLM as an object to be checked under an explicit 'this is part of a fact-checking task' framing; a 'consistent' verdict accepts the output, and an 'inconsistent' verdict triggers regeneration. Evidence-Drift Verification is the complementary loop for evidence-comprehension steps, comparing an evidence-grounded answer with an evidence-free answer and forcing the model to quote a verbatim evidence span when the two coincide. Together they convert the LLM from a generator into a verifier of its own intermediate reasoning.","core_discovery":"On the paper's own terms, the central claim is that post-hoc self-reflection at each intermediate step, rather than better retrieval or bigger models, is what unlocks reliable multi-hop fact verification. ReflectFact reports Macro-F1 of 78.51 on HOVER and 83.76 on EX-FEVER, outperforming the strongest of ten baselines by 3.32 and 2.78 points, and the ablation shows that removing Reasoning Reflection Verification costs more than removing Evidence-Drift Verification, especially on four-hop claims. The two checks are designed to counter specific failure patterns: replacing an implicit entity mention with an entity whose date contradicts the claim's description, and answering from parametric knowledge when the evidence disagrees with it.","pith_inferences":["The paper's own error analysis finds that 82.5 percent of remaining errors are factual hallucinations in Semantic Decomposition; since the reflection checks are limited to consistency with the provided evidence, they cannot correct a model's confident fabrication of a fact that is internally coherent, so a natural extension is to add external verification of generated sub-answers.","The central assumption that LLMs verify better than they generate is testable in isolation: one could measure the consistency check's precision on deliberately corrupted reasoning steps and use that precision to predict when the framework's gain should vanish.","If the equivalence check in Evidence-Drift Verification is the trigger for re-answering, then the framework's benefit depends on the model having a strong parametric prior; models trained to be more faithful to context might see smaller gains."],"forward_implications":["If the reported gains are real, adding the two reflection checks to an existing agent pipeline should improve Macro-F1 without changing the retrieval evidence.","The benefit grows with reasoning depth: the four-hop HOVER gap over the strongest agent baseline is 3.79 points, so reflection matters most when errors have more chances to propagate.","The framework is model-agnostic; applying it to a smaller open-weight backbone should give consistent gains over that backbone's direct prompting, as shown for a proprietary and an open-weight model.","Requiring the model to quote supporting evidence spans when drift is suspected produces explanations that score higher on ROUGE than several dedicated explanation generators, so the check also improves interpretability."],"supporting_citations":[{"why":"Supplies the HOVER dataset and its multi-hop claim set used for evaluation.","marker":"Jiang et al. 2020"},{"why":"Supplies the EX-FEVER dataset, its golden explanations, and the explanation baselines used in the interpretability comparison.","marker":"Ma et al. 2023"},{"why":"Provides the HiSS hierarchical step-by-step prompting baseline that ReflectFact is compared against.","marker":"Zhang and Gao 2023"},{"why":"Provides the ProgramFC program-guided reasoning baseline.","marker":"Pan et al. 2023"},{"why":"Provides the Factcheck-GPT agent baseline and the Factcheck-Bench evaluation context.","marker":"Wang et al. 2024"},{"why":"Provides the BiDeV multi-agent baseline, the strongest agent baseline ReflectFact outperforms.","marker":"Liu et al. 2025"},{"why":"Provides the StepByStepFV agent baseline used in the comparison.","marker":"Vladika et al. 2025"},{"why":"Supplies the GPT-4o-mini backbone used as the engine for ReflectFact and the agent baselines.","marker":"OpenAI et al. 2024"},{"why":"Supplies the Qwen3 backbone used to test the framework's generalizability.","marker":"Yang et al. 2025"},{"why":"Provides the FLAN-T5 vanilla LLM baseline and the T5 module used in the pipeline.","marker":"Chung et al. 2022"}],"fun_headline_variants":["ReflectFact: self-reflection boosts multi-hop fact-checking","Double reflection fixes reasoning flaws in fact verification","Self-correcting agents outdo baselines in multi-hop claims","Two-step reflection yields gains in fact-checking benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM can be trusted to judge whether its own intermediate reasoning step is consistent: any output the verification prompt calls 'consistent' is accepted as-is, and if the model frequently mislabels flawed steps as consistent, the reflection loop would certify errors instead of catching them.","fun_headline_variants_meta":{"raw":{"variants":["ReflectFact: self-reflection boosts multi-hop fact-checking","Double reflection fixes reasoning flaws in fact verification","Self-correcting agents outdo baselines in multi-hop claims","Two-step reflection yields gains in fact-checking benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1973,"prompt_tokens":981,"completion_tokens":992,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":935}},"tokens_in":597,"tokens_out":992,"duration_ms":9488,"temperature":1.0,"reasoning_tokens":935,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:28:16.774250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of multi-hop claims from either dataset, corrupt individual reasoning steps with known bias types such as replacing an entity with a co-referring but inconsistent one or shifting a date or location, and run only the verification prompt of Reasoning Reflection Verification on the corrupted steps; if the verifier accepts a large share of corrupted steps as 'consistent,' the reflection mechanism cannot be what drives the reported gains. Alternatively, ablate the regeneration branch by keeping the verification verdict but discarding the regenerated output; if Macro-F1 does not drop, the mechanism's value lies elsewhere.","supporting_citations":[{"cited_title":"Measuring and Narrowing the Compositionality Gap in Language Models","cited_arxiv_id":null,"evidence_quote":"Provides the HiSS hierarchical step-by-step prompting baseline that ReflectFact is compared against."},{"cited_title":"Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers","cited_arxiv_id":null,"evidence_quote":"Provides the Factcheck-GPT agent baseline and the Factcheck-Bench evaluation context."},{"cited_title":"Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),","cited_arxiv_id":null,"evidence_quote":"Supplies the Qwen3 backbone used to test the framework's generalizability."}],"review_version":1}