{"id":"75cae725-6407-412c-955c-2b6621510ace","arxiv_id":"2506.07075","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A graph-based framework that models the step-by-step evolution of reasoning paths improves multi-hop fact verification accuracy on FEVER and HoVer.","lead":"This paper presents a multi-hop fact-checking model that builds reasoning graphs during evidence retrieval and verification. The authors report improved accuracy on two standard benchmarks, but the gains are modest and the comparisons omit several recent methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'best overall performance' claim is not tested against the authors' own MRR-FV system (ref. [35]), so it is established only for a selected baseline set, not for the field.","rationale":"The central claim is empirical, so the most load-bearing condition is that the comparison set includes the relevant state of the art. The paper cites MRR-FV as the closest multi-hop framework but does not benchmark against it, making the headline untestable against its own prior result. The reader's conditional verdict already flags baseline selection; this concern reinforces that condition rather than changing the verdict. I am not arguing the method is invalid: the ablations are internally consistent and the gains over the listed baselines are plausible. I just cannot accept 'best overall performance' as a general statement without the MRR-FV comparison. The concrete test is cheap and decisive because it uses published numbers and the same standard HoVer/FEVER protocols.","tokens_in":11382,"tokens_out":5161,"duration_ms":55243,"concrete_test":"Add MRR-FV's published HoVer label accuracies (from AAAI 2025) to Table II and its FEVER results to Table I, using the same evaluation protocols; then recompute the hop-wise gains claimed in Section V-B. If MRR-FV outperforms SR-MFV on any subset or overall, the 'best overall performance' claim must be withdrawn or qualified; if SR-MFV still leads, the omission is a presentation flaw rather than a correctness failure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-B asserts that SR-MFV 'achieves the best overall performance in claim verification across both datasets, outperforming all compared baselines.' For HoVer, the strongest relevant published system is MRR-FV, the authors' own AAAI 2025 multi-hop retrieval-and-reasoning model, cited as [35] but absent from Tables I and II and from the baseline list in Section V-A. The claim is therefore only shown for the chosen set, not for the field. If MRR-FV's published label accuracy exceeds SR-MFV on any HoVer hop subset, the headline claim is false as stated. This omission is load-bearing because the paper itself cites [35] as a multi-hop method, and MRR-FV is precisely the comparison that would delimit the novelty and advantage of the proposed progressive reasoning structure. The absence of error bars and the mismatch between the text's claimed gains (1.48%, 1.51%, 3.09% on 2/3/4-hop HoVer) and the table-derived gains (1.15%, 1.15%, 2.35%) further reduce confidence, but the omitted baseline is the central issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SR-MFV, a multi-hop fact verification framework with two modules: a structure-enhanced retrieval module that constructs a reasoning graph from the claim and previously retrieved evidence to guide the next-hop query, and a reasoning-path-guided verification module that incrementally builds subgraphs in retrieval order and processes them with GraphFormers to capture evolving inference trajectories. Experiments on FEVER and HoVer report improvements in label accuracy and retrieval F1 over the selected baselines, with ablations supporting the contribution of each component. The main claims are that explicitly modeling reasoning paths as graphs improves both retrieval precision and verification accuracy, particularly for 4-hop claims on HoVer.","tokens_in":1672,"tokens_out":1847,"duration_ms":45222,"significance":"If the reported results hold, the idea of representing retrieval order as a structural reasoning progression and feeding it into a graph-Transformer verification module is a plausible and potentially useful contribution to multi-hop fact verification. The paper is clearly written, the method is well specified, and the ablation study targets the key design choices. However, the significance is currently limited by the omission of the authors' own strong multi-hop system MRR-FV (ref. [35]) from the baseline comparison, by apparent inconsistencies between claimed gains and the numbers in Tables II and IV, and by the unvalidated assumption that retrieval order approximates logical inference order. These issues need to be addressed before the central claim of consistent superiority can be accepted.","major_comments":[{"comment":"The paper's headline claim that SR-MFV achieves the best overall performance and outperforms all compared baselines is not tested against the authors' own MRR-FV system (ref. [35]), which is a multi-hop retrieval-and-reasoning model specifically designed for complex fact verification and is cited in the Related Work. Since MRR-FV is arguably the most relevant recent baseline for HoVer, omitting it from Tables I and II and from the baseline list in Section V-A makes the claim 'outperforming all compared baselines' true only for a selected subset. Please add MRR-FV to the comparison, report per-hop HoVer results, and qualify the claim accordingly if MRR-FV's published scores are competitive or higher.","section":"Section V-A/B, Tables I and II"},{"comment":"The numerical gains stated in the text do not match the corresponding tables. Section V-B reports consistent gains of 1.48%, 1.51%, and 3.09% on HoVer 2/3/4-hop label accuracy, but Table II gives differences between SR-MFV and the strongest listed baseline (SAGP) of 1.15, 1.15, and 2.35 percentage points. Similarly, Section V-C reports retrieval F1 gains of 0.49%, 0.73%, and 1.12% on 2/3/4-hop, whereas Table IV shows differences versus GMR of 0.4, 0.9, and 0.9 points and versus Baleen of 1.1, 0.6, and 1.1 points. Please correct the text or explicitly state which baseline and table these percentages refer to; as written, the claimed improvements are not reproducible from the reported results.","section":"Section V-B and Section V-C"},{"comment":"The progressive subgraph construction is the core novelty of the verification module, but it rests on the stated assumption that the retrieval order is a proxy for the logical inference process. The paper provides no evidence for this assumption. If retrieval order is noisy, the sequence of subgraphs may not reflect a genuine reasoning trajectory, and any gains could come from the GraphFormer encoder rather than from the progressive structure. Please add a direct test: compare verification accuracy when evidence is fed in retrieval order, in gold-annotated order, and in a random order, and report whether the proposed progressive construction still helps. This would make the central mechanism falsifiable.","section":"Section IV-B1, 'Subgraphs Construction'"},{"comment":"No uncertainty estimates or significance tests are reported. In Table V the ablation gaps are small (for example, 0.42 and 0.28 percentage points on the 2-hop subset for 'w/o Subgraphs Construction' and 'w/o GraphFormers'), and in Table II the claimed improvements over SAGP are 1.15 points on 2-hop and 3-hop. Without multiple runs, confidence intervals, or significance tests, it is unclear whether these differences are meaningful. Please report standard deviations across at least three seeds or provide a significance test for the main comparisons and ablations.","section":"Section V-B and Section V-D"}],"minor_comments":[{"comment":"References [19] and [20] are identical (both are the Baleen paper), and references [26] and [27] duplicate reference [10] (SAGP). Please replace the duplicates with the intended distinct works or remove them.","section":"References"},{"comment":"The table title uses 'HOVER' while the rest of the paper uses 'HoVer'; please make the notation consistent.","section":"Table IV"},{"comment":"The text alternately uses 'SR-MFV' and 'MFV' (e.g., 'these results highlight that MFV's ability'); please use the full model name consistently.","section":"Section V-B"},{"comment":"The threshold tau is described as 'tunable', but the paper does not report its value or sensitivity. Please state the value used and, ideally, add a sensitivity analysis for this hyperparameter.","section":"Equation (7)"},{"comment":"The y-axis is labeled 'Accuracy' but the paper does not say whether the plotted values are dev or test accuracy; please specify the split in the caption or axis label.","section":"Figure 3"},{"comment":"In Section IV-B2, the notation for the node representation h_g^(l) and its augmented version is not fully defined; in particular, Eq. (12) concatenates a vector with h_g^(l), but Eq. (9) defines z_g^(l) as a scalar [CLS] embedding. Please clarify the dimensional conventions.","section":"Appendix/Notation"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the omission of MRR-FV, the authors' own AAAI 2025 system, from the baseline comparison. This is not a circularity problem in the derivation, but it substantially weakens the empirical claim and should be fixed before the paper is considered further. The numerical inconsistencies between the text and Tables II and IV also need correction. The paper's central idea is defensible and the framing is clear, so I believe these issues are addressable within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2506.07075. First, the progressive subgraph verification idea is genuinely new and the ablation study suggests it works. Second, the paper's central claim—'best overall performance'—is only established for a selected baseline list that omits the authors' own MRR-FV system (ref. [35]), and Section V-B's stated gains don't match Table II.\n\nThe novelty here is real: instead of building one static evidence graph, SR-MFV constructs a sequence of subgraphs in retrieval order and fuses them with attention. That is a distinct departure from CO-GAT and SAGP, and the ablations in Table V show each component (subgraph construction, GraphFormers) contributes. The retrieval-side reasoning graph is more standard, but the combination is clean and the write-up is concrete. I believe the method works as described for the baselines tested.\n\nThe soft spots are not fatal individually, but they add up. The MRR-FV omission is the biggest. The authors cite this AAAI 2025 system as a multi-hop retrieval-and-reasoning method, yet it appears nowhere in Tables I or II. That means the strongest comparison is missing, and the 'best overall performance' sentence is true only for the chosen set. A referee should ask for that run.\n\nThere's also a numerical mismatch: Section V-B claims gains of 1.48%, 1.51%, and 3.09% on HoVer 2/3/4-hop, but Table II differences are 1.15, 1.15, and 2.35 points. One of them is wrong, and the paper doesn't acknowledge it. Minor but embarrassing.\n\nNo error bars, no significance tests, no code release. The core assumption that retrieval order approximates logical inference order is stated honestly as a proxy, but never stress-tested. The hyperparameter analysis is nice, but it doesn't speak to that assumption.\n\nWho is this for? Fact-checking researchers, especially those working on graph-based verification. It's an incremental but useful contribution. With MRR-FV added and the arithmetic fixed, it would be a solid conference paper; without them, the central claim is overreached.\n\nI'd send it to a serious referee with instructions to check the baseline comparison and the numbers. It deserves peer review, not a desk reject.","headline":"A solid incremental fact-verification paper whose progressive-subgraph idea is worth a look, but whose headline claim overreaches because it never compares against the authors' own MRR-FV baseline.","tokens_in":12128,"tokens_out":2853,"would_cite":true,"duration_ms":28269,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that explicitly modeling the evolving reasoning path as a sequence of evidence subgraphs improves multi-hop fact verification on FEVER and HoVer, with the largest gains on 4-hop claims.","keywords":["multi-hop fact verification","evidence retrieval","reasoning path","structure-enhanced query","GraphFormers","FEVER","HoVer","progressive subgraphs"],"falsifier":"Take a fixed set of retrieved evidence for a HoVer claim and reverse the order in which the verification module receives it. If SR-MFV's prediction and confidence stay essentially unchanged under reversal, then the progressive subgraph sequence is not carrying the claimed reasoning signal; if accuracy drops sharply, the order is load-bearing.","tokens_in":1660,"feed_emoji":"🔍","tokens_out":1963,"duration_ms":77044,"temperature":0.7,"pith_summary":"The paper tries to establish that multi-hop fact verification improves when the reasoning path itself, not just the retrieved evidence, is treated as an explicit modeling object. Its framework, SR-MFV, builds a reasoning graph at each retrieval step to shape the next query, then turns the ordered evidence into a sequence of growing subgraphs for verification. A sympathetic reader should care because complex claims require evidence to be combined in a specific order and structure, and static or concatenation-based models miss that. The reported results on FEVER and HoVer support the claim: label accuracy and retrieval F1 are consistently higher than strong baselines, with the largest gap on 4-hop HoVer claims.","feed_headline":"Reasoning-path graphs lift multi-hop fact-checking accuracy","feed_subtitle":"Modeling evidence as progressively built subgraphs improves label accuracy and retrieval on FEVER and HoVer.","key_machinery":"The load-bearing object is the progressive subgraph sequence $\\{G_1, G_2, \\dots, G_n\\}$, where each $G_k$ contains the claim and the first $k$ retrieved evidence sentences, preserving retrieval order as a proxy for logical inference order. Each subgraph's edges combine intra-sentence adjacency, inter-sentence coreference, and learned latent edges selected by thresholding pairwise similarities. The verification module reasons over each subgraph with GraphFormers, a Transformer variant that interleaves token-level self-attention with graph-structured aggregation, then fuses the per-subgraph representations with learned attention weights. On the retrieval side, an analogous reasoning graph over the claim and previously retrieved evidence undergoes graph attention message passing to produce the next-hop query, so that each retrieval step is grounded in the evolving evidence structure rather than in plain text concatenation.","core_discovery":"On its own terms, the paper's central discovery is that the reasoning trajectory of a multi-hop verification system can be used as a modeling signal instead of being discarded after retrieval. The proposed SR-MFV builds a reasoning graph during retrieval to generate each next-hop query, then turns the ordered evidence list into a sequence of nested subgraphs, each containing the claim and the first $k$ evidence sentences. A GraphFormers encoder reasons over each subgraph, and an attention-weighted fusion of the subgraph states produces the final SUPPORTED, REFUTED, or NOT ENOUGH INFO prediction. On FEVER, this reaches 80.17 label accuracy and 77.62 FEVER score on the test set; on HoVer it reaches 79.05, 77.93, and 78.36 label accuracy on 2-, 3-, and 4-hop claims, the best in each column, with the largest advantage on 4-hop claims. The paper also reports retrieval F1 gains on both datasets, at the cost of slightly lower recall on FEVER due to a more conservative retrieval strategy.","pith_inferences":["A natural extension outside the paper is to randomize or reverse retrieval order before verification; if the attention-weighted subgraph fusion still performs as well, the benefit comes from the graph encoder rather than from reasoning progression.","The subgraph attention weights from Equation (15) could be exposed as a per-hop explanation of which evidence moved the prediction, turning the verification module into an interpretability tool the paper does not build.","Because the retrieval graph and verification subgraphs share the same edge vocabulary, the approach might transfer to open-domain multi-hop question answering, where query decomposition and evidence ordering are similarly load-bearing.","A stress test of the latent edge threshold $\\tau$ in Equation (7) would show whether performance is sensitive to the choice; if a broad range of $\\tau$ works, the method is more likely to generalize."],"forward_implications":["If correct, preserving the original retrieval order through the verification stage is a useful inductive bias, so systems should stop destroying evidence order when building verification graphs.","Structure-enhanced query generation can make retrieval more precise: on FEVER the method raises Precision@5 and F1@5 over CO-GAT while accepting lower recall.","The optimal number of hops is dataset-dependent, roughly matching the annotated claim complexity on HoVer and peaking at two hops on FEVER, so deployed systems should tune retrieval depth rather than fix it.","GraphFormers-style interleaving of token attention and graph aggregation is a viable backbone for evidence-graph reasoning, not just for the original textual-graph task."],"supporting_citations":[{"why":"Supplies the FEVER dataset, its three-label scheme, and the official FEVER score metric used in evaluation.","marker":"[1]"},{"why":"Supplies the HoVer dataset with its 2-, 3-, and 4-hop evidence chain annotations that define the complexity breakdown.","marker":"[2]"},{"why":"GEAR is a graph-based fact verification baseline that SR-MFV must beat on both datasets.","marker":"[3]"},{"why":"KGAT provides the BERT Concat and GAT baselines and the fine-grained verification approach that SR-MFV extends.","marker":"[4]"},{"why":"EvidenceNet is a one-hop evidence fusion baseline compared on both FEVER and HoVer.","marker":"[6]"},{"why":"CO-GAT supplies the unified-graph reasoning baseline and the FEVER retrieval evaluation setup that SR-MFV is compared against.","marker":"[8]"},{"why":"SAGP is the strongest multi-hop baseline on HoVer, providing the per-hop numbers that SR-MFV claims to exceed.","marker":"[10]"},{"why":"GraphFormers is the Transformer-with-graph-aggregation architecture used as the reasoning engine over each evidence subgraph.","marker":"[34]"}],"fun_headline_variants":["Reasoning-path graphs sharpen multi-hop fact-checking","Graph-structured reasoning paths improve fact verification","Reasoning-path signals boost multi-hop fact-checking accuracy","Graphs from reasoning paths improve veracity prediction"],"cache_read_input_tokens":14336,"weakest_assumption_plain":"The load-bearing premise is that the order in which evidence sentences are retrieved is a faithful proxy for the order of logical inference, so that subgraphs built in retrieval order capture the true reasoning trajectory; if retrieval order is noisy, the progressive construction may not reflect genuine reasoning dynamics.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning-path graphs sharpen multi-hop fact-checking","Graph-structured reasoning paths improve fact verification","Reasoning-path signals boost multi-hop fact-checking accuracy","Graphs from reasoning paths improve veracity prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3013,"prompt_tokens":933,"completion_tokens":2080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2020}},"tokens_in":549,"tokens_out":2080,"duration_ms":16503,"temperature":1.0,"reasoning_tokens":2020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:42:17.358196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of retrieved evidence for a HoVer claim and reverse the order in which the verification module receives it. If SR-MFV's prediction and confidence stay essentially unchanged under reversal, then the progressive subgraph sequence is not carrying the claimed reasoning signal; if accuracy drops sharply, the order is load-bearing.","supporting_citations":[{"cited_title":"FEVER: a large-scale dataset for fact extraction and verification,","cited_arxiv_id":null,"evidence_quote":"Supplies the FEVER dataset, its three-label scheme, and the official FEVER score metric used in evaluation."},{"cited_title":"HoVer: a dataset for many-hop fact extraction and claim verification,","cited_arxiv_id":null,"evidence_quote":"Supplies the HoVer dataset with its 2-, 3-, and 4-hop evidence chain annotations that define the complexity breakdown."},{"cited_title":"GEAR: graph-based evidence aggregating and reasoning for fact verification,","cited_arxiv_id":null,"evidence_quote":"GEAR is a graph-based fact verification baseline that SR-MFV must beat on both datasets."},{"cited_title":"Fine-grained fact verification with kernel graph attention network,","cited_arxiv_id":null,"evidence_quote":"KGAT provides the BERT Concat and GAT baselines and the fine-grained verification approach that SR-MFV extends."},{"cited_title":"EvidenceNet: evidence fusion network for fact verification,","cited_arxiv_id":null,"evidence_quote":"EvidenceNet is a one-hop evidence fusion baseline compared on both FEVER and HoVer."},{"cited_title":"Multi-evidence based fact verification via a confidential graph neural network,","cited_arxiv_id":null,"evidence_quote":"CO-GAT supplies the unified-graph reasoning baseline and the FEVER retrieval evaluation setup that SR-MFV is compared against."},{"cited_title":"GraphFormers: GNN-nested transformers for representation learning on textual graph,","cited_arxiv_id":null,"evidence_quote":"GraphFormers is the Transformer-with-graph-aggregation architecture used as the reasoning engine over each evidence subgraph."}],"review_version":1}