{"id":"8fc51388-540e-4029-9de7-7a39051e234d","arxiv_id":"2412.12632","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"External knowledge satisfying intent, evidence nodes, and evidence relations is preferred by LLMs, improving multi-hop QA accuracy and robustness, and can be used to enhance RAG, poisoning, and defense systems.","lead":"Researchers propose that language models prefer external knowledge that forms a 'chain of evidence': information relevant to the question whose pieces mutually support each other. They show that selecting such evidence improves retrieval-augmented question answering across five LLMs, while also making models more easily misled when the chain contains false facts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Non-CoE controls are constructed by removing or abstracting the exact evidence chain, so the reported ACC/ASR gaps may reflect answer presence rather than a preference for interconnectivity.","rationale":"The central claim of the paper is not just that a proposed feature set correlates with performance, but that LLMs prefer external knowledge exhibiting CoE (relevance plus interconnectivity), and that this preference explains robustness and deception. That claim is established primarily by the CoE-vs-Non-CoE comparisons in Tables 1-3. If the Non-CoE condition is constructed by removing the very pieces of evidence required to answer, then the comparison is close to definitional: the LLM does better when the answer-bearing chain is present. The paper's perturbation loop (Appendix E) explicitly removes sentences containing evidence nodes or replaces nodes with hypernyms until GPT-4o says the chain is broken, so Non-CoE contexts are not merely 'relevant but lacking interconnectivity.' They often lack the answer or the link needed to infer it. Appendix D addresses a different issue (extraction errors) and actually supports the view that label noise is not the central weakness: simulated extraction errors change accuracy by only about one point. Thus even perfect GPT-4o labels would not fix the control. At the same time, I would not reject the paper: the RAG+CoE comparison against a standard reranker (Table 4) is a real, less confounded application, and the poisoning and defense results, while also affected by design choices, suggest the CoE features have practical leverage. The right next step is an answerability-matched control or a relation-only perturbation that preserves all evidence nodes and the answer while removing only the relational links. My verdict therefore remains conditional on that control.","tokens_in":18062,"tokens_out":10757,"duration_ms":99868,"concrete_test":"On a random sample of 100 CoE and 100 paired Non-CoE (SenP and WordP) contexts per dataset, have three independent annotators attempt to answer the multi-hop question using only the provided context, with no other knowledge. If a substantial fraction (e.g., >30%) of Non-CoE contexts are unanswerable while the paired CoE contexts are answerable, then the Table 1 ACC gap is explained by answer derivability rather than CoE preference. Report per-condition answerability rates and inter-annotator agreement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The comparative evidence for the headline claim is confounded in the construction of the control condition. CoE knowledge is the gold supporting-fact chain from HotpotQA/2WikiMultihopQA (Section 4.1); Non-CoE is produced by removing the sentences that contain the question's evidence nodes (SenP) or by replacing named evidence nodes with hypernyms (WordP), iterating until the GPT-4o discriminator says the CoE is broken (Section 4.2, Appendix E). This means Non-CoE contexts differ from CoE contexts not only in interconnectivity but also in whether the correct answer can be derived from the context at all. The 16-22 point ACC/ASR gaps in Tables 1-2 may therefore reflect the presence versus absence of answer-bearing evidence, not a preference for coherent chains. Human-validated GPT-4o labels would not resolve this, because the control is degraded by design. The RAG+CoE reranking result (Table 4) is a less confounded demonstration, but the 'significance, deceptiveness, robustness' analyses that constitute the paper's central characterization do not isolate interconnectivity from information content.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Chain of Evidence (CoE) as a characterization of external knowledge that LLMs prefer in multi-hop QA under imperfect contexts. CoE is operationalized through three features extracted from the question: intent, evidence nodes, and evidence relations. The authors use GPT-4o to extract these features and to classify whether given external knowledge exhibits them, then construct a dataset from HotpotQA and 2WikiMultihopQA by keeping only CoE-compliant supporting facts. Non-CoE controls are created by sentence-level removal of evidence-bearing sentences (SenP) or word-level replacement of evidence nodes with hypernyms (WordP), iterating until the GPT-4o discriminator says CoE is absent. The paper reports comparative experiments across significance (accuracy under extraneous information), deceptiveness (attack success rate for incorrect CoE), and robustness (accuracy under conflicting information), as well as three usability tasks: RAG-based multi-hop QA, knowledge poisoning attacks, and poisoning defense. In all settings, CoE-aligned knowledge outperforms or more strongly affects LLMs than Non-CoE knowledge, and CoE-enhanced variants improve over the original baselines.","tokens_in":18311,"tokens_out":3681,"duration_ms":33788,"significance":"If the central claim holds, the paper offers a concrete, operationalizable account of what makes external knowledge effective for multi-hop QA, going beyond single-feature relevance or coherence biases studied earlier. The empirical scope is substantial: five LLMs spanning closed and open families, two multi-hop datasets, and three downstream tasks, with consistently large effect sizes (10--20+ percentage point gaps in Tables 1--4). The authors also provide a reproduction package and openly discuss limitations, including extraction-error sensitivity and the textual-level constraint of their reranking approach. The main risk is that the evidence for the characterization is entangled with how the Non-CoE controls and the IR+CoE defense are constructed, both of which may introduce answer-presence or extra-knowledge confounds that are not intrinsic to the CoE notion.","major_comments":[{"comment":"The Non-CoE controls are constructed by removing sentences containing question evidence nodes (SenP) or replacing evidence nodes with hypernyms (WordP), iterating until the GPT-4o discriminator says the CoE is broken. This means Non-CoE contexts differ from CoE contexts not only in interconnectivity but also in whether the correct answer can be derived from the context at all. The 16--22 point ACC/ASR gaps in Tables 1--3 may therefore reflect presence versus absence of answer-bearing evidence rather than a preference for coherent evidence chains. A control that preserves all answer facts while breaking the chain (e.g., reordering sentences, substituting relations with relationally false but still answer-bearing statements, or adding unrelated but answer-independent text) is needed to isolate interconnectivity.","section":"Section 4.2 and Appendix E"},{"comment":"All CoE labels--feature extraction, feature discrimination, and the selection of which supporting facts count as CoE--are produced by GPT-4o, with no human annotation or independent model validation reported. Since the same model family (GPT-4/GPT-4o) is also used to judge ACC and ASR in Sections 5.1 and 6.1, the measured 'preference' for CoE could partly reflect the labeler's own text preferences. The open-source models in the evaluation mitigate this concern somewhat, but the paper should include a human agreement study on a sample of the CoE/Non-CoE labels, or reproduce the discrimination with a different model family, to show the effect is not an artifact of the labeler.","section":"Section 3.2 and Section 4.1"},{"comment":"The IR+CoE defense is confounded: the variant injects CoE-structured knowledge into the knowledge base, so IR+CoE has access to additional correct documents that the IR baseline does not. The reported 27.2% ACC improvement and 35.8% ASR reduction may therefore come from the extra injected knowledge rather than from the CoE-based validation mechanism. An ablation where IR receives the same number and type of extra documents without CoE structuring, or where the CoE validation step is applied to the original retrieved set, is necessary to support the defense claim.","section":"Section 8.3 and Table 4"},{"comment":"The paper states in Section 5.1 that GPT-4 is used as the judge for accuracy, but Section 6.1 and Section 7.1 state GPT-4o. Aside from the inconsistency, both are from the same model family used to define and label CoE. Reporting agreement with human evaluation or an independent judge on a random sample would strengthen the validity of all ACC/ASR comparisons.","section":"Section 5.1 and Section 6.1"}],"minor_comments":[{"comment":"The typo 'chodse' in the opening sentence of Section 8 should be corrected to 'chose'.","section":"Section 8"},{"comment":"The citation 'Chen et al.;' appears incomplete; the reference is not resolved in the bibliography.","section":"Section 7.1"},{"comment":"The text says accuracy drops from 90.2% to 89.3%/89.4%, but Table 9 shows 91.9% at proportion 0 for the 'Our' condition. Please make the values consistent.","section":"Appendix D"},{"comment":"The abbreviation 'COE' is used in several places (e.g., 'the COE group' and 'COE completeness') instead of the established 'CoE'. Please unify the capitalization.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the empirical scale is appropriate for cs.CL. The major revision should focus on de-confounding the Non-CoE controls and the IR+CoE defense, and on adding human or independent-model validation of the CoE labels. If the confounds are resolved, the paper could be a solid contribution; as it stands, the headline characterization is not yet isolated from simpler answer-presence and extra-knowledge explanations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper has a genuinely useful practical trick—rerank retrieved snippets by whether they cover the question's intent, evidence nodes, and relations—and it moves the needle across five LLMs. But the paper's headline claim, that LLMs 'prefer' chain-of-evidence knowledge, is not actually supported by the experiments, because the Non-CoE control is constructed by removing or abstracting the exact sentences that contain the answer. The 16–22 point gaps in Tables 1–3 are what you'd expect when you compare contexts that contain the answer to contexts that don't.\n\nWhat's new and good: the three-feature operationalization (intent, evidence nodes, relations) is concrete and easy to apply. Table 4 shows RAG+CoE improves multi-hop QA accuracy by about 10 points on average, and the poisoning and defense variants also improve consistently. Those are practical, reproducible gains, and the paper includes its reproduction package. The writing is clear and the limitations section is honest about not verifying the correctness of CoE and about the text-level limitation.\n\nWhere it goes soft: first, the control. SenP removes sentences containing evidence nodes until the GPT-4o discriminator says CoE is broken. WordP replaces evidence nodes with hypernyms. Both remove the ability to derive the answer from the context. So the comparative analyses in Sections 5–7 confound interconnectivity with answer presence. The stress-test note is correct; human-validated labels wouldn't fix that, because the control is degraded by design. Second, the labeling and judging pipeline is GPT-4o end-to-end: it extracts features, judges CoE status, selects snippets, and evaluates answers. There's no human annotation study, and Appendix D only analyzes node-extraction errors, not the accuracy of the CoE judgments. Third, the IR+CoE defense experiment injects extra CoE-structured knowledge into the knowledge base, so it's not a controlled comparison with IR. Fourth, no error bars or multiple runs; only single numbers with p-values.\n\nWho's this for? People building RAG systems or studying knowledge poisoning will find the reranking heuristic useful. The characterization claim, as stated, needs to be reworked. I'd send this to a serious referee, but expect major revision: fix the control or reframe the claim, add human validation of the discrimination, and report variance. If the authors can show that CoE features help even when the answer-bearing content is held constant, then the preference claim would have legs.","headline":"A useful practical reranking heuristic for RAG, but the central 'CoE preference' claim is undercut by a confounded control that removes the answer-bearing evidence.","tokens_in":18852,"tokens_out":3260,"would_cite":false,"duration_ms":29063,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLMs prefer external knowledge that forms a chain of evidence — relevant to the question and internally interconnected — and that this preference improves multi-hop QA, poisoning, and defense pipelines.","keywords":["chain of evidence","multi-hop question answering","retrieval-augmented generation","knowledge poisoning","LLM knowledge preference","context robustness"],"falsifier":"Re-label the 1,336 samples' CoE status with human annotators following the paper's own definitions, then rerun the significance, deceptiveness, and robustness comparisons; if the CoE advantage (for instance, the 22.5% accuracy gap) disappears or reverses under human labels, the effect is an artifact of GPT-4o's self-consistent labeling rather than a preference LLMs generally hold.","tokens_in":17856,"feed_emoji":"🔗","tokens_out":4043,"duration_ms":34595,"temperature":0.7,"pith_summary":"The paper asks what makes external knowledge usable by LLMs in multi-hop QA when the context is noisy or misleading. It characterizes the preferred knowledge as forming a Chain of Evidence (CoE): relevant to the question and internally interconnected through three textual features — intent, evidence nodes, and evidence relations. Using an automated discriminator built on GPT-4o, the authors label knowledge as CoE or Non-CoE and show that CoE-aligned knowledge substantially outperforms perturbed alternatives across five LLMs and two multi-hop datasets. They then integrate CoE features into RAG, knowledge-poisoning, and poisoning-defense pipelines and report consistent improvements over strong baselines.","feed_headline":"Chain-of-evidence knowledge beats noisy context by 22.5%","feed_subtitle":"Aligned snippets resist extraneous and conflicting info, boosting multi-hop QA, poisoning, and defense results.","key_machinery":"The carrying mechanism is the CoE discrimination pipeline: for each question, GPT-4o extracts an intent (the noun phrase describing the desired answer type), evidence nodes (key entities for reasoning), and evidence relations (logical predicates linking pairs of nodes). A second GPT-4o stage checks whether a given piece of external knowledge satisfies all three features — intent via textual entailment, nodes and relations via containment classification — and only then is the knowledge labeled CoE. This operationalization of relevance plus interconnectivity is what generates the CoE versus Non-CoE comparison and the downstream task variants.","core_discovery":"The paper's central claim is that external knowledge exhibiting CoE features is preferred by LLMs: it resists extraneous and conflicting information in multi-hop QA better than knowledge that is merely relevant but lacks interconnectivity. Empirically, CoE knowledge achieves an average accuracy of 92.0% across five LLMs and two datasets, outperforming the Non-CoE sentence-perturbed and word-perturbed variants by 22.5% and 16.3% respectively, with the gap widening as extraneous information increases. When the CoE contains factual errors, it is also more deceptive, reaching an average attack success rate of 85.4%. Structuring knowledge as CoE improves a standard RAG pipeline by 10.4% accuracy, a poisoning attack by 11.0% attack success rate, and a defense framework by 27.2% accuracy while lowering attack success by 35.8%.","pith_inferences":["The paper leaves implicit that CoE preference may unify previously reported single-hop biases — coherence bias, completeness bias, and confirmation bias — into a single structural factor: relevance plus mutual support among evidence pieces.","Because the same model family (GPT-4o) both extracts the CoE features and evaluates the downstream answers, part of the measured effect could reflect GPT-4o's self-consistent labeling style; a human-annotated CoE label set would test this directly.","Since CoE-structured false knowledge is highly deceptive, adding CoE-aware reranking to a RAG system without verifying answer correctness could increase vulnerability to poisoning — a risk the paper acknowledges but does not quantify beyond the deceptiveness results.","The CoE features could plausibly be used as a reward signal to train LLMs to ignore irrelevant context, but the paper does not test this training-based extension."],"forward_implications":["Retrievers and rerankers that prioritize CoE features should improve multi-hop QA accuracy in noisy retrieval settings, since CoE-aligned context degrades far less as extraneous information is added.","Knowledge poisoning attacks become more potent when false documents are structured as CoE, so RAG defenses must check factual correctness of evidence chains, not just their structural fit.","Defense frameworks can be strengthened by generating or selecting CoE-structured supporting knowledge: the paper's InstructRAG variant improves accuracy and sharply reduces attack success rate.","The CoE preference appears consistent across closed- and open-source LLMs and across both HotpotQA and 2WikiMultihopQA, suggesting a general behavioral tendency rather than a model-specific artifact."],"supporting_citations":[{"why":"Supplies the legal Chain of Evidence concept (relevance plus interconnectivity) that the paper adapts to multi-hop QA.","marker":"Murphy, 2013"},{"why":"Documents LLMs' coherent-knowledge bias and provides the perturbation strategies used to construct Non-CoE samples.","marker":"Xie et al., 2023"},{"why":"Shows LLMs prioritize highly relevant knowledge; the paper extends this from single-hop to multi-hop by adding interconnectivity.","marker":"Chen et al., 2022"},{"why":"Establishes completeness bias, which CoE formalizes as interconnectivity, and serves as a key baseline for knowledge-preference findings.","marker":"Zhang et al., 2024"},{"why":"Provides PoisonedRAG, the SOTA knowledge-poisoning attack baseline that the PR+CoE variant improves.","marker":"Zou et al., 2024"},{"why":"Provides InstructRAG, the SOTA defense baseline that the IR+CoE variant improves.","marker":"Wei et al., 2024"},{"why":"Supplies the HotpotQA dataset used for construction of CoE and Non-CoE samples and for downstream evaluations.","marker":"Yang et al., 2018"},{"why":"Supplies the 2WikiMultihopQA dataset used to construct CoE and Non-CoE samples.","marker":"Ho et al., 2020"}],"fun_headline_variants":["LLMs prefer interconnected evidence: 22.5% QA gain over noisy context","Chain-of-evidence knowledge lifts RAG by 10.4%, defense by 27.2%","CoE knowledge resists extraneous info: 22.5% better multi-hop QA","Interconnected evidence outperforms isolated facts in LLM QA","LLMs choose chain-of-evidence knowledge: 22.5% accuracy boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole study assumes that GPT-4o, given hand-crafted prompts, reliably extracts evidence nodes and relations from questions and correctly judges whether external knowledge exhibits them, with no human annotation study to validate these labels.","fun_headline_variants_meta":{"raw":{"variants":["LLMs prefer interconnected evidence: 22.5% QA gain over noisy context","Chain-of-evidence knowledge lifts RAG by 10.4%, defense by 27.2%","CoE knowledge resists extraneous info: 22.5% better multi-hop QA","Interconnected evidence outperforms isolated facts in LLM QA","LLMs choose chain-of-evidence knowledge: 22.5% accuracy boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000431,"raw_usage":{"total_tokens":2187,"prompt_tokens":917,"completion_tokens":1270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1161}},"tokens_in":533,"tokens_out":1270,"duration_ms":8580,"temperature":1.0,"reasoning_tokens":1161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:52:35.614419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the 1,336 samples' CoE status with human annotators following the paper's own definitions, then rerun the significance, deceptiveness, and robustness comparisons; if the CoE advantage (for instance, the 22.5% accuracy gap) disappears or reverses under human labels, the effect is an artifact of GPT-4o's self-consistent labeling rather than a preference LLMs generally hold.","supporting_citations":[],"review_version":1}