{"id":"6f86723d-605b-4a50-a74d-78d37dfb3fa1","arxiv_id":"2505.00023","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CORG, a graph-based context grouping framework, improves disambiguated answer recall on QA with distracting, ambiguous, counterfactual, and duplicated contexts, and reaches performance comparable to per-document processing at lower cost.","lead":"This paper introduces CORG, a three-stage pipeline that groups related documents before asking a language model a question, aimed at cases where documents contradict, duplicate, or ambiguously refer to the same entity. The method improves disambiguated answer recall on synthetic multi-document QA benchmarks compared with simpler grouping baselines, at lower compute than processing each document separately.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's counterfactual propagation copies 'duplicated' edges from the reference context to the counterfactual context, although the paper's own definitions imply the edge should be 'counterfactual'; this can merge contradictory contexts into one group and breaks the claimed separation…","rationale":"Good-faith reading: the paper's contribution is a specific grouping algorithm, and the central empirical claim depends on the graph constructor producing edges that reflect the four defined relationships. The reader's weakest_assumption pinpoints a real logical flaw in Algorithm 1; I checked it against the formal definitions in §2.1 and the example in Appendix A.1. The flaw is load-bearing because false duplicate edges cause the reranker to deduplicate genuinely counterfactual contexts, which is exactly the failure mode CORG is designed to avoid. I do not call the empirical results fraudulent; the issue is that the method as written cannot be expected to implement the claimed separation, and no code or graph validation is released to show that the actual implementation avoids the flaw. The verdict should therefore remain CONDITIONAL: the idea is plausible and the numbers are coherent, but the mechanism needs correction or verification before acceptance. No change from the reader's verdict.","tokens_in":18839,"tokens_out":6271,"duration_ms":62311,"concrete_test":"Implement Algorithm 1 exactly and feed it the three contexts from Appendix A.1 (main: IIHF/Slovakia; counterfactual: IIHF/Canada; duplicated: IIHF/Slovakia), using the paper's own GPT-4 labeling prompt for REL. Check whether the propagated edge between the counterfactual and duplicated contexts is labeled 'dup' rather than 'counter'. Then rerun CORG on the AmbigDocs+ subset with the corrected propagation rule (compute counterfactual/duplicated edges from descriptor and answer equality instead of copying the pivot's edge type) and compare D-F1; if the corrected graph changes group composition or D-F1 by a non-trivial amount, the reported mechanism is not the one being evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (Graph Constructor), lines 8–11, treats 'counter' and 'dup' as interchangeable for propagation: when the newly compared context ci is counterfactual with the pivot c′, it copies every edge (c′, cj, rj) to (ci, cj, rj). Under the definitions in §2.1, this is only valid when ci is duplicated with c′. If c′ is duplicated with cj but counterfactual with ci, then ci and cj share the same descriptor but have different answers, so they must be counterfactual, not duplicated. Concretely, in the Appendix A.1 example, the main context (IIHF, Slovakia), the counterfactual context (IIHF, Canada), and the duplicated context (IIHF, Slovakia) form exactly this pattern: the algorithm labels the counterfactual and duplicated contexts as duplicates. The reranker then randomly keeps one of them and loses one answer, directly undermining the stated design goal of separating counterfactual contexts into different groups. There is no reported validation of the GPT-4 relation labels or of the constructed graph, so this is not known to be corrected in implementation. The empirical D-F1 numbers may survive, but the mechanism as specified is internally inconsistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies four types of inter-context relationships in multi-document QA (distracting, ambiguous, counterfactual, duplicated), constructs extended datasets AmbigDocs+ and ConflictQA+ that mix all four factors, and analyzes simple per-factor solutions using Llama2-7B. It then proposes CORG, a training-free inference-time pipeline composed of a GPT-4-based graph constructor, a reranker that removes duplicates, drops descriptor-less contexts, and separates counterfactual contexts into groups, and an aggregator that pluralizes questions and concatenates cited answers. The paper reports D-F1 and related metrics across eight models and four datasets, claiming CORG consistently outperforms six baselines while matching the much more expensive Separate baseline, with the best performance-per-FLOPs trade-off on AmbigDocs+.","tokens_in":19101,"tokens_out":6384,"duration_ms":55064,"significance":"If the claims held, CORG would be a useful, practical contribution: an inference-time method that improves answer recall on mixed-relationship multi-document QA without training, with broad evaluation across model families and sizes. The paper includes human evaluation of the constructed datasets, ablations of the reranker/aggregator components, and a token-level efficiency analysis, which are valuable. However, the central mechanism has a documented logical inconsistency in the graph propagation rule, the evaluation design selects the per-factor solutions on the test corpora, the headline claim that CORG beats all six baselines is not supported by the reported tables, and the efficiency analysis excludes a per-query cost. These issues are load-bearing for the paper's main claims, though they may be addressable with additional validation and revised claims.","major_comments":[{"comment":"The mirroring rule is logically inconsistent with the definitions in §2.1. When the newly processed context ci is counterfactual to the pivot c′ (same descriptor, different answer) and c′ is duplicated with cj (same descriptor, same answer), the edge (ci, cj) must be counterfactual under the paper's own taxonomy, but the algorithm copies the 'dup' edge, labeling ci and cj as duplicates. In the Appendix A.1 example, the main context (IIHF, Slovakia), the counterfactual context (IIHF, Canada), and the duplicated context (IIHF, Slovakia) form exactly this pattern; the reranker then randomly keeps one of the two 'duplicates' and loses an answer, contradicting the stated design goal of separating counterfactual contexts. Because no accuracy check of GPT-4 relation labels or of the resulting graph is reported, this flaw is not known to be corrected in the implementation and can directly undermine the claimed mechanism.","section":"Algorithm 1 (Graph Constructor), lines 8–11"},{"comment":"The per-factor solutions (pluralizing the question, converting ambiguous contexts to distracting ones, separating counterfactual contexts, and deduplicating) are selected using Llama2-7B on AmbigDocs+ and ConflictQA+ instances, and the same corpora are then used for the final D-F1 evaluation in Tables 5, 6, and 10. The ablation in Table 11 is also on AmbigDocs+ with Llama2-7B. This means the design decisions are fit to the test distribution, and the reported gains over baselines may be optimistic. The paper should evaluate on a held-out development set or describe a model-selection procedure that does not use the test data.","section":"§3.2 and Tables 2–4; §5.3"},{"comment":"The claim of 'consistently improves performance over six baselines across eight different models' is not supported by the reported numbers. In Table 5, Separate exceeds CORG in multiple rows: ConflictQA+ Llama2-13B (37.5 vs 30.8), Llama2-70B (40.1 vs 31.4), Llama3-8B (25.6 vs 22.8), Llama3-70B (35.2 vs 28.7), and ChatGPT (37.1 vs 35.9); in AmbigDocs+, Separate exceeds CORG for Llama2-70B (18.7 vs 17.9), Llama3-70B (22.4 vs 14.6), Mistral-7B (28.8 vs 27.5), and GPT-4o (32.1 vs 31.4). Similar patterns appear in Table 10 for ConflictQA. The accurate claim is that CORG outperforms grouping-based baselines and is comparable to Separate, not that it outperforms all six baselines. This should be corrected throughout the abstract and results sections.","section":"Abstract; §5.3; Tables 5 and 10"},{"comment":"The efficiency analysis excludes graph construction from the FLOPs calculation, with the footnote stating that steps like graph construction are 'performed only once per dataset.' However, the graph constructor runs per question (Algorithm 1 takes the context list for a question), not once per dataset. Excluding this cost materially favors CORG in the performance/FLOPs comparison against baselines that do not perform relation extraction. The paper should either include the graph-construction cost or explicitly state that the comparison is for generation FLOPs only, then discuss the implication for the practical efficiency claim.","section":"§5.4, Figure 4, note 8"},{"comment":"The graph constructor uses an input format that adds a 'None' relationship (different descriptor with one being Null and different answer), which is not part of the four-type taxonomy in §2.1, and the prompt example in Figure 9 labels the pair 'Context2 - None' even though Context2 (Slovakia) and Context1 (Canada) appear to satisfy the counterfactual definition (same entity, different answers). This ambiguity in the labeling schema, combined with the absence of any accuracy or inter-annotator agreement measure for the GPT-4 relation labels, makes it impossible to verify that the constructed graph matches the paper's definitions. Please report the accuracy of relation labeling on a sample and reconcile the 'None' category with §2.1.","section":"Appendix C.1 and Figure 9"}],"minor_comments":[{"comment":"The text says 'We evaluate five baselines' but then lists six (Base, Retrieve, Summarize, Random, KMeans, Separate); §5.3 correctly says 'six baselines.' Please harmonize the count.","section":"§5.2"},{"comment":"The heading 'Affect of each factor' should be 'Effect of each factor.'","section":"§3.1 heading"},{"comment":"The reranker description says 'for contexts in a distracting relationship, when a context with a descriptor is available, we remove the one without it,' but a pair in which one context lacks a descriptor is ambiguous, not distracting; this appears to conflict with §3.2's 'Change to Distracting' solution and should be clarified.","section":"§4, Reranker paragraph"},{"comment":"Variant (2) is labeled 'without converting distracting to ambiguous,' which is the reverse of the §3.2 solution 'Change to Distracting'; please align the terminology with the prose.","section":"Table 11"},{"comment":"Section 3.1 refers to 'Figure 3' for performance trends, while Figure 3 is used for the CORG system overview; the cross-references and figure numbers should be fixed.","section":"Figure numbering"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical systems paper with a potentially useful framework, but the headline claim is contradicted by its own tables, the core graph propagation rule has a logical error that can merge counterfactual contexts, and the per-factor solutions are selected on the test corpora. These are correctable with additional validation (graph-label accuracy, held-out development set) and revised claims, so I do not recommend rejection, but the revisions are substantial enough to require a major round."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick version: this paper has a genuinely useful taxonomy and a wide empirical sweep, but the graph-construction algorithm at its core has a logical bug that can merge contradictory contexts, and the authors don't release code or validate the graph. I'd send it to review, not desk-reject.\n\nWhat's new and good: the paper organizes multi-document QA around four context relationship types — distracting, ambiguous, counterfactual, duplicated — and shows that no single intervention handles all four. The per-factor analysis (Tables 2–4) gives clean evidence that pluralization, changing ambiguous to distracting, separating counterfactuals, and deduplication each help, and the ablations in Table 11 show that the grouping and query reformulation are what drive the gains. The experiments are broad: eight models, two extended datasets with human checks on a 10% sample. The efficiency plot makes the practical case that CORG gets the best performance-to-FLOPs balance.\n\nThe soft spot is real, and it's load-bearing. Algorithm 1's mirroring rule assumes that if the pivot is counterfactual or duplicated with a new context, the new context's relations to the rest mirror the pivot's relations. That holds for duplicated, but not for counterfactual. If c' is duplicated with cj and counterfactual with ci, then ci and cj are counterfactual (same descriptor, different answers), but the algorithm copies the 'dup' edge. The reranker then randomly keeps one of the pair and drops an answer — exactly the failure mode the framework is supposed to prevent. The paper doesn't validate the GPT-4 relation labels or the resulting graph, so we don't know how often this happens. This is fixable — either compute relations directly or handle the counterfactual case separately — but as specified, the mechanism is inconsistent. The paper also admits it couldn't ablate the graph constructor (Appendix D.3), which is the component with the bug.\n\nOther issues are secondary. No code/data, no error bars; the D-F1 differences between CORG and the next-best baseline are often a couple of points, and we can't assess significance. The per-factor solutions were selected on the same benchmark distribution used for final evaluation, so there is some circularity. And the abstract says 'consistently improves over six baselines,' but Separate beats CORG in several rows of Table 5 — the claim is a bit generous.\n\nWho it's for: anyone working on RAG robustness or knowledge conflicts. The taxonomy and the extended datasets are useful contributions even if the framework needs repair.\n\nRecommendation: deserve a serious referee. The bug is technical, not conceptual, and the experimental breadth is worth pushing on. I'd ask for a corrected graph constructor, validation of the labels, and code/data.","headline":"A useful taxonomy and broad experiments, but Algorithm 1's mirroring rule is logically wrong for mixed counterfactual/duplicated triples, so the mechanism as written can drop answers; still worthy of peer review.","tokens_in":19605,"tokens_out":4355,"would_cite":true,"duration_ms":42853,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a graph-based, training-free context organizer raises disambiguated answer recall across eight models on question answering over complex, interrelated corpora.","keywords":["multi-document question answering","context relationships","distracting contexts","ambiguous contexts","counterfactual contexts","duplicated contexts","inference-time grouping","disambiguation"],"falsifier":"Construct a hand-made triple in which context A is counterfactual with B and duplicated with C (same descriptor, A and C share an answer, A and B do not), run Algorithm 1, and check whether the B-C edge is labeled counterfactual or copied as duplicated. Since the correct relation is counterfactual, a duplicated label directly falsifies the propagation mechanism; a supplementary check is to manually label all pairwise relations on a random AmbigDocs+ sample and report the graph constructor's disagreement rate.","tokens_in":18609,"feed_emoji":"🧩","tokens_out":10295,"duration_ms":89634,"temperature":0.7,"pith_summary":"Real-world corpora rarely present one clean answer: the same named entity can appear with different descriptors, with no descriptor at all, with contradictory facts, or with the same fact repeated verbatim. The paper classifies these as distracting, ambiguous, counterfactual, and duplicated relationships, and claims that no single prompt-level trick handles all four at once. It introduces CONTEXT ORGANIZER (CORG), an inference-time, training-free pipeline that builds a graph of pairwise context relations, reranks contexts into groups, and aggregates per-group, citation-backed answers. Across eight models of different sizes, CORG reports the highest disambiguated answer recall (Disambig-F1) among grouping methods on AmbigDocs+, ConflictQA+, and their single-factor bases, beating six baselines and roughly matching the far more expensive one-context-per-run method at lower cost. The practical claim is that how contexts are grouped for generation matters more than model scale, and that a cheap organization step can capture most of the benefit of full isolation.","feed_headline":"Context grouping beats six baselines in messy multi-answer QA","feed_subtitle":"Splits conflicting contexts, drops duplicates, and pluralizes the question to lift disambiguation recall.","key_machinery":"The load-bearing object is the graph constructor of Algorithm 1: it labels pairwise context relations with GPT-4, then propagates labels by assuming that when two contexts are counterfactual or duplicated, their relations to every third context are identical, so only missing edges need labeling in later iterations. The reranker consumes this graph and applies per-factor rules: for distracting pairs it removes the context lacking a descriptor; for duplicated pairs it keeps one randomly; and for counterfactual pairs it places each context in a separate group, distributing the remaining contexts across groups. The aggregator then runs each group through the model with a pluralized question and concatenates the outputs with citations. Together, the three components convert the four relationship types into a small number of independent inference runs, which is what produces the reported gains in answer recall and disambiguation.","core_discovery":"The paper's central claim is that multi-document question answering over corpora with mixed context relationships is best served by separating counterfactual contexts into different inference groups, deduplicating repeated contexts, dropping ambiguous descriptor-less contexts when a distracting counterpart exists, and reformulating the question into a plural form so the model lists all answers. On AmbigDocs+ and ConflictQA+, CORG's grouping yields the best Disambig-F1—a model-based metric that checks whether the correct answer and its disambiguating descriptor both appear—among six baselines across all eight tested models, and it matches the much more expensive Separate method, which runs each context in its own forward pass. A supporting finding is that grouping by similarity (KMeans) is worse than random grouping, because similar-but-conflicting contexts blur entity descriptors; this locates the benefit in the graph's relationship labels rather than in clustering or plain input ordering.","pith_inferences":["The propagation rule in Algorithm 1 is not generally truth-preserving: if context A is counterfactual with B and duplicated with C, then B and C must be counterfactual, yet the copy rule labels B-C as duplicated. A direct check against human relation labels on AmbigDocs+ would reveal how often this error changes the groups and the final answer.","The paper reports no accuracy check for the GPT-4 relation labels feeding the graph. A controlled comparison using a cheaper labeler or human labels would separate the benefit of the grouping policy itself from the benefit of a labeler that happens to agree with the generator's biases.","The pluralization result points to a testable refinement: make question reformulation depend on the number of distinct descriptors in each group, rather than applying plural form to every multi-context group, which could reduce the over-generation the paper notes for single-context inputs."],"forward_implications":["On corpora that mix all four relationship types, an inference-time grouping step can beat both single-pass reading and similarity-based clustering without any model training.","Because each group's output carries citations, users can verify counterfactual or outdated claims from the source contexts instead of relying on the model to filter them.","The method transfers to newly released models directly, since its only adaptation is prompt-level graph labeling; the paper demonstrates consistent gains across eight models of different sizes.","Typical runs use roughly two inference groups per question rather than one group per context, so the answer-recall gains of separated processing are available at a small fraction of the cost."],"supporting_citations":[{"why":"Supplies the AmbigDocs dataset and the entity/answer/D-F1 evaluation protocol the paper extends and reports.","marker":"Lee et al. (2024c)"},{"why":"Supplies ConflictQA, the counterfactual-context dataset that ConflictQA+ builds on.","marker":"Zhou et al. (2023)"},{"why":"GPT-4 is used for relation labeling, context generation, and the Retrieve/Summarize baselines.","marker":"Achiam et al. (2023)"},{"why":"Introduces the Disambig-F1 metric used as the primary outcome measure.","marker":"Stelmakh et al. (2022)"},{"why":"Llama2 chat models of three sizes carry the factor analyses and part of the main evaluation grid.","marker":"Touvron et al. (2023)"},{"why":"BERT embeddings define the KMeans baseline, the key contrast showing similarity-based grouping hurts.","marker":"Devlin et al. (2019)"},{"why":"Inspires the Summarize baseline, the strongest single-inference competitor CORG is compared against.","marker":"Xu et al. (2023)"},{"why":"Provides the diversity-based ranking idea behind the Retrieve baseline.","marker":"Min et al. (2021)"}],"fun_headline_variants":["CORG beats six baselines by grouping conflicting contexts","Separating counterfactual contexts lifts QA disambiguation","Pluralizing questions helps multi-doc answer recall","Graph-based context grouping matches costly single-pass baseline","Context relationship labels beat KMeans grouping for QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that once two contexts are marked counterfactual or duplicated, every other context relates to them in exactly the same way, so one context's relation labels can be copied onto the other without rechecking.","fun_headline_variants_meta":{"raw":{"variants":["CORG beats six baselines by grouping conflicting contexts","Separating counterfactual contexts lifts QA disambiguation","Pluralizing questions helps multi-doc answer recall","Graph-based context grouping matches costly single-pass baseline","Context relationship labels beat KMeans grouping for QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1705,"prompt_tokens":873,"completion_tokens":832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":757}},"tokens_in":489,"tokens_out":832,"duration_ms":8827,"temperature":1.0,"reasoning_tokens":757,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:26:31.543136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a hand-made triple in which context A is counterfactual with B and duplicated with C (same descriptor, A and C share an answer, A and B do not), run Algorithm 1, and check whether the B-C edge is labeled counterfactual or copied as duplicated. Since the correct relation is counterfactual, a duplicated label directly falsifies the propagation mechanism; a supplementary check is to manually label all pairwise relations on a random AmbigDocs+ sample and report the graph constructor's disagreement rate.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies ConflictQA, the counterfactual-context dataset that ConflictQA+ builds on."}],"review_version":1}