{"id":"53920fbe-1ebd-4ff6-8e4a-5acfc05afab1","arxiv_id":"2508.01273","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KCR uses logic-trace disentanglement and reinforcement learning to improve LLM adjudication of contradictory contexts, reporting strong gains on popQA and strategyQA.","lead":"This paper presents KCR, a framework that trains large language models to resolve contradictory retrieved information by first breaking the conflicting evidence into structured reasoning paths and then reinforcing a coherent adjudication policy. It reports that a 7B model trained this way outperforms much larger commercial models on two knowledge-conflict benchmarks, though the comparison is against zero-shot proprietary baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline superiority over GPT-5.1 rests on an uncontrolled comparison: KCR is trained on the benchmark split while proprietary baselines are zero-shot, and the identical CUAD/AdaCAD/CoCoA rows in Table 2 suggest unreliable baselines.","rationale":"The reader's weakest_assumption is correct. I read the full manuscript in good faith: the disentangling pipeline is clearly described, the reward design is novel, and the internal ablations (Figure 4) support that the coherence/consistency rewards add value over SFT and vanilla GRPO. Those internal results are not the problem. The problem is the headline claim 'outperform GPT-4o and GPT-5.1', which depends entirely on Table 2. The table compares models trained on the benchmark's training split against zero-shot proprietary baselines; that is a train/test asymmetry, not a property of the method. The identical CUAD/AdaCAD/CoCoA rows reinforce the concern that the baseline column was not carefully produced. A matched-protocol rerun is the single check that would settle it. If the gap to a fine-tuned or in-context-adapted GPT-5.1 disappears, the paper's contribution should be reframed as improving open-weight backbones over zero-shot closed models, not as beating GPT-5.1. Because the method itself appears promising and the concern is about the comparative framing, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":15249,"tokens_out":5209,"duration_ms":60937,"concrete_test":"Run a matched-protocol comparison: (1) execute CUAD, AdaCAD, and CoCoA from their released code on the same test split and confirm the three rows are not numerically identical; (2) fine-tune GPT-5.1 (or, if API fine-tuning is unavailable, use 8-shot in-context examples drawn from the KCR training split) and evaluate it with the same judge. If a fine-tuned or adapted GPT-5.1 reaches ACC_L close to KCR's 86.67-87.5% on popQA, or if the corrected CUAD/AdaCAD/CoCoA values differ materially, the 'significantly outperform GPT-5.1' claim is not supported and should be downgraded to a claim about KCR vs zero-shot baselines.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is comparative: a 7B KCR model 'significantly outperform[s] leading proprietary models, including GPT-4o and GPT-5.1' (abstract; Table 2). The load-bearing premise is that Table 2's comparisons are apples-to-apples. They are not. Section 4.1 partitions popQA and strategyQA into 8:1:1 train/val/test, and KCR is trained for 10 epochs on the training split; the proprietary baselines (GPT-4o, GPT-5.1, Astute RAG) are evaluated without fine-tuning on that split. The reported gaps (e.g., KCR Llama 3.2-3B 87.5% ACC_L vs Astute RAG GPT-5.1 66.94% on popQA) therefore conflate the method's contribution with the advantage of supervised exposure to the benchmark's answer distribution and prompt format. This is especially damaging because the RLVR reward in Eq. (6) uses ground-truth labels ('let A1 denote the correct answer'), so training directly optimizes the test-like objective. Internal ablations (Figure 4) suggest KCR helps relative to SFT/GRPO, but they do not establish superiority over fine-tuned proprietary models. Additionally, in Table 2 the CUAD (Default), ADACAD (Default), and COCOA (Default) rows list identical values on all metrics for both datasets (popQA: 0.3431/0.2958/0.3056; strategyQA: 0.5040 for all), which is implausible for three distinct decoding strategies and signals that the baseline table may contain placeholder or mislabeled results. If so, the 'Overall Improvement vs SOTA' row is unreliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KCR (Knowledge Conflict Reasoning), a two-stage framework for explicit knowledge conflicts. In the first stage, it decomposes each candidate answer's supporting context into textual reasoning traces and local knowledge graphs. In the second stage, it trains a backbone LLM with GRPO using two reward signals: a logic coherence reward based on JS divergence between consecutive reasoning steps, and a logic consistency reward based on similarity between the generated trace/answer and candidate trace/answer sets. The authors evaluate on popQA and strategyQA against CAD, CUAD, AdaCAD, CoCoA, and Astute RAG, and report large gains over backbone models, claiming that KCR-enhanced 7B and 8B models outperform GPT-4o and GPT-5.1. The paper also reports ablations showing that both rewards contribute and provides qualitative cases of structured adjudication.","tokens_in":15633,"tokens_out":5382,"duration_ms":62694,"significance":"If the empirical results are taken at face value, KCR would be a meaningful advance for conflict adjudication: the hybrid text/graph trace disentanglement and the RLVR-style two-reward design are a principled attempt to separate valid from spurious reasoning paths. The paper releases code, provides detailed prompts, and is candid about limitations such as external LLM dependency, dyadic-only conflict scope, and English-only evaluation. However, the headline comparative claims are currently not supportable because the evaluation protocol does not match the training conditions of the proprietary baselines and because the baseline table contains an implausible artifact. The internal ablations and backbone-level improvements are plausible and interesting, but the central superiority claim needs a corrected and fairer comparison before it can be accepted.","major_comments":[{"comment":"The headline comparison against proprietary models is not apples-to-apples. KCR is trained for 10 epochs on the 8:1:1 training split of popQA and strategyQA, while the proprietary baselines (GPT-4o, GPT-5.1, and Astute RAG built on them) are evaluated zero-shot or with prompting only. The reported gaps, such as KCR with Llama 3.2-3B reaching 87.5% ACC_L on popQA versus 66.94% for Astute RAG (GPT-5.1), therefore conflate the method's contribution with supervised exposure to the benchmark's answer distribution and prompt format. The 'Overall Improvement vs SOTA' row should be recomputed after adding a fine-tuned or in-distribution proprietary baseline, or after evaluating a KCR variant that is not trained on the benchmark, and the paper should state the training/evaluation asymmetry explicitly.","section":"Section 4.1, Table 2"},{"comment":"The rows for CUAD (Default), ADACAD (Default), and COCOA (Default) report numerically identical values on all three metrics for both datasets (popQA: 0.3431/0.2958/0.3056; strategyQA: 0.5040 for all). Three distinct decoding strategies cannot plausibly produce identical results at this precision, which suggests placeholder or mislabeled values. These rows must be rerun and corrected, and the final 'Overall Improvement vs SOTA' row must be recomputed from the corrected values.","section":"Table 2"},{"comment":"The coherence reward uses the ground-truth answer to choose which reference trace set is 'correct' ('Without loss of generality, let A1 denote the correct answer'). This is a legitimate training signal, but it means the reward is partly a re-encoding of answer labels into a reasoning-style signal. The paper should clearly separate this from the claim in Section 3.2.2 that 'ground-truth labels are not utilized' for the consistency reward. Moreover, Section 3.2.3 introduces an R_correct term in the cumulative reward without defining it; if that term is also label-based, the no-label claim is further weakened and the composition of the final reward needs to be specified precisely.","section":"Section 3.2.1, Eq. (6)"},{"comment":"The coherence score l(·) is a scalar sum of JS divergences between consecutive steps in a single trace, and Δl compares that scalar to the scalar of a reference trace set. This does not measure whether the generated trace's steps are structurally or semantically aligned with the correct trace; two logically unrelated chains can have identical aggregate divergence. The reward therefore does not implement the stated objective of 'approximat[ing] the logical structure present in valid contexts.' The authors should either define a trace-level alignment score or provide evidence that aggregate JS divergence correlates with trace-level correctness.","section":"Section 3.2.1, Eqs. (4)-(7)"},{"comment":"Normalized Levenshtein similarity is defined for strings, but the reward compares 'sets' of reasoning traces RT_i as well as answer strings A_i. The manuscript does not define how set-level Levenshtein similarity is computed (e.g., maximum over pairs, average over pairs, or a sequence alignment over the set). The reward's behavior depends on this choice, so the definition must be specified and the chosen default justified with an ablation or example.","section":"Section 3.2.2, Eqs. (8)-(9)"}],"minor_comments":[{"comment":"The notation is inconsistent: the summation limit uses len(R_i) while the text uses |R_i| for the number of steps; use one notation throughout.","section":"Section 3.2.1, Eq. (4)"},{"comment":"The symbol ε is used both for the numerical stability constant in the advantage normalization (Eq. 10) and for the GRPO clipping parameter (Eq. 12); use distinct symbols to avoid ambiguity.","section":"Section 3.2.3, Eqs. (10) and (12)"},{"comment":"The cumulative reward is defined as ri = R_coh + R_con + R_correct, but R_correct is never formally defined; define it explicitly and state whether it uses the ground-truth answer.","section":"Section 3.2.3"},{"comment":"There is a typo in the heading: 'Reprodicibility Statement' should be 'Reproducibility Statement'.","section":"Appendix A.2"},{"comment":"The LLaMA answer extraction patterns list several alternative answer_patterns without explaining the order or conditions under which each is applied; clarify how the final answer string is selected.","section":"Appendix B.1"},{"comment":"The figures would benefit from explicit axis labels and error bars, especially since the paper states that experimental results are averaged over three runs; currently the reported variability is not visible.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I want to flag the identical CUAD/ADACAD/COCOA rows in Table 2 as a data-integrity concern rather than a mere presentation issue; the authors should be asked to provide logs or running scripts for these baselines before further review. My recommendation is major_revision rather than reject because the methodological core is plausible and well-scoped, and the central comparative claim may be salvageable with a fair evaluation protocol and corrected baseline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for its core idea: instead of asking a model to adjudicate conflicting contexts on the fly, KCR first disentangles each side into structured reasoning traces (textual chains and local knowledge graphs), then uses RLVR with coherence and consistency rewards to teach a reasoning policy. That combination is genuinely new, and the internal ablations in Figure 4 show the method does much better than SFT or vanilla GRPO on the same backbone. The authors also ship code and are candid about the computational cost and the dyadic/English-only scope. Credit where it's due: the framework is coherent and the internal gains look real.\n\nThe problems are in the headline comparisons. The abstract says a 7B KCR model outperforms GPT-4o and GPT-5.1, but those proprietary models are evaluated zero-shot while KCR is trained for 10 epochs on the training split of popQA and strategyQA. That conflates method quality with benchmark exposure. The stress-test note is right, and reading the paper doesn't dissolve it — Table 2 confirms the setup. Worse, the rows for CUAD, AdaCAD, and CoCoA (Default) are numerically identical on all metrics for both datasets. Three distinct decoding strategies producing exactly the same numbers is implausible; that looks like placeholder values, and it makes the \"Overall Improvement vs SOTA\" row unreliable.\n\nThere are also methodological soft spots in the rewards. The coherence reward in Eq. (4)-(6) compares the generated trace to the reference sets via the absolute difference of a smoothness measure (sum of JS divergences between adjacent steps), not a structural or semantic match. So a generated trace that is simply \"smooth\" can be rewarded even if its content is wrong. The consistency reward in Eq. (8) uses a \"normalized Levenshtein similarity\" that is never formally defined, and it compares a generated chain against a set of reference traces without saying how the set-level similarity is computed. And Eq. (6) explicitly uses ground-truth labels to decide which side is \"correct\" for the coherence reward, so the training signal partly re-encodes the answer labels into a reasoning-style reward. These are fixable, but they weaken the stated mechanism.\n\nAll that said, the central idea is plausible and the internal results are meaningful. I would send this to review, with a clear request: control the comparison (add a fine-tuned proprietary baseline, or evaluate KCR zero-shot), re-run the suspicious baseline rows, and define the similarity functions properly. A reader interested in knowledge conflicts or RLVR for reasoning will get value from the framework even if the headline claim needs to be walked back.","headline":"Worth a referee's time for the KCR framework and internal gains, but the headline '7B beats GPT-5.1' claim is unsupported because the comparison is uncontrolled.","tokens_in":16152,"tokens_out":1960,"would_cite":false,"duration_ms":25596,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that by disentangling conflicting contexts into text and graph reasoning traces and training with verifiable logical-coherence and consistency rewards, KCR lets small open models adjudicate explicit knowledge conflicts…","keywords":["explicit knowledge conflicts","knowledge conflict reasoning","reasoning traces","local knowledge graphs","reinforcement learning with verifiable rewards","GRPO","logic coherence reward","retrieval-augmented generation"],"falsifier":"Re-run the three decoding baselines (CUAD, AdaCAD, CoCoA) independently on popQA and check whether their ACC_L, ACC_EM, and ACC_CEM values really are identical; if they remain numerically identical across distinct decoding strategies, the baseline column is not trustworthy. Then fine-tune a strong proprietary model (e.g., GPT-5.1) on the same training split and evaluate it on the same test split: if its ACC_L reaches or exceeds KCR's 87.5% on popQA, the claim of surpassing proprietary models reduces to a training-asymmetry effect.","tokens_in":15048,"feed_emoji":"⚖️","tokens_out":9394,"duration_ms":96836,"temperature":0.7,"pith_summary":"The paper proposes KCR (Knowledge Conflict Reasoning), a framework that tries to establish that explicit knowledge conflicts—questions paired with two candidate answers and two contradictory supporting contexts—can be resolved reliably by structurally disentangling the underlying reasoning logic rather than by scaling the model or hand-crafting fixed fusion rules. It claims that the root difficulty is logic entanglement and case heterogeneity, and that converting conflicting contexts into discrete textual and graph reasoning traces, then training the backbone with reinforcement learning over verifiable rewards for logical coherence and consistency, instills a generalizable adjudication policy. On the popQA and strategyQA benchmarks the authors report that KCR lifts a Llama 3.2-3B backbone to 87.5% semantic accuracy on popQA and a Qwen 2.5-7B backbone to 86.67%, outperforming Astute RAG running on GPT-5.1, which reaches 66.94%. If this holds, conflict adjudication becomes trainable into small open models instead of depending on proprietary reasoning systems, and the method offers a template for reasoning-path supervision where no single gold step-level chain exists.","feed_headline":"New training method lets 7B models beat GPT-5.1 on conflicts","feed_subtitle":"By splitting contradictions into text and graph reasoning chains, small open models jump past proprietary baselines.","key_machinery":"The load-bearing object is the paired reasoning-trace representation: for each candidate answer, a set of textual traces $RT^{(T)}_i$ and a set of graph traces $RT^{(G)}_i$ are extracted, and these reference sets serve as the target structure the generator must align with. The training machinery is GRPO with a composite reward $R_{\\text{coh}} + R_{\\text{con}} + R_{\\text{correct}}$, where the coherence term is a binary comparison of absolute differences in Jensen-Shannon divergence between the generated trace and the two reference sets, and the consistency term is a binary agreement check between the generated answer and generated trace on the same candidate side. The design choice of discrete rewards matters: the authors argue it sharpens group-relative advantage estimation during RLVR sampling and stabilizes convergence.","core_discovery":"On its own terms, the paper's central claim is that a model's failure on explicit knowledge conflicts is a reasoning-structure failure, not a scale failure, and that this failure can be corrected by a two-phase procedure. First, KCR decomposes each candidate answer and its context into two complementary sets of reasoning traces: textual traces written as alternating entity-to-relation-to-entity chains, and graph traces taken as directed paths in a locally constructed knowledge graph that intersect the query entity or relation. Second, it trains the backbone as a policy with GRPO, scoring each rollout with a binary logic coherence reward that compares the Jensen-Shannon divergence of the generated reasoning chain against the correct and incorrect trace sets, plus a binary logic consistency reward that checks the generated answer and the generated trace side with the same candidate, using normalized Levenshtein similarity. The authors report that the trained 3B and 7B models substantially exceed both their backbones and the strongest proprietary baseline, with the discrete reward variants giving stable convergence and the coherence reward carrying the main improvement in the ablation.","pith_inferences":["Editorial inference: the reported margin over GPT-5.1 may be inflated by an asymmetry in training exposure—KCR is fine-tuned on the benchmark's training split while the proprietary baselines are evaluated zero-shot or with prompting only—so a matched fine-tuned proprietary baseline could shrink the gap.","Editorial inference: the method's dependence on GPT-4o-mini for knowledge-graph construction and as judge introduces an API dependency; a fully open variant that builds graphs with an open extractor would test whether the graph traces themselves, rather than the proprietary extractor, cause the gain.","Editorial inference: the binary coherence reward compares the generated chain against pre-extracted trace sets, so the whole pipeline inherits the quality of those extractions; if the frozen Qwen2.5-7B or GPT-4o-mini extractor makes systematic errors, the reward can reinforce the wrong side.","Editorial inference: the dyadic two-candidate formulation leaves open whether the policy transfers to multi-party conflicts or partially correct sources; a direct extension would construct trace sets for three or more candidates and score coherence against each, testing whether the learned policy scales beyond binary choice."],"forward_implications":["If the central claim is right, explicit knowledge conflict adjudication no longer requires frontier closed-source models: 3B and 7B open backbones trained with KCR can surpass a GPT-5.1-based Astute RAG pipeline on the evaluated benchmarks.","The ablations imply that the logic coherence reward is the main source of improvement, which means the value of the method comes primarily from structurally comparing generated reasoning with correct versus incorrect trace sets, not from the consistency check alone.","The success of both text and graph trace variants across backbones suggests that the learned adjudication policy is not tied to a single surface format, and that graph traces can help even when the backbone was not explicitly trained on graph inputs.","Because the reward is verifiable at the process level rather than token level, the method applies to conflict scenarios where there is no single gold reasoning chain, a situation that rules out ordinary supervised step-level training.","KCR trained on popQA and strategyQA is reported to improve performance specifically on questions requiring stricter logical chains, so the method's benefit should concentrate on multi-step adjudication rather than trivial factual recall."],"supporting_citations":[{"why":"Defines the explicit knowledge conflict task and supplies the popQA and strategyQA datasets that KCR is evaluated on.","marker":"Xie et al., 2024"},{"why":"Supplies GRPO, the group-relative policy optimization algorithm used to train the backbone with the verifiable rewards.","marker":"Shao et al., 2024"},{"why":"Motivates the local knowledge graph construction used for the graph reasoning traces.","marker":"Edge et al., 2024"},{"why":"Provides the Jensen-Shannon divergence used to score logical coherence between reasoning traces.","marker":"Lin, 1991"},{"why":"Provides the KL divergence from which the JS coherence score is built.","marker":"Kullback and Leibler, 1951"},{"why":"Provides the normalized Levenshtein similarity used in the logic consistency reward.","marker":"Haldar and Mukhopadhyay, 2011"},{"why":"Supplies the LLM-as-judge protocol used to compute the semantic accuracy metric ACC_L.","marker":"Zheng et al., 2023"},{"why":"Astute RAG is the strongest proprietary baseline that KCR is compared against.","marker":"Wang et al., 2025a"}],"fun_headline_variants":["7B model beats GPT-5.1 by disentangling conflict logic","Graph+text reasoning lets 7B top GPT-5.1 on contradictions","RLVR-trained 7B surpasses GPT-5.1 via logic traces","Splitting logic into text and graph makes 7B beat GPT-5.1","Small model outperforms GPT-5.1 after logic disentanglement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the performance comparison is fair: KCR is fine-tuned on the training split of popQA and strategyQA, whereas the GPT-4o and GPT-5.1 baselines it beats are only prompted, so the observed gap could reflect benchmark exposure rather than the disentanglement mechanism itself.","fun_headline_variants_meta":{"raw":{"variants":["7B model beats GPT-5.1 by disentangling conflict logic","Graph+text reasoning lets 7B top GPT-5.1 on contradictions","RLVR-trained 7B surpasses GPT-5.1 via logic traces","Splitting logic into text and graph makes 7B beat GPT-5.1","Small model outperforms GPT-5.1 after logic disentanglement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1762,"prompt_tokens":951,"completion_tokens":811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":708}},"tokens_in":567,"tokens_out":811,"duration_ms":9074,"temperature":1.0,"reasoning_tokens":708,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:42:16.291477+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three decoding baselines (CUAD, AdaCAD, CoCoA) independently on popQA and check whether their ACC_L, ACC_EM, and ACC_CEM values really are identical; if they remain numerically identical across distinct decoding strategies, the baseline column is not trustworthy. Then fine-tune a strong proprietary model (e.g., GPT-5.1) on the same training split and evaluate it on the same test split: if its ACC_L reaches or exceeds KCR's 87.5% on popQA, the claim of surpassing proprietary models reduces to a training-asymmetry effect.","supporting_citations":[{"cited_title":"<think>(.*?)</ think>","cited_arxiv_id":null,"evidence_quote":"Motivates the local knowledge graph construction used for the graph reasoning traces."}],"review_version":1}