{"id":"e007da6b-c02a-462a-bffd-62d935a83901","arxiv_id":"2508.17670","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"CoCoA, a confidence- and context-aware adaptive decoding algorithm, resolves knowledge conflicts between an LLM's parametric memory and external context and reports state-of-the-art gains over AdaCAD.","lead":"This paper introduces CoCoA, a decoding method that decides when an LLM should trust its training memory versus newly supplied text. It reports large accuracy and factuality gains over an existing adaptive decoding baseline across QA, summarization, and long-form QA tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Conflict-detection statistics may conflate knowledge conflict with ordinary distribution shift; selectivity is untested.","rationale":"The reader's weakest assumption identified the same core risk: CoCoA's adaptive rule depends on statistics that must separate conflict from ordinary uncertainty. My stress-test agrees and makes the concern more concrete—specifically, contextual peakedness and divergence can be high even when context is merely informative, not conflicting. Because the abstract does not include ablations, theoretical motivation, or threshold sensitivity analysis, the central claim remains unverified. This does not move the verdict away from UNVERDICTED; it reinforces that the paper cannot be accepted on the abstract alone. A targeted probe test would determine whether the conflict signal is selective or merely a proxy for context informativeness.","tokens_in":644,"tokens_out":2135,"duration_ms":29185,"concrete_test":"Obtain the authors' code (or re-implement) and run CoCoA on a constructed probe set where the provided context is topically relevant but non-conflicting (e.g., generic Wikipedia paragraphs appended to questions whose parametric answer is correct). Measure the fraction of tokens for which CoCoA switches from the parametric distribution to the contextual distribution. If this switch rate is materially nonzero in the no-conflict condition, the conflict-detection signal is not selective and the 'low conflict' performance claim is at risk.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CoCoA's token-level override rule is driven by confidence-aware measures (entropy gap, contextual peakedness) and a generalized divergence between parametric and contextual distributions. For that rule to be principled, these scalar signals must separate true knowledge conflict from benign deviations between the two distributions. The abstract reports strong QA/summarization gains, but provides no evidence—theoretical or empirical—that the constructed thresholds are invariant across domains, models, or context styles. If, for instance, contextual peakedness is high whenever in-context information is locally coherent (regardless of whether it contradicts parametric memory), CoCoA will override memory in many non-conflict settings; the abstract's claim that it 'maintains strong performance even in low conflict settings' would then rest on benchmark-specific tuning rather than a general mechanism. Since the full text is unavailable, this is the weakest link: the claimed SOTA gains could be an artifact of fitting thresholds to the specific benchmark distribution of conflict frequencies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoCoA, a token-level adaptive decoding algorithm aimed at resolving knowledge conflicts between an LLM's parametric memory and external context. The proposed method uses confidence-aware measures (entropy gap and contextual peakedness) and a generalized divergence between parametric and contextual distributions to decide when context should override memory. The abstract reports state-of-the-art performance over a strong baseline (AdaCAD) on QA, summarization, and long-form QA, with gains up to 9.2 and 2.5 average points respectively, while also claiming maintained performance in low-conflict settings and superior sensitivity to conflict variations.","tokens_in":941,"tokens_out":1752,"duration_ms":24355,"significance":"If the claims are substantiated, CoCoA would address an important practical problem in faithful generation: avoiding degradation when no conflict is present while still resolving genuine conflicts. The proposal of a confidence- and context-aware adaptive rule is a reasonable next step beyond contrastive decoding methods that are tuned specifically for conflict-heavy settings. The abstract's emphasis on robustness in low-conflict settings and sensitivity to conflict variations is well aligned with real deployment concerns. However, because the full text is not available, none of the technical details or empirical results can be verified. The significance is conditional: the idea is promising, but the evidence needed to establish it is entirely absent from the abstract.","major_comments":[{"comment":"The abstract states gains 'up to 9.2 points on average' over AdaCAD, but this is a maximum reported gain, not a robust measure of central tendency, and 'on average' is ambiguous. No error bars, number of benchmarks, model/seed variance, or per-task results are given. Without these, the central empirical claim cannot be assessed from the abstract alone.","section":"Abstract, headline empirical claim"},{"comment":"The core mechanism—'entropy gap,' 'contextual peakedness,' and 'generalized divergence'—is named but not defined. No equations, threshold constructions, or invariance properties are provided. This leaves unresolved the load-bearing question of whether these scalar statistics separate true knowledge conflict from ordinary distribution shift; if they do not, the adaptive rule could degrade to threshold-tuned switching rather than principled conflict resolution.","section":"Abstract, mechanism definition"},{"comment":"This claim is vague: no metric, experimental protocol, or baseline comparison for sensitivity is described. It is impossible to determine whether 'sensitivity' is measured by accuracy under varying conflict frequencies, by calibration, or by some other quantity. This claim is central to the paper's adaptive-decoding identity and needs a concrete definition and evaluation.","section":"Abstract, 'superior sensitivity to conflict variations'"}],"minor_comments":[{"comment":"The phrase 'points on average' is ambiguous; it could mean percentage points or absolute accuracy differences. Clarify the units and aggregation procedure.","section":"Abstract, terminology"},{"comment":"'Generalized divergence' is not formally specified. At minimum, the abstract should indicate the divergence family (e.g., Jensen–Shannon, Rényi) so that the claim is interpretable.","section":"Abstract, notation"}],"recommendation":"uncertain","confidential_remarks":"This review is necessarily abstract-only, so the recommendation is uncertain rather than accept/reject. The editor may want to obtain the full text before further consideration. The main risk, as flagged in the stress-test note, is that the reported gains could be benchmark-specific tuning artifacts of the conflict-detection thresholds. The abstract does not provide enough information to rule that out. I recommend proceeding only with full experimental details and definitions of the adaptive rule."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the abstract proposes a sensible decoding patch—use confidence and distribution-divergence signals to decide when context should override parametric memory—and the evaluation plan is the right shape. But there is no way to tell from the abstract whether the conflict-detection statistics are actually separating conflict from ordinary distribution shift, or whether the reported gains are benchmark-specific threshold tuning.\n\nWhat's new: the combination of entropy gap, contextual peakedness, and generalized divergence as a token-level override rule. The individual pieces are known (AdaCAD, contrastive decoding), but the paper's pitch is that the combination avoids the low-conflict degradation. That's a legitimate and useful thing to test.\n\nWhat looks good: the paper targets a real failure mode, claims multiple LLMs and three task families, and compares against AdaCAD, a strong baseline. The 'up to 9.2 points' phrasing is a max, not a central tendency, but the fact that they report a baseline comparison at all is a plus.\n\nSoft spots: without the full text we have no equations, no threshold values, no ablations, no error bars, no code. The central worry is selectivity: the paper assumes that low entropy and high contextual peakedness mean 'conflict,' but those signals could simply indicate local coherence. If so, CoCoA would override memory in plenty of non-conflict settings, and the 'maintains strong performance in low conflict' claim could be a tuned artifact. That concern doesn't land as a verdict yet—it's exactly what the missing sections would answer.\n\nWho it's for: people building retrieval-augmented generation pipelines who care about when to trust retrieved text over the model's memory. It's not a theory paper; it's a system/methods paper.\n\nRecommendation: yes, send it to review. The claims are concrete and falsifiable, the problem is important, and the baseline comparison is the right one. A reviewer should ask for the selectivity analysis—how often does CoCoA override in benign contexts—and for threshold sensitivity.","headline":"CoCoA offers a plausible adaptive decoding fix for knowledge conflicts, but the abstract alone can't show the conflict signals aren't just tuned to the benchmarks.","tokens_in":1302,"tokens_out":2408,"would_cite":false,"duration_ms":29559,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoCoA resolves LLM knowledge conflicts with confidence- and context-aware token decoding.","keywords":["knowledge conflicts","faithful generation","adaptive decoding","entropy gap","contextual peakedness","divergence","question answering","summarization"],"falsifier":"Run CoCoA on a dataset where contexts are deliberately perturbed to contradict the model when the model is right, and check whether its token-level choices track an oracle that knows which context tokens are actually true; if CoCoA's accuracy on non-conflicting tokens drops below a static context-preference decoding (or below plain parametric decoding) while its conflict statistics are indistinguishable from ordinary uncertainty, the central claim would be false.","tokens_in":631,"feed_emoji":"🎯","tokens_out":3170,"duration_ms":31319,"temperature":0.7,"pith_summary":"The paper introduces CoCoA, a token-level decoding algorithm that decides, for each generated token, whether the model should trust its parametric memory or an external context (e.g., a retrieved passage). It uses two confidence-aware statistics (entropy gap and contextual peakedness) plus a generalized divergence between the parametric and contextual token distributions to detect when a conflict is real. The claim is that this adaptive rule improves faithful generation across QA, summarization, and long-form QA benchmarks while retaining strong performance when conflicts are rare. If true, it would give a lightweight, training-free way to make LLMs more faithful to provided context without sacrificing accuracy on non-conflicting inputs.","feed_headline":"Adaptive decoding gains up to 9.2 QA points over AdaCAD","feed_subtitle":"CoCoA uses entropy gap and divergence to decide when context should override a model's memory.","key_machinery":"The central objects are three token-level statistics: entropy gap (difference in entropy between the parametric and contextual next-token distributions), contextual peakedness (concentration of the contextual distribution on the highest-probability token), and a generalized divergence (a measure of disagreement between the parametric and contextual distributions). Together they act as a conflict detector: when the gap is large, the context is peaked on a different token, and the divergence is high, CoCoA's decoding distribution is steered toward the context; when signals are weak, it falls back to the parametric distribution. This per-token, context-sensitive gating is what allows CoCoA to s","core_discovery":"CoCoA's central claim is that knowledge conflicts can be resolved at generation time by monitoring the disagreement between what the model 'knows' and what the context says. At each token step, CoCoA measures an entropy gap (how much the contextual distribution sharpens the parametric one), contextual peakedness (how confident the context is about a token), and a generalized divergence between the two distributions. When these signals indicate a genuine conflict, the decoding distribution is shifted toward the context; otherwise the parametric memory is kept. The paper reports that this method outperforms strong adaptive baselines such as AdaCAD, with up to a 9.2-point average improvement in","pith_inferences":["The same confidence statistics could serve as a gating signal in retrieve-then-read pipelines to decide which passages to attend to, or to flag tokens that should be double-checked by an external validator; the paper does not test these downstream uses.","If the statistics capture conflict rather than mere uncertainty, they might also be used to calibrate abstention: when both parametric and contextual distributions are peaked on contradictory tokens, the system could refuse to answer; this is a natural extension not explored here.","The reported gains are averages over benchmarks; a per-token analysis of when CoCoA disagrees with AdaCAD would show whether the advantage comes from the confidence terms, the divergence term, or their interaction, and whether any single term is disposable."],"forward_implications":["QA accuracy improves by up to 9.2 average points over AdaCAD across multiple LLMs and QA benchmarks.","Factuality in summarization and long-form QA improves by up to 2.5 average points over AdaCAD on key benchmarks.","The method maintains strong performance in low-conflict settings, where contrastive decoding methods tuned for conflict typically degrade.","CoCoA's sensitivity to conflict variations is superior to strong baselines, meaning it better aligns its behavior with the actual presence of conflict.","Because the method works at token level and requires no fine-tuning, it can be dropped into existing decoding pipelines."],"supporting_citations":[],"fun_headline_variants":["CoCoA: +9.2 QA over AdaCAD","CoCoA decoding boosts QA by up to 9.2 points","Adaptive decoder CoCoA beats AdaCAD by 9.2 QA","CoCoA: context-aware decoding resolves conflicts, +9.2","Entropy-aware decoding: CoCoA gains 9.2 QA over AdaCAD"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The approach assumes that the entropy gap, contextual peakedness, and generalized divergence computed from next-token distributions reliably separate genuine knowledge conflicts from ordinary uncertainty; if these numbers look the same for both, the adaptive rule cannot know when to override memory, and the reported gains would not generalize beyond the tested benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["CoCoA: +9.2 QA over AdaCAD","CoCoA decoding boosts QA by up to 9.2 points","Adaptive decoder CoCoA beats AdaCAD by 9.2 QA","CoCoA: context-aware decoding resolves conflicts, +9.2","Entropy-aware decoding: CoCoA gains 9.2 QA over AdaCAD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001147,"raw_usage":{"total_tokens":4595,"prompt_tokens":743,"completion_tokens":3852,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":3768}},"tokens_in":487,"tokens_out":3852,"duration_ms":31540,"temperature":1.0,"reasoning_tokens":3768,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:46:58.476026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CoCoA on a dataset where contexts are deliberately perturbed to contradict the model when the model is right, and check whether its token-level choices track an oracle that knows which context tokens are actually true; if CoCoA's accuracy on non-conflicting tokens drops below a static context-preference decoding (or below plain parametric decoding) while its conflict statistics are indistinguishable from ordinary uncertainty, the central claim would be false.","supporting_citations":[],"review_version":1}