{"id":"ca83852f-0516-4aed-beb8-96d8ae7b05cd","arxiv_id":"2509.06596","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"HAVE uses head-adaptive gating and value calibration to build token evidence and fuse it with the LM distribution, yielding modest QA gains over DAGCD.","lead":"This paper proposes HAVE, a decoding-time framework that reweights attention heads and combines attention with value vector magnitudes to reduce hallucination in large language models. It reports small accuracy gains over existing decoding methods on several question-answering benchmarks, but the method as written is not fully reproducible.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are not reproducible from the paper: baseline rows are taken from DAGCD rather than measured in the same runs, and Eqs. 12-13 plus undisclosed w,b leave the evaluated method ambiguous.","rationale":"I read the paper in good faith: the idea of using value-vector norms to calibrate attention is plausible, and a single-pass, no-finetuning decoding intervention is a worthwhile goal. The load-bearing requirement for the central claim is a valid comparison. The paper itself undercuts that requirement by saying baselines share prompts and context while the table note says baseline numbers were copied from DAGCD. A copied baseline row is only usable if the evaluation is bit-identical to the original DAGCD runs, and the paper does not establish that; it only asserts it. The method's own definition is ambiguous because Eqs. 12 and 13 define U_ctx differently, and the estimator parameters are undisclosed, so even a reader with the base models cannot reproduce HAVE. These are not disagreements with consensus; they are internal-consistency and reproducibility problems. The reader's weakest assumption about the value-norm proxy and mismatched estimator features is real and related, but I would weight the comparison provenance and definitional ambiguity first, because even a perfect proxy would not save the conclusion if the numbers being compared were produced under different conditions. Therefore the verdict should remain a rejection, pending either released code with exact parameters or a fully self-contained rerun of all baselines and HAVE under identical conditions.","tokens_in":8232,"tokens_out":5913,"duration_ms":53778,"concrete_test":"Run a single controlled reproduction on one model and dataset, e.g., LLaMA2-7B-Chat on HotpotQA: implement HAVE exactly once using a single definition of U_ctx (state whether Eq. 12's m(j) is applied), obtain or reconstruct w,b, and run Greedy, CAD, COIECD, DAGCD, and HAVE on the same 1,000-question subset with identical prompts, context construction, seeds, and decoding parameters. Compare the reproduced DAGCD EM/F1 with the Table 1 values (55.31/68.61) and the reproduced HAVE values with 55.65/68.81. If DAGCD's numbers shift by more than rounding error, or if the HAVE margin disappears, the state-of-the-art claim is an artifact of mismatched evaluation and should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: HAVE outperforms strong baselines and sets new state of the art on HotpotQA, SQuAD, and NQ. For that claim to hold, the numbers in Table 1 must come from a controlled comparison. They do not, as written. Section 4.1 says \"All baselines share the same prompts and retrieval/context construction,\" but the Table 1 caption says \"Baseline numbers are taken from DAGCD [2].\" These two statements cannot both be true unless the baseline rows were copied from another paper's runs. If DAGCD's numbers were produced under different prompts, context construction, or dataset subsets, every HAVE-minus-DAGCD margin in Table 1 is contaminated by evaluation mismatch, and the \"consistently outperforms\" conclusion is unsupported. The ambiguity is compounded by the method definition: Eq. 12 sets U_ctx_t(j)=m(j)·Σ w ̂r, while Eq. 13 redefines the same symbol as Σ w ̂r with no m; the paper never says which variant was deployed. The estimator's w,b and base head priors w^(base) are also not reported, and the estimator is claimed to be reused from DAGCD after changing its input features to Attn×V, with no evidence that the old parameters are appropriate for the new features. Any one of these gaps would make the headline result hard to verify; together they make the central claim currently unsubstantiated. The concern is not that the method cannot work, but that the paper as written does not permit a reader to distinguish a real decoding improvement from a mismatch in how the numbers were produced.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a decoding-time method, HAVE, to mitigate hallucinations in retrieval-augmented/long-context generation. It computes instance-level head weights via a softmax over context-sensitivity scores, calibrates token evidence by multiplying attention weights by value-vector norms and masking sink tokens, and fuses the resulting utilization distribution with the LM distribution through an uncertainty-scaled additive residual. The authors claim this is a parameter-free, single-forward-pass intervention that consistently outperforms strong baselines and achieves state-of-the-art results on HotpotQA, SQuAD, and NQ. The manuscript reports experiments on LLaMA2-7B/13B-Chat and Mistral-7B-Instruct over five QA datasets, plus ablations and sensitivity analyses.","tokens_in":8675,"tokens_out":7296,"duration_ms":58890,"significance":"HAVE addresses a real problem and the design is attractive in principle: value-vector norms are a more plausible proxy for residual-stream contribution than raw attention weights, and instance-level head gating is more flexible than static head weighting. The paper's strengths are the clear problem framing, the modular framework, and the inclusion of ablations. However, the central empirical claim is not supported by the manuscript as written: the comparative evaluation is not controlled (baseline numbers are taken from DAGCD), the method definition is ambiguous (conflicting definitions of U_ctx and undisclosed estimator parameters), and the fusion rule in Eq. (16) does not produce a normalized distribution. Because the paper's contribution is primarily empirical, these issues are decisive. Should the authors rerun all baselines under identical conditions and fix the formal and reproducibility issues, the method could be worth revisiting.","major_comments":[{"comment":"Section 4.1 states \"All baselines share the same prompts and retrieval/context construction,\" but the Table 1 caption states \"Baseline numbers are taken from DAGCD [2].\" These statements are incompatible. If the baseline rows were copied from DAGCD's published runs, then the HAVE-minus-DAGCD margins in Table 1 are not measured under the same prompts, context construction, or dataset subsets, and the claimed consistent superiority on HotpotQA, SQuAD, and NQ is not established. Please provide a fully self-contained evaluation in which all baselines are run under identical conditions.","section":"Section 4.1 and Table 1"},{"comment":"Equation (16) defines S_t = P_t + alpha * H_norm(P_t) * U_t. Since P_t sums to 1 and U_t is normalized over R_t (Eq. 15), S_t sums to 1 + alpha * H_norm(P_t) and is not a probability distribution. No renormalization step is described after Eq. (16). If a renormalization is intended, the support-preservation property in Eq. (4) needs to be re-derived, because outside R_t the probabilities would change by the normalization constant. As written, the fusion policy is not a valid decoding distribution.","section":"Section 3.5, Eq. (16)"},{"comment":"Equations (12) and (13) give conflicting definitions of the same quantity U_ctx_t(j): Eq. (12) multiplies by the estimator mask m(j), while Eq. (13) omits m(j). The paper never states which variant was used in the experiments. Moreover, the feature vector f(j) used in m(j) is not defined, and the estimator parameters w and b are neither reported nor shown to remain valid after switching the input features from DAGCD's pure-attention features to Attn times V features. Without this information, the evaluated method is ambiguous and not reproducible.","section":"Section 3.4, Eqs. (12) and (13)"},{"comment":"Table 1 shows that HAVE is systematically worse than DAGCD on SearchQA for all three models (e.g., LLaMA2-7B-Chat EM 53.83 vs 54.25; Mistral-7B-Instruct EM 44.85 vs 47.17), yet the abstract and Section 5 claim that HAVE \"consistently reduces hallucinations and outperforms strong baselines, including DAGCD.\" This contradiction, together with the uncontrolled baseline comparison, means the paper's headline claim is not supported by the reported data.","section":"Abstract, Section 4.2, and Table 1"},{"comment":"The abstract and introduction describe HAVE as \"parameter-free,\" but the method depends on the fusion scale alpha, the support size R, the floor eta, the base head priors w^(base), the sink mask M, and the estimator parameters (w,b). Even if the latter are reused from DAGCD, they are parameters that must be specified for reproducibility. The manuscript does not report their values or how they were selected, which weakens the \"parameter-free\" claim and the generality of the results.","section":"Abstract, Section 3, and Section 4.1"}],"minor_comments":[{"comment":"Equation (1) contains garbled glyphs (\"Pt =∼⋊℧≈⋗⅁↶(z t)\") and should be typeset as a standard conditional distribution, e.g., P_t = f(z_t | X).","section":"Section 3.1, Eq. (1)"},{"comment":"The vector f(j) in Eq. (12) is never defined; please specify its construction, dimension, and normalization.","section":"Section 3.4, Eq. (12)"},{"comment":"Section 4.1 says hyperparameters follow the released DAGCD configuration but does not list the actual values of alpha, R, and eta; this prevents reproduction of Figure 3's sensitivity analysis.","section":"Section 4.1"},{"comment":"Figures 2 and 3 are referenced in the text but the plots are not visible in the manuscript; either include the figures or remove the references.","section":"Figures 2 and 3"},{"comment":"The \"robust fallback\" described at the end of Section 3.4 and Section 3.5 is underspecified: it does not state what condition triggers the fallback or which \"uniform-head Attn times V baseline\" is used.","section":"Section 3.4 and Section 3.5"},{"comment":"The phrase \"parameter-free\" should be replaced by a precise statement such as \"no additional trained parameters\" throughout, given the existence of alpha, R, eta, and the estimator parameters.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"To the editor: The manuscript's reliance on DAGCD's published baseline numbers and its undisclosed estimator parameters make it impossible to verify whether the reported gains reflect a controlled comparison or an evaluation mismatch. In addition, the method's family resemblance to DAGCD (with a modified feature set) means the novelty contribution should be carefully assessed against [2]. If the authors can supply a self-contained re-run of all baselines and a complete method specification, the work could be reconsidered; as it stands, the central claim is not verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Xin, quick take on arXiv:2509.06596. The idea is decent and the implementation has real content: head-adaptive gating (instance-level soft head reweighting) and value calibration (Attn×||V|| with sink masking) are legit extensions of DAGCD, not just re-packaging. No fine-tuning, single forward pass, and the ablations show both modules contribute. If the numbers hold, it is a useful incremental step for hallucination mitigation in RAG.\n\nBut the paper as written does not support the numbers. Three concrete problems. First, Eq. 16 defines S_t = P_t + α·H_norm(P_t)·U_t. Since P_t and U_t are each normalized over the vocabulary, S_t sums to 1 + α·H_norm, not 1. The paper calls S_t a distribution but never renormalizes. For greedy decoding the ranking may survive, but as a probability model it is broken, and the paper never says which is used.\n\nSecond, Eqs. 12 and 13 both define U_ctx_t(j) differently: Eq. 12 multiplies by the estimator mask m(j), Eq. 13 drops it. The method actually deployed is ambiguous. That is load-bearing, because the estimator is claimed to reuse DAGCD's parameters after changing features from attention to Attn×V. Since w,b are not reported, and no evidence is given that old parameters transfer to new features, the configuration is not reproducible.\n\nThird, the baseline comparison. Section 4.1 says all baselines share the same prompts and retrieval/context construction; Table 1's caption says baseline numbers are taken from DAGCD. Those two statements cannot both be true unless the runs are identical copies. If the baseline rows come from DAGCD's paper, then every HAVE-minus-DAGCD margin is an apples-to-oranges comparison unless you control for exact prompts, decoding settings, and dataset splits. The paper acknowledges following DAGCD's configuration, which mitigates it, but does not verify that DAGCD's published numbers were produced under exactly those conditions. No code, no baselines rerun, no parameter disclosure.\n\nIn short: the core idea is plausible, but the current write-up cannot be reproduced from the information given. I would not trust the SOTA claims until the authors fix the normalization, disambiguate Eq. 12/13, release w,b or train their own estimator, and either rerun the baselines or explicitly justify the transfer. That is a serious revision, not a desk reject: the method is worth a careful look once the evidence is solid.\n\nRecommendation: send to peer review, with the clear expectation that these issues be addressed. I'd bring it to reading group as a case study in how not to report baseline comparisons.","headline":"A plausible decoding-time extension of DAGCD with two genuinely new modules, but the paper as written has a normalization error, an unresolved equation conflict, and borrowed baseline numbers, so the headline gains are not yet supported.","tokens_in":9138,"tokens_out":2880,"would_cite":false,"duration_ms":25238,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HAVE is a parameter-free, single-forward-pass decoding intervention that reduces hallucinations in LLMs by reweighting attention heads per input and calibrating attention weights with value-vector magnitudes.","keywords":["hallucination mitigation","decoding-time intervention","attention head gating","value calibration","retrieval-augmented generation","factual grounding","large language models"],"falsifier":"Measure the true effect of each context token by zeroing its value-vector contribution in a forward pass and comparing next-token probabilities; if the rank correlation between measured effects and HAVE's evidence scores is near zero, the core proxy fails. Independently, refit the estimator mask on a held-out split and check whether the reported EM and F1 gains persist; if they collapse, the headline results depend on fitted parameters rather than on the value-calibration proxy.","tokens_in":8040,"feed_emoji":"🎯","tokens_out":7549,"duration_ms":59889,"temperature":0.7,"pith_summary":"The paper tries to establish that hallucinations in retrieval-augmented generation can be reduced without retraining by fixing how attention is read: treat head importance as input-dependent and replace raw attention weights with a value-magnitude-weighted estimate of each token's write-back contribution. It claims this two-step correction, fused into the LM distribution through an uncertainty-scaled additive rule, outperforms strong decoding baselines including DAGCD across five QA benchmarks and three open LLMs, with new state-of-the-art Exact Match and F1 scores on HotpotQA, SQuAD, and NQ. If true, a lightweight, interpretable decoding layer can make off-the-shelf LLMs more factual at modest computational cost, with no finetuning and no extra forward passes.","feed_headline":"Parameter-free decoder cuts LLM hallucinations on QA benchmarks","feed_subtitle":"One forward pass, no retraining: head reweighting plus value calibration trims wrong answers on three QA benchmarks.","key_machinery":"The load-bearing object is the write-back proxy $r_{\\ell,h}(j)=\\tilde a_{\\ell,h}(j)\\,\\Vert V_{\\ell,h}(j)\\Vert_2$: the attention weight after sink masking multiplied by the Euclidean norm of the value vector, which the paper treats as each token's contribution to the residual-stream update. The proxy is aggregated across heads with adaptive weights from a softmax over per-head context-sensitivity scores, refined by a multiplicative estimator mask $m(j)=\\sigma(w^\\top f(j)+b)$, and fused with the LM distribution by $S_t=P_t+\\alpha\\,H_{\\mathrm{norm}}(P_t)\\,U_t$. This proxy carries the argument: it converts raw attention into a contribution-aware, vocabulary-aligned evidence distribution without adding parameters.","core_discovery":"HAVE claims that faithful decoding can be built from two internal signals already available in a single forward pass: instance-level head weights $w_{\\ell,h}$ from a softmax over context-sensitivity scores, and token evidence $r_{\\ell,h}(j)=\\tilde a_{\\ell,h}(j)\\,\\Vert V_{\\ell,h}(j)\\Vert_2$ normalized per head, with sink tokens masked. The resulting utilization distribution $U_t$ is projected onto the vocabulary, restricted to the Top-$R$ candidates, and added to the language-model distribution with coefficient $\\alpha\\,H_{\\mathrm{norm}}(P_t)$, so evidence has no effect outside the support and is monotone within it. The paper reports consistent Exact Match and F1 gains over greedy, CAD, COIECD, and DAGCD on HotpotQA, SearchQA, SQuAD, Natural Questions, and NQ-Swap across LLaMA2-7B, LLaMA2-13B, and Mistral-7B, with ablations showing both modules contribute and are complementary.","pith_inferences":["The paper leaves implicit that the value-norm proxy is only one possible write-back estimator; a direct test would compare $\\Vert V(j)\\Vert_2$ against the measured change in the residual stream when a token's contribution is removed.","Because the estimator mask's parameters are inherited from DAGCD without a stated fitting procedure, the cleanest validation is to refit those parameters on a disjoint set and confirm the gains transfer; if they do not, the reported improvement may be tied to the specific benchmark setup.","HAVE's uncertainty-weighted additive fusion resembles contrastive decoding, so the marginal contribution of value calibration could be isolated by running HAVE with and without the attention-times-value feature while keeping the fusion policy fixed."],"forward_implications":["Any retrieval-augmented system that can read internal attention and value activations could improve factual grounding by adding a parameter-free decoding layer, without changing the underlying model.","Because the fusion is uncertainty-scaled, confident generations are barely altered, so the intervention should not degrade fluent or deterministic outputs.","The ablation result implies the two modules target different failure modes; using either one alone would forfeit part of the hallucination reduction.","Compatibility with grouped-query attention and sliding-window KV caches means the method could carry over to long-context and streaming settings where full attention matrices are unavailable."],"supporting_citations":[{"why":"Supplies the DAGCD baseline that HAVE must beat, the estimator-mask design that HAVE reuses, and the hyperparameter configuration for fair comparison.","marker":"[2]"},{"why":"CAD is a contrastive decoding baseline whose context-aware versus context-free contrast HAVE is compared against.","marker":"[8]"},{"why":"COIECD is a conflict-aware decoding baseline evaluated on the same QA benchmarks.","marker":"[9]"},{"why":"Defines LLaMA2-7B-Chat and LLaMA2-13B-Chat, two of the off-the-shelf models on which HAVE's gains are claimed.","marker":"[12]"},{"why":"Defines Mistral-7B-Instruct, the third base model family used in the experiments.","marker":"[13]"},{"why":"HotpotQA is the multi-hop reasoning benchmark where HAVE claims the largest gains and a new state of the art.","marker":"[14]"},{"why":"SQuAD provides the single-paragraph reading-comprehension setting where HAVE claims state-of-the-art EM and F1.","marker":"[16]"},{"why":"Natural Questions is the document-level QA benchmark where HAVE claims a new state of the art.","marker":"[17]"},{"why":"NQ-Swap is the synthetic conflict stress test where HAVE shows robustness under interleaved distractor passages.","marker":"[18]"}],"fun_headline_variants":["Training-free decoder with adaptive head gating cuts LLM hallucinations","One-pass head reweighting and value calibration reduce wrong answers","No retraining: adaptive head and value signals slash hallucinations","Single forward pass: adaptive head weights and value scaling reduce hallucinations","Head-adaptive gating plus value calibration cut QA hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that a token's attention weight times the size of its value vector, together with an estimator mask whose parameters are taken from an earlier method, correctly measures how much that token actually changes the model's next-word prediction.","fun_headline_variants_meta":{"raw":{"variants":["Training-free decoder with adaptive head gating cuts LLM hallucinations","One-pass head reweighting and value calibration reduce wrong answers","No retraining: adaptive head and value signals slash hallucinations","Single forward pass: adaptive head weights and value scaling reduce hallucinations","Head-adaptive gating plus value calibration cut QA hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001858,"raw_usage":{"total_tokens":7308,"prompt_tokens":970,"completion_tokens":6338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":6254}},"tokens_in":586,"tokens_out":6338,"duration_ms":39235,"temperature":1.0,"reasoning_tokens":6254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:15:09.236529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the true effect of each context token by zeroing its value-vector contribution in a forward pass and comparing next-token probabilities; if the rank correlation between measured effects and HAVE's evidence scores is near zero, the core proxy fails. Independently, refit the estimator mask on a held-out split and check whether the reported EM and F1 gains persist; if they collapse, the headline results depend on fitted parameters rather than on the value-calibration proxy.","supporting_citations":[{"cited_title":"Dynamic attention-guided context decoding for mitigating context faithfulness hallucinations in large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the DAGCD baseline that HAVE must beat, the estimator-mask design that HAVE reuses, and the hyperparameter configuration for fair comparison."},{"cited_title":"Discerning and resolving knowledge conflicts through adaptive decoding with contextual information-entropy constraint","cited_arxiv_id":null,"evidence_quote":"COIECD is a conflict-aware decoding baseline evaluated on the same QA benchmarks."},{"cited_title":"Entity-based knowledge conflicts in question answering","cited_arxiv_id":null,"evidence_quote":"NQ-Swap is the synthetic conflict stress test where HAVE shows robustness under interleaved distractor passages."}],"review_version":2}