REVIEW 5 major objections 6 minor 19 references
HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models
T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read HAVE is a parameter-free, single-forward-pass decoding intervention that reduces hallucinations in LLMs by reweighting attention heads per input and calibrating attention weights with value-vector magnitudes.
desk verdict A plausible decoding-time extension of DAGCD with two genuinely new modules, but the paper as written has a normalization error, an unresolved equation conflict, and borrowed baseline numbers, so the headline gains are not yet supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the write-back proxy $r_{\ell,h}(j)=\tilde a_{\ell,h}(j)\,\Vert V_{\ell,h}(j)\Vert_2$: the attention weight after sink masking multiplied by the Euclidean norm of the value vector, which the paper treats as each token's contribution to the residual-stream update. The proxy is aggregated across heads with adaptive weights from a softmax over per-head context-sensitivity scores, refined by a multiplicative estimator mask $m(j)=\sigma(w^\top f(j)+b)$, and fused with the LM distribution by $S_t=P_t+\alpha\,H_{\mathrm{norm}}(P_t)\,U_t$. This proxy carries the argument: it converts raw attention into a contribution-aware, vocabulary-aligned evidence distribution without adding parameters.
What would settle it
Measure the true effect of each context token by zeroing its value-vector contribution in a forward pass and comparing next-token probabilities; if the rank correlation between measured effects and HAVE's evidence scores is near zero, the core proxy fails. Independently, refit the estimator mask on a held-out split and check whether the reported EM and F1 gains persist; if they collapse, the headline results depend on fitted parameters rather than on the value-calibration proxy.
Extended reading notes
Core claim
HAVE claims that faithful decoding can be built from two internal signals already available in a single forward pass: instance-level head weights $w_{\ell,h}$ from a softmax over context-sensitivity scores, and token evidence $r_{\ell,h}(j)=\tilde a_{\ell,h}(j)\,\Vert V_{\ell,h}(j)\Vert_2$ normalized per head, with sink tokens masked. The resulting utilization distribution $U_t$ is projected onto the vocabulary, restricted to the Top-$R$ candidates, and added to the language-model distribution with coefficient $\alpha\,H_{\mathrm{norm}}(P_t)$, so evidence has no effect outside the support and is monotone within it. The paper reports consistent Exact Match and F1 gains over greedy, CAD, COIECD, and DAGCD on HotpotQA, SearchQA, SQuAD, Natural Questions, and NQ-Swap across LLaMA2-7B, LLaMA2-13B, and Mistral-7B, with ablations showing both modules contribute and are complementary.
Load-bearing premise
The method rests on the assumption that a token's attention weight times the size of its value vector, together with an estimator mask whose parameters are taken from an earlier method, correctly measures how much that token actually changes the model's next-word prediction.
Editorial extensions
If this is right
- Any retrieval-augmented system that can read internal attention and value activations could improve factual grounding by adding a parameter-free decoding layer, without changing the underlying model.
- Because the fusion is uncertainty-scaled, confident generations are barely altered, so the intervention should not degrade fluent or deterministic outputs.
- The ablation result implies the two modules target different failure modes; using either one alone would forfeit part of the hallucination reduction.
- Compatibility with grouped-query attention and sliding-window KV caches means the method could carry over to long-context and streaming settings where full attention matrices are unavailable.
Reading between the lines
- The paper leaves implicit that the value-norm proxy is only one possible write-back estimator; a direct test would compare $\Vert V(j)\Vert_2$ against the measured change in the residual stream when a token's contribution is removed.
- Because the estimator mask's parameters are inherited from DAGCD without a stated fitting procedure, the cleanest validation is to refit those parameters on a disjoint set and confirm the gains transfer; if they do not, the reported improvement may be tied to the specific benchmark setup.
- HAVE's uncertainty-weighted additive fusion resembles contrastive decoding, so the marginal contribution of value calibration could be isolated by running HAVE with and without the attention-times-value feature while keeping the fusion policy fixed.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a decoding-time method, HAVE, to mitigate hallucinations in retrieval-augmented/long-context generation. It computes instance-level head weights via a softmax over context-sensitivity scores, calibrates token evidence by multiplying attention weights by value-vector norms and masking sink tokens, and fuses the resulting utilization distribution with the LM distribution through an uncertainty-scaled additive residual. The authors claim this is a parameter-free, single-forward-pass intervention that consistently outperforms strong baselines and achieves state-of-the-art results on HotpotQA, SQuAD, and NQ. The manuscript reports experiments on LLaMA2-7B/13B-Chat and Mistral-7B-Instruct over five QA datasets, plus ablations and sensitivity analyses.
Significance. HAVE addresses a real problem and the design is attractive in principle: value-vector norms are a more plausible proxy for residual-stream contribution than raw attention weights, and instance-level head gating is more flexible than static head weighting. The paper's strengths are the clear problem framing, the modular framework, and the inclusion of ablations. However, the central empirical claim is not supported by the manuscript as written: the comparative evaluation is not controlled (baseline numbers are taken from DAGCD), the method definition is ambiguous (conflicting definitions of U_ctx and undisclosed estimator parameters), and the fusion rule in Eq. (16) does not produce a normalized distribution. Because the paper's contribution is primarily empirical, these issues are decisive. Should the authors rerun all baselines under identical conditions and fix the formal and reproducibility issues, the method could be worth revisiting.
major comments (5)
- [Section 4.1 and Table 1] Section 4.1 states "All baselines share the same prompts and retrieval/context construction," but the Table 1 caption states "Baseline numbers are taken from DAGCD [2]." These statements are incompatible. If the baseline rows were copied from DAGCD's published runs, then the HAVE-minus-DAGCD margins in Table 1 are not measured under the same prompts, context construction, or dataset subsets, and the claimed consistent superiority on HotpotQA, SQuAD, and NQ is not established. Please provide a fully self-contained evaluation in which all baselines are run under identical conditions.
- [Section 3.5, Eq. (16)] Equation (16) defines S_t = P_t + alpha * H_norm(P_t) * U_t. Since P_t sums to 1 and U_t is normalized over R_t (Eq. 15), S_t sums to 1 + alpha * H_norm(P_t) and is not a probability distribution. No renormalization step is described after Eq. (16). If a renormalization is intended, the support-preservation property in Eq. (4) needs to be re-derived, because outside R_t the probabilities would change by the normalization constant. As written, the fusion policy is not a valid decoding distribution.
- [Section 3.4, Eqs. (12) and (13)] Equations (12) and (13) give conflicting definitions of the same quantity U_ctx_t(j): Eq. (12) multiplies by the estimator mask m(j), while Eq. (13) omits m(j). The paper never states which variant was used in the experiments. Moreover, the feature vector f(j) used in m(j) is not defined, and the estimator parameters w and b are neither reported nor shown to remain valid after switching the input features from DAGCD's pure-attention features to Attn times V features. Without this information, the evaluated method is ambiguous and not reproducible.
- [Abstract, Section 4.2, and Table 1] Table 1 shows that HAVE is systematically worse than DAGCD on SearchQA for all three models (e.g., LLaMA2-7B-Chat EM 53.83 vs 54.25; Mistral-7B-Instruct EM 44.85 vs 47.17), yet the abstract and Section 5 claim that HAVE "consistently reduces hallucinations and outperforms strong baselines, including DAGCD." This contradiction, together with the uncontrolled baseline comparison, means the paper's headline claim is not supported by the reported data.
- [Abstract, Section 3, and Section 4.1] The abstract and introduction describe HAVE as "parameter-free," but the method depends on the fusion scale alpha, the support size R, the floor eta, the base head priors w^(base), the sink mask M, and the estimator parameters (w,b). Even if the latter are reused from DAGCD, they are parameters that must be specified for reproducibility. The manuscript does not report their values or how they were selected, which weakens the "parameter-free" claim and the generality of the results.
minor comments (6)
- [Section 3.1, Eq. (1)] Equation (1) contains garbled glyphs ("Pt =∼⋊℧≈⋗⅁↶(z t)") and should be typeset as a standard conditional distribution, e.g., P_t = f(z_t | X).
- [Section 3.4, Eq. (12)] The vector f(j) in Eq. (12) is never defined; please specify its construction, dimension, and normalization.
- [Section 4.1] Section 4.1 says hyperparameters follow the released DAGCD configuration but does not list the actual values of alpha, R, and eta; this prevents reproduction of Figure 3's sensitivity analysis.
- [Figures 2 and 3] Figures 2 and 3 are referenced in the text but the plots are not visible in the manuscript; either include the figures or remove the references.
- [Section 3.4 and Section 3.5] The "robust fallback" described at the end of Section 3.4 and Section 3.5 is underspecified: it does not state what condition triggers the fallback or which "uniform-head Attn times V baseline" is used.
- [Throughout] The phrase "parameter-free" should be replaced by a precise statement such as "no additional trained parameters" throughout, given the existence of alpha, R, eta, and the estimator parameters.
Circularity Check
No significant circularity: HAVE's decoding intervention is constructive and self-contained; the paper's main weaknesses are reproducibility and evaluation mismatch, not circular derivation.
full rationale
The paper's central claim is empirical: a single-forward-pass, no-finetuning decoding intervention (Head-Adaptive Gating plus Value Calibration plus an uncertainty-scaled fusion policy) improves QA metrics over strong baselines. Walking the derivation chain in Eqs. 3-17, every component is defined from the model's own attention and value activations: head weights w_lh come from instance-level scores (Eqs. 6-8), token evidence comes from value-norm-augmented attention (Eqs. 9-11), and the final distribution is an additive residual on the LM distribution (Eqs. 16-17). No predicted quantity is defined in terms of the benchmark answer, no parameter is fitted to the evaluation data and then renamed a prediction, and no load-bearing claim rests on a uniqueness theorem or self-citation. The estimator m(j) in Eq. 12 is attributed to DAGCD [2], whose authors do not overlap with the present authors; although its parameters w,b are undisclosed and the feature switch to Attn x V is not justified, that is a reproducibility/validity gap, not a circular reduction. The redefinition of U_ctx_t(j) in Eqs. 12 and 13 is an internal inconsistency that makes the deployed method ambiguous, but it is not a self-referential derivation. Likewise, the Table 1 caption saying 'Baseline numbers are taken from DAGCD [2]' conflicts with Section 4.1's 'All baselines share the same prompts and retrieval/context construction'; this can undermine the controlled-comparison claim, but it does not make HAVE's derivation circular. Under the hard rule that circularity must be exhibited as an equation reducing to its own input or a fitted parameter renamed as prediction, no such step is present. Score 0.
Assumptions & free parameters
free parameters (6)
- alpha (fusion scale) =
not reported (uses DAGCD config)
- R (top-rank support) =
not reported (uses DAGCD config)
- eta (floor) =
not reported
- w_base (base head priors) =
not reported
- w, b (estimator parameters) =
not reported
- sink mask M =
binary mask on special and whitespace tokens
assumptions (5)
- ad hoc to paper Attention-head usefulness is adequately scored by the context-sensitivity score s_lh in Eq. 6.
- ad hoc to paper The value-vector norm ||V_lh(j)|| approximates a token's write-back contribution to the residual stream.
- domain assumption The lightweight estimator m(j) = sigma(w^T f(j)+b) from DAGCD remains valid when features are changed to Attn times V.
- ad hoc to paper Additive uncertainty-scaled fusion in Eq. 16 preserves a valid decoding distribution.
- domain assumption Sink tokens can be identified by a binary mask M(j).
Cite this review
Pith. "Pith review of HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models." pith.science (2026). https://pith.science/paper/32CPAHPX
@misc{pith2026250906596,
author = {Pith},
title = {Pith review of: HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/32CPAHPX}},
note = {Machine review of arXiv:2509.06596}
}
read the original abstract
Large Language Models (LLMs) often produce hallucinations in retrieval-augmented or long-context generation, even when relevant evidence is present. This stems from two issues: head importance is treated as input-agnostic, and raw attention weights poorly reflect each token's true contribution. We present HAVE (Head-Adaptive Gating and ValuE Calibration), a parameter-free decoding framework that directly addresses both challenges. HAVE introduces head-adaptive gating, which performs instance-level soft reweighing of attention heads, and value calibration, which augments attention with the magnitude of value vectors to approximate write-back contribution. Together, these modules construct token-level evidence aligned with model updates and fuse it with the LM distribution through a lightweight uncertainty-scaled policy. HAVE requires no finetuning and operates in a single forward pass, making it efficient and broadly applicable. Experiments across multiple QA benchmarks and LLM families demonstrate that HAVE consistently reduces hallucinations and outperforms strong baselines, including DAGCD, with modest overhead. The framework is transparent, reproducible, and readily integrates with off-the-shelf LLMs, advancing trustworthy generation in real-world settings.
Figures
Reference graph
Works this paper leans on
-
[2]
Yanwen Huang, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, and Jing Xiao. Dynamic attention-guided context decoding for mitigating context faithfulness hallucinations in large language models. page 5174–5193, 2025
work page 2025
-
[1]
Retrieval-augmented generation for knowledge- intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. InProceedings of the 34th International Conference on Neural Information Processing Systems, pages 9459–9474, 2020
work page 2020
-
[3]
Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023
arXiv 2023
-
[4]
Minda Hu, Bowei He, Yufei Wang, Liangyou Li, Chen Ma, and Irwin King. Mitigating large language model hallucination with faithful finetuning.arXiv preprint arXiv:2406.11267, 2024
arXiv 2024
-
[5]
Aligning large multimodal models with factually augmented rlhf
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics ACL 2024, pages 13088–13110, 2024
work page 2024
-
[6]
Loka Li, Zhenhao Chen, Guangyi Chen, Yixuan Zhang, Yusheng Su, Eric Xing, and Kun Zhang. Confidence matters: Revisiting intrinsic self-correction capabilities of large language models.arXiv preprint arXiv:2402.12563, 2024
arXiv 2024
-
[7]
Can we edit factual knowledge by in-context learning? InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4862–4876, 2023
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4862–4876, 2023
2023
-
[8]
Trusting your evidence: Hallucinate less with context-aware decoding
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 783–791, 2024
2024
Show all 19 references
-
[9]
Discerning and resolving knowledge conflicts through adaptive decoding with contextual information-entropy constraint
Xiaowei Yuan, Zhao Yang, Yequan Wang, Shengping Liu, Jun Zhao, and Kang Liu. Discerning and resolving knowledge conflicts through adaptive decoding with contextual information-entropy constraint. InFindings of the Association for Computational Linguistics ACL 2024, pages 3903–...
2024
-
[10]
Genaudit: Fixing factual errors in language model outputs with evidence.arXiv preprint arXiv:2402.12566, 2024
Kundan Krishna, Sanjana Ramprasad, Prakhar Gupta, Byron C Wallace, Zachary C Lipton, and Jeffrey P Bigham. Genaudit: Fixing factual errors in language model outputs with evidence.arXiv preprint arXiv:2402.12566, 2024. 8 Running Title for Header
2024 arXiv
-
[11]
Converge to the truth: factual error correction via iterative constrained editing
Jiangjie Chen, Rui Xu, Wenxuan Zeng, Changzhi Sun, Lei Li, and Yanghua Xiao. Converge to the truth: factual error correction via iterative constrained editing. InProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovat...
2023
-
[12]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023
2023 arXiv
-
[13]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...
2023
-
[14]
Hotpotqa: A dataset for diverse, explainable multi-hop question answering
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Process...
2018
-
[15]
Searchqa: A new q&a dataset augmented with context from a search engine.arXiv preprint arXiv:1704.05179, 2017
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, V olkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine.arXiv preprint arXiv:1704.05179, 2017
2017 arXiv
-
[16]
Squad: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016
2016
-
[17]
Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:453–466, 2019
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research.Transactions of the Association for Computat...
2019
-
[18]
Entity-based knowledge conflicts in question answering
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, 2021
2021
-
[19]
Beyond single-turn: A survey on multi-turn interactions with large language models.arXiv preprint arXiv:2504.04717, 2025
Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. Beyond single-turn: A survey on multi-turn interactions with large language models.arXiv preprint arXiv:2504.04717, 2025. 9
2025 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.