{"id":"2c9c6bfc-e68b-43ee-8914-e3452c757e87","arxiv_id":"2502.05825","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Delta improves context-based question answering by subtracting the word scores of a randomly masked version of the prompt from the word scores of the original prompt during decoding.","lead":"This paper describes Delta, a decoding-time method that masks random parts of a prompt and subtracts the masked model's word scores from the original model's word scores to reduce hallucinations. On four question-answering benchmarks it reports gains of a few percentage points, but it is a close variant of existing contrastive decoding methods and is evaluated without code or error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Masked-prompt logits are not shown to be a hallucination prior; SQuAD v2 gains are dominated by abstention, so token-level dehallucination is unproven.","rationale":"The paper's own Table 1 is the best internal evidence, and it creates the concern: the largest reported gain (14.53pt no-answer EM on SQuAD v2 w/o sampling) is paired with a decline in has-answer EM. A method that cancels hallucinated tokens at the token level should improve answerable questions too; instead, the visible effect is a shift toward 'no answer.' The motivating example in §3.2 is illustrative but not a measurement, and Eq. 3's premise is asserted. The missing CAD baseline matters because CAD (§2) uses the same equation with an unconditional prior; without it, the improvement over vanilla decoding cannot be attributed to the masking mechanism. The paper deserves credit for reporting a concrete method, a fixed parameter set, and a negative result on CommonsenseQA/MMLU, which is a useful boundary condition. But the current evidence does not establish the mechanism, and the conditional verdict should remain unless the diagnostic check confirms the masked logits behave as a hallucination prior. The concern is not that the numbers are fabricated; it is that the interpretation is underdetermined.","tokens_in":10147,"tokens_out":7783,"duration_ms":83208,"concrete_test":"Run the SQuAD v2 and TriviaQA evaluations with the same rmask=0.7 but two restricted masking policies: (i) mask only passage/context tokens, leaving the question untouched; (ii) mask only question tokens, leaving the context untouched. If the no-answer EM and answerable EM gains persist under (i), the proposed masking-prior mechanism is plausible; if they require (ii) or uniform masking, the effect is driven by question corruption, not by a hallucination prior. Report the comparison across at least 10 random masking seeds with mean and standard error so stochasticity is controlled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on Eq. 3: subtracting logitθ(y|mask(z)) from the original logits cancels hallucinated content. That requires Pθ(y|mask(z)) to be a faithful 'hallucination prior,' but the paper never measures it. With rmask=0.7 (§3.3, §4.2), masking is applied uniformly to the whole prompt, including the question tokens, so mask(z) is often a badly corrupted prompt rather than a mildly fuzzified context. The subtracted logits may then be dominated by generic language priors or by the MASK/eos token, suppressing correct context-dependent tokens rather than selectively eliminating hallucinations. The data are consistent with this: on SQuAD v2 w/o sampling, the 14.53pt no-answer EM gain is accompanied by a 1.60pt drop in has-answer EM, suggesting the method mainly shifts the model toward abstention. Without masking-seed error bars, a CAD baseline (the same contrast using unconditional logits, cited in §2), or a token-level diagnostic showing masked logits are elevated specifically on hallucinated answers, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Delta, an inference-time contrastive decoding method that randomly masks a portion of the input prompt and subtracts the masked-prompt logits from the original logits to suppress hallucinated tokens. The method is defined in Eq. 5, with an adaptive plausibility constraint. The authors evaluate Delta on Llama-3.1-8B-Instruct (4-bit) across SQuAD v1.1/v2, TriviaQA, Natural Questions, CommonsenseQA, and MMLU, reporting exact-match gains of roughly 3 points on SQuAD v1.1, 6 points on SQuAD v2, 7 points on TriviaQA, and 2 points on Natural Questions under sampling, with a particularly large gain on SQuAD v2 no-answer exact match. The paper also includes an ablation over the masking ratio and alpha on SQuAD v1.1.","tokens_in":10401,"tokens_out":2621,"duration_ms":28196,"significance":"If the central claim holds, Delta would be a simple, training-free, inference-only method for reducing hallucinations in context-rich QA, with potentially broad applicability because it does not require external models or fine-tuning. The paper is honest about the method's limitation on context-free benchmarks, and the ablation shows robustness to hyperparameters on one dataset. However, the core premise—that masked-prompt logits are a faithful proxy for the model's hallucination prior—is asserted rather than demonstrated, and the headline improvements on SQuAD v2 are driven heavily by increased abstention on unanswerable questions. The experiments appear to be single-run without reported variance, which is a serious concern given the stochastic masking procedure. The contribution is potentially useful but requires substantially stronger evidence to be established.","major_comments":[{"comment":"The central claim that subtracting logit_theta(y | mask(z)) removes hallucinated content relies on the premise that mask(z) produces a distribution in which hallucinated tokens are specifically elevated. This premise is illustrated with the moldy banana example in §3.2 but is never directly measured. Without a token-level diagnostic showing that masked logits are elevated on hallucinated answers and not on correct context-dependent answers, Eq. (3) remains an unvalidated heuristic; the observed gains could also come from generic suppression of answer tokens or from the MASK/eos token dominating the subtracted distribution. Please provide such a diagnostic, or a controlled experiment varying the masking distribution and showing a monotonic relationship between masking-induced logit increases and actual hallucination rates.","section":"§3.2–§3.4, Eq. (3)"},{"comment":"On SQuAD v2 without sampling, the no-answer exact match improves by +14.53 points (23.63 to 38.17) while the has-answer exact match drops by 1.60 points (59.08 to 57.47). This pattern suggests that a large part of Delta's improvement is an increased tendency to abstain on unanswerable questions, rather than a token-level dehallucination that preserves correct answer extraction. The paper does not report a combined SQuAD v2 metric such as EM averaged over both answerable and unanswerable examples, nor does it analyze the HasAns/NoAns trade-off. Please report these joint metrics with error bars and discuss the mechanism explicitly.","section":"Table 1, §5.1"},{"comment":"All experiments appear to be run once with a single random masking seed (or no seed control). Because the masking indices are randomly drawn (Eq. 2 and §3.3), the reported differences of 1–3 percentage points on several benchmarks may be within the noise of the stochastic procedure. The paper should report means and standard deviations over multiple masking seeds, or at a minimum state that a fixed seed was used and justify that choice.","section":"§4.2, §5 (Tables 1–2)"},{"comment":"With rmask = 0.7, the masking is applied uniformly to the whole prompt, including the question tokens, so mask(z) is often a heavily corrupted input rather than a mildly fuzzified context. This makes it unclear whether the subtracted logits encode a 'hallucination prior' or simply a distribution conditioned on a partially unreadable prompt. The method should be tested with masking applied only to the context passage (leaving the question intact), and the effect of mask placement on the answer distribution should be analyzed. This directly bears on the validity of Eq. (3).","section":"§3.3, §4.2"},{"comment":"The related work cites context-aware decoding (CAD; Shi et al., 2024), which is conceptually the closest baseline: CAD contrasts logits with and without context. Since Delta is essentially a context-fuzzing variant of the same contrastive idea, the paper should include CAD as a direct baseline under identical conditions. Without this comparison, it is unclear whether Delta offers any improvement over simply contrasting against the no-context or weakly-context distribution, which would be a much simpler method.","section":"§2, §4.2, Table 1"}],"minor_comments":[{"comment":"There is a notation inconsistency between the definition of V_head(x_{<t}) in Eq. (4) and its use in Eq. (5) as V_head(z_{<t}). The sentence in §3.6 saying 'If z does not belong to V_head(x_{<i})' mixes brackets and subscripts; please unify.","section":"§3.5, Eq. (4) and (5)"},{"comment":"The sentence 'All experiments utilize the end-of-sequence (eos) token as the MASK token' is surprising, because the EOS token is a normal vocabulary token with its own learned semantics. Please clarify what this means operationally and whether the model was explicitly trained with such a masking token.","section":"§4.2"},{"comment":"The ablation reports 'standard deviations of 0.66 for exact match and 0.21 for F1' over the heatmap cells. Since the heatmap varies hyperparameters and not random seeds, calling these standard deviations is misleading; they are ranges across settings, not estimates of run-to-run variability. Please rephrase.","section":"§6"},{"comment":"There are grammar and citation errors in the sentence beginning 'In addition, the work context-aware decoding (CAD) has Shi et al. (2024) demonstrated'; the citation should be parenthetical and the verb should be finite.","section":"§2"},{"comment":"The reference to Wettig et al. is incomplete: the title is cut off as 'Should you mask 15' and the page numbers are missing. Several other references also lack full bibliographic details; please check the reference list for completeness.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially interesting idea but the evidence as presented is not yet convincing. The strongest concern is that the SQuAD v2 gain is driven by abstention on NoAns questions, and the paper does not demonstrate that masked logits are a specific hallucination prior. The absence of any repeated-run variability or CAD baseline makes the empirical claims hard to evaluate. I believe the central claim is defensible in principle, so I recommend major revision rather than rejection. I would also suggest the editors ask for code or a reproducibility appendix, since the method is simple and the experiments should be easy to replicate if the prompts and masking are specified precisely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Delta applies the visual-contrastive-decoding trick to text by randomly masking input tokens, then subtracting the masked-prompt logits from the original logits during generation. That is a legitimate idea and a natural extension of CAD, but the paper never compares against CAD, reports no variance for a stochastic masking procedure, and its headline SQuAD v2 gain is mostly abstention on no-answer questions. The core idea is worth taking seriously, but the evidence as written is conditional.\n\nWhat is genuinely new: the specific combination of random token masking plus contrastive decoding for text QA is not in prior work. The paper also does something right by testing on context-rich and context-free datasets, and the authors are honest that Delta hurts or does nothing on CommonsenseQA and MMLU. The ablation covering masking ratios and alpha values, with all settings above baseline on SQuAD v1.1, is a useful robustness check.\n\nThe soft spots are real but not fatal. First, Eq. 3 is structurally the same logit contrast as CAD with mask(z) in place of the no-context prompt, so without a CAD baseline you cannot tell whether Delta adds anything beyond existing context-aware decoding. Second, the paper reports single-run EM/F1 numbers despite sampling masks stochastically at rmask=0.7; a couple of random seeds could shift the numbers by more than the reported gains. Third, the SQuAD v2 improvement is concentrated in NoAns EM (up 14.53 w/o sampling) while HasAns EM drops 1.60, which means the method is largely shifting the model toward abstention rather than demonstrably dehallucinating token-level content. Fourth, the central premise—that masked prompt logits are a faithful hallucination prior—is asserted via the moldy banana example but never measured. That is a real gap, because if masked logits are dominated by generic language priors, the method could be suppressing correct answers rather than selectively removing hallucinated ones.\n\nNone of this kills the paper. A competent referee could ask for a CAD baseline, error bars, and a token-level diagnostic, and the authors could plausibly deliver them. The paper is short and readable; the idea is simple and easy to reproduce; and the direction of effect is consistent with the stated mechanism. I would send it to peer review, but I would not treat the current numbers as strong evidence.\n\nFor you: this is a reasonable reading-group piece for people interested in inference-time hallucination mitigation, and worth citing once the authors add the missing baseline. I would advise the editor to invite a revision rather than reject.","headline":"Delta is a reasonable inference-only contrastive decoding variant, but the paper skips the closest baseline and reports single-run numbers, so the gains are plausible, not proven.","tokens_in":10927,"tokens_out":1137,"would_cite":false,"duration_ms":14614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that hallucinated tokens can be removed by contrasting the model's logits on an intact prompt with its logits on a randomly masked version, reporting exact-match gains of 3–7 percentage points on four context-rich QA…","keywords":["hallucination mitigation","contrastive decoding","random masking","inference-time decoding","question answering","large language models","adaptive plausibility constraints","SQuAD"],"falsifier":"Measure whether $\\Delta$'s per-token penalization aligns with baseline error rates: if tokens most strongly suppressed by subtracting masked logits are not the tokens the model gets wrong without $\\Delta$, then the masked logits are not functioning as a hallucination prior and the reported gains come from some other re-ranking effect.","tokens_in":9956,"feed_emoji":"🧠","tokens_out":7116,"duration_ms":66970,"temperature":0.7,"pith_summary":"$\\Delta$ is an inference-time decoding rule that tries to suppress hallucinated tokens without retraining: at each step it masks a random fraction of the prompt, computes the model's logits on both original and masked inputs, and samples from a distribution that amplifies the original logits and subtracts the masked ones. The paper's hypothesis is that a model given a corrupted prompt falls back on generic statistical associations, so the masked-input logits approximate the hallucination prior, and removing them leaves tokens that are actually supported by the context. On context-rich question answering, the paper reports exact-match gains of roughly 3 percentage points on SQuAD v1.1, 6 on SQuAD v2, 7 on TriviaQA, and 2 on Natural Questions under sampling, with an even larger jump on SQuAD v2's no-answer exact match. The paper also reports that $\\Delta$ does not help on context-free benchmarks such as CommonsenseQA and MMLU, which it presents as a limitation tied to the masking mechanism.","feed_headline":"Delta decoding lifts exact-match QA scores by up to 7 points","feed_subtitle":"Masking part of the prompt and subtracting the masked logits cuts hallucinations in context-rich question answering.","key_machinery":"The central object is the contrastive decoding distribution in Eq. 5, which combines the model's original logits with logits from a masked version of the same prompt: $P_{\\mathrm{delta}}(y_t \\mid z) = \\mathrm{softmax}[(1+\\alpha)\\mathrm{logit}_\\theta(y \\mid z) - \\alpha\\,\\mathrm{logit}_\\theta(y \\mid \\mathrm{mask}(z))]$. The masking procedure replaces $m = \\lfloor r_{\\mathrm{mask}} \\cdot n \\rfloor$ randomly chosen prompt tokens with a single MASK token, and the adaptive plausibility constraint restricts sampling to tokens that are at least a $\\beta$ fraction as probable as the top token under the original model. This machinery turns the paper's 'moldy banana' intuition—masking removes a fact like 'moldy,' so the model reverts to the prior 'yellow'—into a usable per-token penalty applied at every decoding step.","core_discovery":"On its own terms, $\\Delta$ claims that hallucinated content can be identified by the gap between a model's predictions on an intact input and on a randomly masked version of that input. The decoding rule is $y_t \\sim \\mathrm{softmax}[(1+\\alpha)\\mathrm{logit}_\\theta(y \\mid z) - \\alpha\\,\\mathrm{logit}_\\theta(y \\mid \\mathrm{mask}(z))]$ restricted to tokens whose unmasked probability clears an adaptive threshold, with fixed settings $r_{\\mathrm{mask}}=0.7$, $\\alpha=0.3$, and $\\beta=0.1$ in the experiments. This rule is intended to boost context-dependent tokens while penalizing tokens the model would have produced from its priors alone. The paper reports that the rule improves exact match on SQuAD v1.1, SQuAD v2, TriviaQA, and Natural Questions under sampling, and most strongly improves the ability to answer 'no answer' on SQuAD v2, while leaving context-free benchmarks essentially unchanged.","pith_inferences":["Beyond the paper, the per-token gap between masked and unmasked logits could be used as a hallucination score during answer verification, with strongly penalized tokens flagged as fabrication candidates.","The paper fixes the mask ratio at 0.7 and uses the end-of-sequence token as the MASK; choosing masks that avoid question tokens or target content words could make the contrast sharper, since random masks can hit the very tokens that carry the answer.","A testable corollary is that Delta should help most when an answer is derivable from a few salient context tokens and least when the answer depends on global reasoning; TriviaQA's lack of gain without sampling is consistent with this, though the paper does not study mask locations.","If the masked-input distribution is truly a hallucination prior, then Delta's benefit should correlate with baseline error rates, which would support using Delta as both a decoder and a diagnostic."],"forward_implications":["If Delta works as claimed, any instruction-tuned LLM can get a hallucination reduction on context-rich tasks by adding one extra masked forward pass per decoding step, with no fine-tuning or extra data.","The largest reported effect, more than ten points on SQuAD v2 no-answer exact match, implies that the worst hallucinations in reading comprehension come from cases where the context simply does not contain an answer, and Delta's contrast most helps there.","Because TriviaQA and Natural Questions gains appear mainly with sampling, Delta should be used with sampling decoding when the task involves long, noisy context.","Context-free QA is a known boundary: on CommonsenseQA and MMLU, Delta gives small accuracy declines, so its benefit is tied to external context."],"supporting_citations":[{"why":"Supplies the visual contrastive-decoding template of corrupting the input and contrasting output distributions, which Delta adapts from images to text by masking.","marker":"Leng et al. (2024)"},{"why":"Provides the original contrastive decoding formulation and the Adaptive Plausibility Constraints used in Eq. 3 and Eq. 5.","marker":"Li et al. (2023a)"},{"why":"Defines SQuAD v1.1 and v2, the primary benchmarks where Delta reports exact-match and no-answer gains.","marker":"Rajpurkar et al. (2016)"},{"why":"Supplies TriviaQA, one of the two long-context benchmarks where Delta gains about 7 points under sampling.","marker":"Joshi et al. (2017)"},{"why":"Supplies Natural Questions, the open-domain benchmark where Delta reports a 2-point sampling gain.","marker":"Kwiatkowski et al. (2019)"},{"why":"Supports the random token-masking strategy Delta uses to create ambiguous inputs.","marker":"Wettig et al. (2023)"},{"why":"Context-aware decoding, the nearest inference-time contrastive baseline that Delta positions itself against.","marker":"Shi et al. (2024)"}],"fun_headline_variants":["Delta boosts no-answer exact match by over 10 points","Delta: subtract masked logits to silence fake answers","Masking prompts reveals and suppresses LLM fabrications","Delta: contrastive decoding with masked prompt cuts hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that after part of the prompt is randomly replaced by a MASK token, the model's token probabilities become a trustworthy estimate of what it would hallucinate; if masking suppresses correct answers too, subtracting those logits can re-rank rather than remove hallucinations.","fun_headline_variants_meta":{"raw":{"variants":["Delta boosts no-answer exact match by over 10 points","Delta: subtract masked logits to silence fake answers","Masking prompts reveals and suppresses LLM fabrications","Delta: contrastive decoding with masked prompt cuts hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001251,"raw_usage":{"total_tokens":5131,"prompt_tokens":951,"completion_tokens":4180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":4115}},"tokens_in":567,"tokens_out":4180,"duration_ms":32263,"temperature":1.0,"reasoning_tokens":4115,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:48:37.175006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure whether $\\Delta$'s per-token penalization aligns with baseline error rates: if tokens most strongly suppressed by subtracting masked logits are not the tokens the model gets wrong without $\\Delta$, then the masked logits are not functioning as a hallucination prior and the reported gains come from some other re-ranking effect.","supporting_citations":[],"review_version":1}