{"id":"2d5eb8f3-c607-47c1-bb28-07e90917af98","arxiv_id":"2608.08167","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A Wiener-style spectral filter, estimated from paired truthful and hallucinated hidden states and absorbed into feed-forward weights, suppresses object hallucination in VLMs without changing inference speed.","lead":"The authors propose a training-free, post-hoc method that edits the internal representations of vision-language models to reduce object hallucination, by applying a Wiener-style spectral filter derived from paired truthful and hallucinated captions. The filter is absorbed into the model weights, adds no inference cost, and shows consistent hallucination reductions across LLaVA-1.5, MiniGPT-4, mPLUG-Owl2, Gemma3, and video and dialogue benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix L Table 9 reports normalized cross-covariance exactly 0 for both paired and shuffled data, which is implausible; the MMSE-optimality claim therefore rests on an invalidated diagnostic and is not supported.","rationale":"Table 9 is the only direct evidence supporting Assumption 1, and its exact-zero cross-covariance for both paired and random-pair data is not credible as a finite-sample estimate; either the diagnostic is computed incorrectly or the reported numbers are rounded or orthogonalized. Since Section 3.3 explicitly depends on C = E[s n^T] being negligible to identify the mode-wise Wiener filter with the MMSE estimator, this invalidates the theoretical contribution as presented. I credit the paper for acknowledging the diagonal approximation and for reporting the additivity error (0.099) and the random-pair control, which are the right checks; the problem is the suspicious zero entry. The reader's CONDITIONAL verdict is appropriate; the condition should be an honest recomputation of rho_cross and, if it is non-negligible, a reframing of the method as a covariance-shaped spectral filter rather than an MMSE-optimal Wiener solution. Secondary issues (FaithDial paragraph vs Table 3, no released code) reinforce the need for verification but are not the main threat to the central claim.","tokens_in":19220,"tokens_out":14377,"duration_ms":144818,"concrete_test":"Recompute rho_cross on Gemma3-4B-it layers 12-26 using the Appendix L definitions in float64: s_i = x^-_i - mu_T, n_i = d_i - mu_d, Sigma_sn = (1/N) sum_i s_i n_i^T, with no orthogonalization or rounding; report raw values and bootstrap 95% CIs for paired and shuffled settings. Then compare CHAIRS/CHAIRI on the CHAIR split for the deployed F_alpha, for F_1, and for the exact correlated LMMSE A*_C formula given in Section 3.3. If rho_cross is materially above 0, or if A*_C differs materially from F_alpha, the optimality claim is falsified even if the heuristic still reduces hallucination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is that the mode-wise Wiener gains in Eq. (10)-(12) are the MMSE-optimal solution. This requires Assumption 1 (Eq. 7), E[s n^T] = 0, and Section 3.3 itself concedes the deployed F_alpha is exact only when Sigma_T and Sigma_H are jointly diagonalizable. The only direct validation of Assumption 1 is Appendix L/Table 9, which reports rho_cross = 0 for both the paired setting and the random-pair control. For finite high-dimensional data, a normalized sample cross-covariance cannot be exactly zero in both settings unless the implementation orthogonalizes the paired residuals or reports a different quantity than the definition. If the true rho_cross is non-negligible, the MMSE-optimal operator is A*_C = (Sigma_T + C)(Sigma_T + Sigma_H + C + C^T)^-1, not the diagonal Wiener filter, and the claimed closed-form optimality is unestablished. In addition, the deployed filter uses model-specific tuned alpha (10-60) and layer ranges, so it is a heuristic spectral shrinkage rather than the unmodified MMSE solution. The empirical CHAIR/POPE reductions may still be real, but the theoretical framing is not supported by the presented evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, post-hoc representation-space filter for suppressing object hallucinations in vision-language models. Hidden states are modeled as the sum of a truthful component and a hallucination-associated distortion, and the method derives a Wiener-type estimator from paired truthful/hallucinated calibration data. The resulting mode-wise spectral filter is absorbed into the feed-forward output projections of selected deep layers, so inference-time cost is unchanged. Experiments on LLaVA-1.5, MiniGPT-4, mPLUG-Owl2, Gemma3, and a diffusion-based dialogue model report consistent hallucination reductions on CHAIR, POPE, MME, TempCompass, and FaithDial, including cross-domain transfer results.","tokens_in":19452,"tokens_out":8764,"duration_ms":87891,"significance":"If the empirical findings hold, the method is practically valuable: it is training-free, requires no inference-time overhead, and generalizes across model families and modalities (image, video, and grounded dialogue). The consistent gains across benchmarks and the cross-domain transfer experiments are strengths, as is the clear presentation of the offline weight-edit procedure. However, the paper's central theoretical claim—that the deployed filter is the closed-form MMSE-optimal Wiener estimator—is not supported by the implementation or by the presented diagnostics. The actual method involves a tuned sharpness exponent and layer range, and the validation of the key uncorrelatedness assumption is implausible as reported. The contribution is better framed as a heuristic spectral shrinkage method with strong empirical support than as a provably optimal estimator. With appropriate revisions of the theoretical claims and diagnostics, the paper could be a useful contribution to hallucination mitigation.","major_comments":[{"comment":"The deployed filter F_alpha in Eq. (12) is not the MMSE-optimal operator A* in Eq. (9) unless Sigma_T and Sigma_H are jointly diagonalizable, which the paper does not establish; the text itself concedes this at the end of Section 3.3. Since the sharpness exponent alpha and the edited layer range are selected per model on a validation set (Section 4, Implementation Details), the method actually evaluated is a tuned spectral shrinkage, not the closed-form Wiener solution highlighted in the abstract. The optimality language in the abstract and introduction should be removed or substantially qualified, and the authors should quantify, on the calibration set, the distance ||F_alpha - A*|| in the actual operating regime (for example, for the reported alpha values) so that the approximation error is explicit.","section":"Section 3.3, Eqs. (9)-(12)"},{"comment":"The normalized cross-covariance rho_cross is reported as exactly 0 in both the paired condition and the random-pair control. For finite-sample high-dimensional activations, an exact zero in both settings is implausible and strongly suggests the reported quantity is not the sample cross-covariance defined in the appendix, or that an unintended orthogonalization was applied. Because the derivation of Eq. (9) rests on Assumption 1 (Eq. (7)) and Section 3.3 explicitly points to Appendix L as the empirical validation of that assumption, this diagnostic is load-bearing. The authors must recompute rho_cross according to its definition, report the nonzero values (with reasonable precision), and explain why the random-pair control gives the same number as the paired condition. Without a credible cross-covariance estimate, the MMSE interpretation is unsupported.","section":"Appendix L, Table 9"},{"comment":"The covariances Sigma_T and Sigma_H are estimated from sequence-averaged representations x_i^+ and x_i^- as defined in Eq. (4), and the MMSE derivation concerns those averaged vectors. However, the correction is applied token-wise through the FFN output matrix, acting on every token-level hidden state at inference (Algorithm 1, line 11). The paper does not justify that per-token covariance statistics match the sequence-averaged ones; if they differ, the MMSE optimality (even under Assumption 1) does not transfer to the actual edited object. The authors should either calibrate on per-token features or provide evidence that the per-token and sequence-average distributions are comparable in the layers where the filter is applied.","section":"Section 3.2 (Eq. 4) vs. Algorithm 1 (line 11)"}],"minor_comments":[{"comment":"The table rows for CHAIRS and CHAIRI contain concatenated numbers (for example, “14.013.0” and “5.04.9”), which makes the comparison between the “Our (layers 24–32)” column and the others ambiguous; please format the values as separate columns.","section":"Section 4.2, Table 4"},{"comment":"The text states that FaithCritic reports a reduction from 92% to 74% and that a Claude Sonnet 4.6 judge shows a reduction from 93% to 50%, but Table 3's FaithCritic column shows 85.92 to 75.39. These numbers need to be reconciled; report the LLM-judge results in the table or clarify which metric corresponds to which set of numbers.","section":"Section 4.1, Table 3 and text"},{"comment":"Lemma 1 restates the definition of an operator-Lipschitz function; the proof is correct but the lemma itself adds little beyond that definition. Consider stating the stability property directly for the specific spectral gains used here, or moving the lemma to the appendix and keeping only the non-expansiveness observation in the main text.","section":"Section 3.3, Lemma 1"},{"comment":"For mPLUG-Owl2, the BLEU score under Ours (0.142) is notably lower than several baselines (e.g., 0.151–0.154). The text says the method maintains “competitive” caption quality, but the drop should be addressed quantitatively, especially since caption fluency is one of the stated desiderata.","section":"Section 4.1, Table 1"},{"comment":"The sensitivity of alpha is only shown for LLaVA-1.5; since the reported alpha values differ across models (10, 20, 60), please show the analogous sensitivity curves for MiniGPT-4 and mPLUG-Owl2 (or state that the behavior is similar and give a representative plot).","section":"Appendix E"},{"comment":"Sigma_H is defined as the sample second moment of the residuals d_i without explicit mean centering. Since the text later discusses centering the residual distribution, please clarify whether the mean is subtracted in practice and whether that affects the definition of Sigma_H in Eq. (6).","section":"Section 3.2, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The paper is an extension of the authors' earlier spectral filtering work (Ali et al., 2024), and its relationship to Nullu's hard projection is acknowledged. The empirical scope is broader (multiple models, video, dialogue, cross-domain), and the zero-overhead weight-edit framing is appealing. The main issue is not the experiments but the gap between the theoretical narrative and the implemented method, plus the implausible assumption diagnostic in Appendix L. These are fixable with an honest reframing and corrected diagnostics, so I view major revision (not rejection) as appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a solid empirical paper with an over-stated theoretical frame. The core trick—estimate covariances from paired truthful and hallucinated activations, build a spectral filter in the eigenbasis of the distortion covariance, absorb it into FFN output projections—is a legitimate extension of Nullu and the authors' own spectral truncation. Doing it as soft Wiener-style attenuation rather than hard projection is new, and it is genuinely useful: no training, no inference overhead, and consistent CHAIR/POPE/MME reductions across three older VLMs plus Gemma3, TempCompass, and a diffusion dialogue model. The cross-domain transfer (LURE→TruthfulQA, HaluEval→CHAIR) is a nice bonus.\n\nThe paper itself concedes the main mismatch: the deployed F_alpha with tuned alpha and selected layers is not the MMSE-optimal A* from Eq. 9. That alone is fine—a heuristic shrinkage that works is still a contribution—but the abstract and intro repeatedly call the gains \"closed-form optimal,\" which is more than the evidence supports. The validation of Assumption 1 is where I get more worried. Appendix L reports normalized cross-covariance exactly 0 for both the paired and the random-pair control. For finite high-dimensional samples, exact zero in both settings is not credible; either the quantity is mis-defined or the implementation orthogonalizes implicitly. The additivity error of 0.099 for paired data is at least a real diagnostic, and it is reasonably small, so the uncorrelatedness assumption is probably not load-bearing in practice. But the paper should fix the Table 9 reporting and state clearly that optimality is conditional on a joint-diagonalizability approximation.\n\nAlso minor: Lemma 1 is a standard operator-Lipschitz stability result and does not specifically validate that the empirical covariance errors are small; it is a bit of a placeholder. The FaithDial narrative in the text (92%→74%, Claude 93%→50%) does not match Table 3 (85.92→75.39, no Claude column)—probably a copy-paste inconsistency, but it needs fixing. No code release, which matters for a weight-editing method.\n\nWho is this for? Anyone working on post-hoc hallucination mitigation; it is a useful baseline and an interesting way to think about representation-space editing. It deserves a serious referee; the empirical core is reproducible in principle and the method is simple enough to verify. I would want the theoretical framing toned down and the diagnostics cleaned up before publication, but this is not a desk-reject.","headline":"A genuinely useful training-free hallucination filter with an over-stated MMSE-optimality story; the empirics are solid, the theory framing needs a reset, and one diagnostic table looks wrong.","tokens_in":20010,"tokens_out":1709,"would_cite":true,"duration_ms":17204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Object hallucination in vision-language models can be suppressed by a closed-form Wiener filter on hidden representations, folded into the weights so inference runs at the same speed.","keywords":["object hallucination","vision-language models","Wiener filter","representation-space editing","MMSE estimation","hallucination suppression","low-rank distortion","training-free mitigation"],"falsifier":"Re-run the assumption diagnostics of Appendix L on Gemma3 measuring the normalized cross-covariance $\\rho_{\\mathrm{cross}} = \\|\\Sigma_{sn}\\|_F/\\sqrt{\\|\\Sigma_T\\|_F\\|\\Sigma_H\\|_F}$ from the actual paired activations instead of reporting it as zero, then replace $F_\\alpha$ with the correlated estimator $A^\\star_C = (\\Sigma_T + C)(\\Sigma_T + \\Sigma_H + C + C^\\top)^{-1}$ and compare CHAIR and POPE scores; if $A^\\star_C$ clearly beats $F_\\alpha$, the diagonal mode-wise filter is leaving measurable performance on the table, and if the scores are close, the uncorrelatedness assumption is not load-bearing.","tokens_in":18998,"feed_emoji":"🧠","tokens_out":16842,"duration_ms":138317,"temperature":0.7,"pith_summary":"The paper claims that object hallucination in vision-language models is not random noise but a structured, low-rank distortion in the hidden representations of the language backbone, and that removing it is a linear minimum mean-square error problem. Treating each hidden state as a truthful component plus a hallucination-induced distortion, the optimal linear correction is a Wiener operator whose mode-wise gains are given in closed form from the covariances of paired truthful and hallucinated captions. Because the resulting spectral filter is applied once to the feed-forward output projections of selected deeper layers, the edited model generates at the same speed as the original, with no gradients, fine-tuning, or decoding-time overhead. On CHAIR, POPE, and MME, the filtered LLaVA-1.5, MiniGPT-4, mPLUG-Owl2, and Gemma3 hallucinate less while keeping caption fluency roughly unchanged, and the same filter carries over to TempCompass video reasoning and to grounded dialogue in a discrete diffusion language model.","feed_headline":"A closed-form filter removes VLM hallucinations at no extra cost","feed_subtitle":"No training, no slow decoding: one offline covariance estimate cuts hallucinations across VLMs.","key_machinery":"The load-bearing object is the Wiener spectral filter $F_\\alpha = Q\\,\\mathrm{diag}(\\tilde\\gamma_1,\\dots,\\tilde\\gamma_d)\\,Q^\\top$, a spectral operator built from the eigendecomposition of the hallucination covariance $\\Sigma_H = Q\\Lambda Q^\\top$. For each mode $j$, the distortion-to-signal ratio $\\nu_j = \\lambda_j/\\tau_j^2$, with $\\tau_j^2 = q_j^\\top \\Sigma_T q_j$ the truthful variance along that mode, sets the gain $\\tilde\\gamma_j = (1 + \\nu_j)^{-\\alpha}$: hallucination-dominated modes are attenuated, truthful modes are left nearly intact, and $\\alpha$ sharpens or softens the attenuation without reordering the modes. The filter is calibrated once offline from roughly 3,000 paired truthful/hallucinated LURE captions per model, using only forward passes and empirical second-order statistics, then folded into the FFN output projection $\\tilde W_\\ell^{\\mathrm{out}} = F_\\alpha W_\\ell^{\\mathrm{out}}$ of selected deeper layers, so inference cost is unchanged. A stability lemma, proved with operator-Lipschitz spectral maps, guarantees that small covariance estimation errors induce proportionally bounded changes in the operator.","core_discovery":"The central claim is that hallucination suppression reduces to a closed-form linear filter in the representation space of the language backbone: model each hidden state as $h = s + n$, a superposition of a truthful component $s$ and a hallucination-associated distortion $n$, and assume the two are zero-mean and mutually uncorrelated. Then the covariance of the observed representation decomposes as $\\mathrm{Cov}(h) = \\Sigma_T + \\Sigma_H$, and the linear MMSE estimate of the truthful component is $A^\\star = \\Sigma_T(\\Sigma_T + \\Sigma_H)^{-1}$. The paper implements a diagonalized version of this estimator in the eigenbasis of $\\Sigma_H$: each spectral mode $j$ is attenuated by the gain $\\tilde\\gamma_j = (1 + \\lambda_j/\\tau_j^2)^{-\\alpha}$, where $\\lambda_j$ is the distortion variance, $\\tau_j^2$ is the truthful variance along that mode, and $\\alpha$ is a sharpness exponent. The resulting operator $F_\\alpha = Q\\,\\mathrm{diag}(\\tilde\\gamma_1,\\dots,\\tilde\\gamma_d)\\,Q^\\top$ is absorbed into the FFN output projection of selected deeper layers, $\\tilde W_\\ell^{\\mathrm{out}} = F_\\alpha W_\\ell^{\\mathrm{out}}$, so the deployed model runs unchanged and at the same speed. The empirical claim is that this weight edit lowers object hallucination on CHAIR, POPE, and MME across LLaVA-1.5, MiniGPT-4, mPLUG-Owl2, and Gemma3, extends to TempCompass video reasoning and FaithDial grounded dialogue, and stays stable because the gains are continuous functions of the estimated covariances.","pith_inferences":["The Appendix L diagnostics double as a cheap transfer test for any new VLM: collect a few hundred paired captions and measure the additivity error $\\delta_{\\mathrm{add}}$ and top-16 mode concentration, and if they land near the reported values (about 0.1 and 0.95), the Wiener edit should transfer with little retuning, while large deviations would signal miscalibrated gains.","Table 9 reports the normalized cross-covariance $\\rho_{\\mathrm{cross}}$ as exactly 0 for both paired and random-pair settings, which cannot literally hold for finite high-dimensional samples; the entry is either an idealized or rounded report, or the activations were orthogonalized before the diagnostic ran, and measuring $\\rho_{\\mathrm{cross}}$ honestly and comparing the correlated estimator $A^\\","The exact MMSE gain profile ($\\alpha = 1$) is not the operating point: the paper's sensitivity analysis shows the best hallucination scores at sharpness values in the tens, so the closed-form optimality is a starting configuration that is then sharpened to compensate for covariance estimation error and distribution shift.","Nothing in the derivation is specific to vision: the paired-residual construction requires only truthful and hallucinated renderings of the same input, so the same calibration could be applied to language-only, speech, or retrieval-augmented backbones, and Appendix J's cross-domain results (filter learned on HaluEval, evaluated on CHAIR) already point toward dataset-transferable distortion geometr"],"forward_implications":["Hallucination suppression becomes an offline weight edit: any VLM whose hidden states admit the paired calibration can be corrected before deployment, with zero added inference cost.","The filter touches only the language backbone's FFN output projections, so it applies equally to early-fusion, bridging, and mid-fusion VLM designs without architectural changes.","Because the edit is a weight transformation rather than a decoding rule, it composes with decoding-time strategies; for mPLUG-Owl2 the paper's best POPE numbers come from the filtered model combined with beam search.","The heavily tailed spectrum of $\\Sigma_H$ means a handful of dominant modes captures most hallucination-related variance, so a compact spectral signature of a model's hallucination tendency is available from the top few dozen eigendirections.","The same representation-space edit improves temporal reasoning on TempCompass and grounded dialogue in a discrete diffusion language model, indicating that the distortion geometry is shared across modalities, tasks, and generation paradigms."],"supporting_citations":[{"why":"Supplies the classical Wiener filtering and MMSE estimation framework that the representation-space estimator is derived from.","marker":"(Oppenheim, 1999)"},{"why":"The LURE dataset; provides the paired truthful and hallucinated captions used to estimate $\\Sigma_T$ and $\\Sigma_H$.","marker":"(Zhou et al., 2024)"},{"why":"The authors' earlier spectral representation filtering paper, which this work extends from hard truncation to smooth covariance-driven attenuation.","marker":"(Ali et al., 2024)"},{"why":"Defines CHAIR, the primary object-hallucination metric the filter is evaluated on.","marker":"(Rohrbach et al., 2018)"},{"why":"Defines POPE, the object-presence probing benchmark used in the main evaluation.","marker":"(Li et al., 2023b)"},{"why":"Defines MME, the broader perception benchmark whose per-subset scores test generality.","marker":"(Fu et al., 2023)"},{"why":"HALC; supplies the official CHAIR evaluation scripts and a strong decoding-time baseline.","marker":"(Chen et al., 2024)"},{"why":"Nullu, the weight-editing baseline that hard-projects out hallucination directions, which the Wiener filter softens.","marker":"(Yang et al., 2025)"}],"fun_headline_variants":["Closed-form Wiener filter removes VLM hallucinations","Training-free weight edit suppresses VLM hallucinations","One offline covariance estimate fixes VLM hallucinations","Wiener representation filter cuts VLM hallucinations","No-training closed-form filter reduces VLM hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The filter stands on one statistical assumption: that the hallucination distortion and the truthful part of a hidden state are uncorrelated on average (zero cross-covariance), so the variance of any observed representation is simply the sum of the two variances, and if that cross-covariance is not negligible, the closed-form Wiener gains are no longer optimal and could in principle amplify the hallucination.","fun_headline_variants_meta":{"raw":{"variants":["Closed-form Wiener filter removes VLM hallucinations","Training-free weight edit suppresses VLM hallucinations","One offline covariance estimate fixes VLM hallucinations","Wiener representation filter cuts VLM hallucinations","No-training closed-form filter reduces VLM hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":3129,"prompt_tokens":1181,"completion_tokens":1948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":797,"completion_tokens_details":{"reasoning_tokens":1881}},"tokens_in":797,"tokens_out":1948,"duration_ms":17158,"temperature":1.0,"reasoning_tokens":1881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:19:37.908155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the assumption diagnostics of Appendix L on Gemma3 measuring the normalized cross-covariance $\\rho_{\\mathrm{cross}} = \\|\\Sigma_{sn}\\|_F/\\sqrt{\\|\\Sigma_T\\|_F\\|\\Sigma_H\\|_F}$ from the actual paired activations instead of reporting it as zero, then replace $F_\\alpha$ with the correlated estimator $A^\\star_C = (\\Sigma_T + C)(\\Sigma_T + \\Sigma_H + C + C^\\top)^{-1}$ and compare CHAIR and POPE scores; if $A^\\star_C$ clearly beats $F_\\alpha$, the diagonal mode-wise filter is leaving measurable performance on the table, and if the scores are close, the uncorrelatedness assumption is not load-bearing.","supporting_citations":[{"cited_title":"Object Hallucination in Image Captioning","cited_arxiv_id":null,"evidence_quote":"Defines CHAIR, the primary object-hallucination metric the filter is evaluated on."},{"cited_title":"Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models","cited_arxiv_id":null,"evidence_quote":"HALC; supplies the official CHAIR evaluation scripts and a strong decoding-time baseline."}],"review_version":1}