{"id":"99c2ccff-4df2-48fc-8977-35d90d0fed45","arxiv_id":"2507.07731","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An energy-guided, training-free decoding rule that chooses the layer with minimal energy reduces object hallucination and yes-bias on several benchmarks.","lead":"A simple decoding rule helps vision-language models answer object questions with less 'yes' bias. The rule reads the model's internal layers, picks the one with the lowest energy score, and uses that layer to generate the answer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Energy-guided gains need a fixed-layer control: §4.4 states the energy rule mostly selects the penultimate layer, but no baseline decodes from a fixed penultimate layer, so the reported gains may be early-exit effects rather than energy selection.","rationale":"The paper contributes a simple, training-free, single-forward-pass decoding rule with large reported gains on several benchmarks, and the yes-ratio analysis is a useful empirical observation. The most load-bearing question is whether the proposed energy criterion is causal or merely a proxy for early exiting. Because the authors themselves report that the penultimate layer has the lowest energy and is predominantly selected, the absence of a fixed penultimate-layer baseline leaves the central claim underdetermined. The consistency overclaim is secondary but real: Table 3 shows settings where the method underperforms greedy decoding, so the abstract's wording overstates the evidence. The proposed concrete test directly resolves the attribution issue, and the author's own discussion in §4.4 makes this the natural next experiment. I agree with the reader's weakest-assumption analysis and do not move the verdict: the paper should be conditional on adding the fixed-layer control, releasing code, and revising the consistency claim.","tokens_in":23399,"tokens_out":5145,"duration_ms":62134,"concrete_test":"Run energy-guided decoding and three controls on POPE-GQA (random, popular, adversarial), MME, and MMVP with LLaVA-1.5: (1) fixed penultimate layer using h^{L-1}_t, (2) a fixed layer chosen on a held-out subset by lowest average energy, and (3) a random layer drawn from the same distribution as k*. Also report the fraction of decoding steps where k* equals the penultimate layer. If controls (1) and (2) match energy-guided accuracy, F1, and yes-ratio gap within sampling noise, the energy ranking is not the operative cause; if energy-guided beats both controls, the mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is that argmin_k Energy(h^k_t) identifies the hidden state whose decoding is most grounded. The load-bearing assumption is that this selection rule, rather than simply decoding from an earlier layer, causes the improvements. In §4.4 and §A.8 the authors report that the penultimate layer has the lowest energy score and is 'predominantly utilized' for decoding; Figure 6 shows the penultimate layer's energy is generally the minimum. Yet no experiment decodes from a fixed penultimate layer, and no random-layer control is reported. If fixed penultimate decoding reproduces the accuracy/F1/yes-gap gains, then the energy ranking adds no causal content and the central claim reduces to 'early exit to the penultimate layer helps', which is a different and much simpler claim. The issue is compounded by the consistency overclaim in the abstract: on POPE-MSCOCO with LLaVA-1.5 and InstructBLIP, energy-guided decoding is worse than greedy in accuracy, F1, and yes-ratio gap in Table 3, so the statement that the method 'consistently improves accuracy and F1' is not supported by the paper's own results. A revised version should add the missing controls and correct the overclaim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'energy-guided decoding' for large vision-language models: at each token-generation step, the method projects every decoder hidden state through the LM head, computes the energy score -LogSumExp of the resulting logit vector, and decodes from the layer with the minimal energy score. The method is motivated by an observed transfer of yes/no response bias from language-prior behavior to visual question answering, and requires no training, contrastive decoding, external models, or image perturbation. Experiments on POPE (MSCOCO/A-OKVQA/GQA), MME, MMVP, and CHAIR, with LLaVA-1.5, InstructBLIP, and mPLUG-Owl2, report accuracy and F1 improvements and yes-ratio-gap reductions on several settings, and the abstract claims consistent improvement over baselines. The paper includes layer-wise energy visualizations and a confidence-calibration analysis.","tokens_in":3539,"tokens_out":2132,"duration_ms":111040,"significance":"If the mechanism holds, the method is an attractive minimal-cost intervention: it is hyperparameter-free, requires a single forward pass, and is compatible with standard autoregressive decoding. Its strengths are the absence of fitted parameters, the evaluation across three architectures and several hallucination benchmarks, and the clear analysis of yes-ratio transfer and confidence asymmetry. However, the central causal claim—that the energy score identifies the most reliable layer for decoding—is not yet established, because the energy rule almost always selects the penultimate layer and no fixed-penultimate-layer control is provided. In addition, the paper's own tables contradict the abstract's claim of consistent improvement. Both issues are addressable, so the work is a plausible but currently under-supported contribution.","major_comments":[{"comment":"The paper reports that the penultimate layer has the lowest energy score and is 'predominantly utilized' for decoding, but no experiment decodes from a fixed penultimate layer or from a randomly selected earlier layer. Without such a control, the accuracy/F1/yes-gap gains in Tables 3–5 could be an early-exit effect of decoding from layer L-1 rather than an effect of the energy-based selection rule. Please add fixed-penultimate, fixed-layer (e.g., layer L-2), and random-layer decoding controls on the same benchmarks, and report the distribution of selected layers per dataset and model, to show that the energy ranking itself has predictive content beyond simply decoding from an earlier layer.","section":"§4.4, §A.8, Fig. 6"},{"comment":"The claim that the method 'consistently improves accuracy and F1 score on three VQA datasets across three commonly used VLMs' is contradicted by the paper's own results. On POPE-MSCOCO random with LLaVA-1.5, energy-guided decoding yields 87.50 accuracy and 86.22 F1 versus greedy's 89.37 and 89.33, and the yes-ratio gap worsens from 0.37 to 9.30; with InstructBLIP the corresponding numbers are 86.80/85.10/11.40 versus 90.17/89.86/3.03. On MMVP (Table 5), F1 is slightly lower than greedy for both LLaVA-1.5 (67.67 vs 67.90) and InstructBLIP (66.67 vs 68.12). Please qualify the claims to the settings where improvements actually hold, and define the scope of the 'average accuracy improvement 4.82%' (supplementary A.4 reports a different average, 4.37%, for LLaVA-1.5).","section":"Abstract, Table 3, Table 5"},{"comment":"The MMVP evaluation uses a subset of 122 image-question pairs that yield a yes/no answer after applying the POPE-like template, and the manuscript corrects two dataset answers without detailed justification. The yes-ratio gap in Eq. (4) assumes the expected yes ratio is 0.5, but nothing guarantees that the selected subset is balanced. Please report the ground-truth yes/no counts for the selected subset, state the exact selection procedure, and analyze the sensitivity of the reported gaps and accuracies to the corrected answers and to alternative subsets, including the full 300-pair MMVP benchmark.","section":"§A.1, Eq. (4), Table 5"},{"comment":"The energy score is applied to logits produced from hidden states at different layers, but only the final layer's logits are trained for use with the LM head; intermediate layers may have different logit scales and calibration. The observation that the penultimate layer is almost always chosen raises the possibility that the minimum-energy rule selects a layer by logit scale rather than by representational reliability. To support the OOD/evidence interpretation, please add a variant that normalizes the logits (e.g., by vector norm or temperature) before computing the energy, and compare the selected layers with and without normalization.","section":"§3.3, Eq. (3), Fig. 6"}],"minor_comments":[{"comment":"The variable named 'hidden energy' stores LogSumExp, not the negated energy of Eq. (3), and the code sorts descending while the comment refers to 'highest energy state'; both the naming and the comment should be corrected to make the sign convention unambiguous.","section":"Algorithm 1"},{"comment":"There is a typo, 'accurcy', and a few other grammatical slips; please proofread the experiment section carefully.","section":"§4.3"},{"comment":"The y-axis label 'Expected Yes Ratio' is misleading because the plots show the observed yes ratio; relabel it to 'Yes Ratio' and note in the caption that the green line is the expected ratio.","section":"Figure 1"},{"comment":"Reference [38] is cited as a blog post with no URL; since the method relies heavily on the logit lens, include a full citation with URL and access date.","section":"References"},{"comment":"The text states POPE has 27,000 query-answer pairs while Table 2 lists 9,000 pairs per dataset; please clarify that 27,000 is the total across the three datasets (9,000 × 3) so the reader does not read the table as inconsistent.","section":"§4.2, Table 2"},{"comment":"CHAIR_I increases for both LLaVA-1.5 (5.4 to 9.4) and InstructBLIP (13.3 to 14.4) relative to greedy; since the paper motivates preserving open-ended generation, this regression deserves discussion in the main text rather than only in the experiment section, ideally with a fixed-penultimate-layer comparison to separate early-exit effects from energy-selection effects.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I do not think rejection is warranted, because the missing control and the overclaim are fixable within the manuscript's scope. I would, however, ask the authors to release code or to report per-layer selection statistics in detail, since the sign convention in Algorithm 1 and the dominance of the penultimate layer make reproducibility and interpretation the crux of the paper. The MMVP subset also appears to be a newly defined evaluation protocol, so it should either be validated against the original MMVP protocol or clearly positioned as a new benchmark with balance statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core contribution is the observation that decoding from the layer with minimal energy score (computed via the logit lens) reduces yes-bias and often improves accuracy on POPE, MME, and MMVP across three LVLMs. Second, the paper's own ablation shows the energy rule mostly picks the penultimate layer, yet there is no fixed-penultimate-layer baseline. That missing control is the difference between a true energy-selection effect and a simple 'early exit to layer -1' effect.\n\nWhat's new: applying energy scores to select among layer-wise logits for hallucination mitigation is a genuine combination. The yes-ratio transfer result (Figure 3) is a nice empirical finding. The method is refreshingly simple: one forward pass, no external models, no fitted parameters. The gains on A-OKVQA and GQA are substantial. The numbers look internally consistent, and the supplementary tables are detailed.\n\nSoft spots. The abstract says 'consistently improves accuracy and F1' but Table 3 shows the method is worse than greedy on POPE-MSCOCO with LLaVA-1.5 (87.50 vs 89.37 accuracy; yes-gap 9.30 vs 0.37) and with InstructBLIP in several settings. That overclaim needs correcting. The missing fixed-layer baseline is the bigger issue; without it, the energy score may add no causal content. Also, MMVP is evaluated on a hand-selected subset of 122 pairs, and no variance or significance is reported. Minor: the pseudocode names a variable 'max_idx' when it actually selects the minimum energy; the code works, but the name confuses.\n\nBottom line: this is a plausible, cheap method with real but uneven gains. The mechanism is under-identified. I would send it to a serious referee if the authors add fixed-layer and random-layer controls, correct the consistency claim, and qualify the MMVP subset. As is, I'd call it borderline.","headline":"Simple, training-free decoding trick that cuts yes-bias, but the missing fixed-penultimate-layer baseline leaves the mechanism unresolved.","tokens_in":24207,"tokens_out":1683,"would_cite":false,"duration_ms":18641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decoding from the hidden layer with the lowest energy score reduces object hallucination in vision-language models, improving accuracy and F1 while shrinking the yes-ratio gap.","keywords":["object hallucination","vision-language models","energy-guided decoding","yes-ratio bias","early exiting","hidden states","POPE","MME benchmark"],"falsifier":"Decode identically from a fixed penultimate layer, the layer the energy rule most often picks, across the same POPE, MME, and MMVP evaluations; if accuracy, F1, and yes-ratio gap match the energy-guided results, then the energy ranking itself adds nothing beyond early exiting.","tokens_in":23186,"feed_emoji":"⚖️","tokens_out":9097,"duration_ms":85518,"temperature":0.7,"pith_summary":"The paper sets out to show that a large share of object hallucination in vision-language models is a decoding bias: the language decoder over-produces \"Yes\" answers, and the bias grows as questions get harder. It proposes energy-guided decoding, which at each token position computes an energy score from the vocabulary-head logits of every decoder layer and decodes from the layer with the minimal score. Across POPE, MME, and MMVP benchmarks and three different vision-language models, this single-pass, hyperparameter-free rule improves accuracy and F1 over greedy decoding and narrows the gap between predicted and expected yes ratios, with average gains of 4.82% accuracy and 8.81% yes-ratio gap reduction. A reader should care because it suggests hallucination mitigation can be a byproduct of where you read out the representation, not of extra models, image edits, or training.","feed_headline":"Decoding from the lowest-energy layer cuts VLM yes-bias","feed_subtitle":"One forward pass picks a better layer per token, lifting accuracy and F1 on POPE, MME, and MMVP without extra models.","key_machinery":"The energy score is the central object: for a layer's hidden state $h^k_t$, the vocabulary head $H$ maps it to a logit vector and the energy is the negative LogSumExp of those logits, $E(h^k_t) = -\\operatorname{LogSumExp}[H(h^k_t)]$. The mechanism is per-token early exiting, the logit-lens technique of reading out from any transformer layer, steered by the criterion that the lowest-energy layer is the most reliable representation of the grounded input. The work this does is turning hallucination mitigation into a layer-selection problem solvable in one forward pass.","core_discovery":"The central claim is that the final layer of a vision-language decoder is not the best layer to decode from when the task is visual grounding. Using the \"logit lens\" early-exit idea, the paper projects each decoder layer's hidden states through the learned vocabulary head to obtain logits, defines the energy of layer $k$ at token $t$ as $E(h^k_t) = -\\operatorname{LogSumExp}[H(h^k_t)]$, and selects $k^* = \\arg\\min_k E(h^k_t)$ for decoding. Empirically the penultimate layer usually carries the minimal energy, and decoding from it produces better-calibrated Yes/No confidence, higher accuracy and F1, and a smaller yes-ratio gap than greedy decoding from the final layer on POPE, MME, and MMVP across three open-sourced vision-language models. The paper interprets the phenomenon as the language prior leaking through the final layer, which the energy criterion resists.","pith_inferences":["If the energy rule mostly selects the penultimate layer, as the paper's ablation shows, a fixed-penultimate-layer decoder may reproduce most of the gains, implying the real effect is early exiting rather than dynamic energy ranking.","Because the \"Yes\" bias transfers from text-only inputs, the same exit strategy might mitigate confirmation bias in text-only language models, not just vision-language ones.","The reported CHAIR_I degradation suggests the method sometimes drops correct object mentions; combining energy selection with a visual-attention check could recover instance-level recall in captioning.","A natural test is to vary the exit layer within a single long generation, where the optimal layer for the first token (a short yes/no answer) need not be optimal for subsequent tokens."],"forward_implications":["Deploying energy-guided decoding costs a single forward pass, no fine-tuning, and no external models, so it can replace greedy decoding directly in autoregressive vision-language models.","Across POPE, MME, and MMVP, the method improves accuracy and F1 while cutting the average yes-ratio gap by 8.81% relative to greedy decoding.","The advantage is largest on the hardest, adversarial POPE settings, where the \"Yes\" bias is most severe.","Sentence-level caption hallucination (CHAIR_S) is reduced, though instance-level CHAIR_I degrades, a trade-off the paper reports."],"supporting_citations":[{"why":"Supplies the energy score definition, negative LogSumExp over logits, used as the layer-selection criterion.","marker":"[35]"},{"why":"Introduces the logit-lens early-exit technique that this method uses to decode from per-layer hidden states.","marker":"[38]"},{"why":"Provides the POPE benchmark and its yes-ratio evaluation protocol for the main experiments.","marker":"[29]"},{"why":"Provides the noisy-image contrastive decoding baseline and the evaluation protocol (accuracy, F1, yes ratio) that the paper follows.","marker":"[26]"},{"why":"Provides the adaptive-contrast decoding baseline compared on POPE, MME, and MMVP.","marker":"[7]"},{"why":"Provides the MMVP benchmark with CLIP-blind image pairs used to test visually grounded responses.","marker":"[47]"},{"why":"Provides the MME benchmark whose existence, count, position, and color subtasks are used for hallucination evaluation.","marker":"[15]"},{"why":"Defines the CHAIR metrics used to verify that open-ended captioning does not degrade.","marker":"[44]"}],"fun_headline_variants":["Energy-guided layer selection cuts VLM yes-bias","Decode from minimal energy layer to fix VLM hallucinations","VLM yes-ratio gap shrinks with energy-based decoding","Lowest-energy hidden state reduces object hallucination","Layer energy score picks best decoding point for VLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the layer with the lowest energy score at each step is the layer whose hidden states are most grounded in the image; since the energy rule mostly selects the second-to-last layer and no baseline decodes from that fixed layer, the observed gains could come from early exiting itself rather than from the energy-based ranking.","fun_headline_variants_meta":{"raw":{"variants":["Energy-guided layer selection cuts VLM yes-bias","Decode from minimal energy layer to fix VLM hallucinations","VLM yes-ratio gap shrinks with energy-based decoding","Lowest-energy hidden state reduces object hallucination","Layer energy score picks best decoding point for VLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1869,"prompt_tokens":941,"completion_tokens":928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":850}},"tokens_in":557,"tokens_out":928,"duration_ms":8596,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:33:18.676967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decode identically from a fixed penultimate layer, the layer the energy rule most often picks, across the same POPE, MME, and MMVP evaluations; if accuracy, F1, and yes-ratio gap match the energy-guided results, then the energy ranking itself adds nothing beyond early exiting.","supporting_citations":[{"cited_title":"Energy-based out-of-distribution detection","cited_arxiv_id":null,"evidence_quote":"Supplies the energy score definition, negative LogSumExp over logits, used as the layer-selection criterion."},{"cited_title":"interpreting gpt: the logit lens","cited_arxiv_id":null,"evidence_quote":"Introduces the logit-lens early-exit technique that this method uses to decode from per-layer hidden states."},{"cited_title":"Evaluating object hallucination in large vision-language models","cited_arxiv_id":null,"evidence_quote":"Provides the POPE benchmark and its yes-ratio evaluation protocol for the main experiments."},{"cited_title":"Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding","cited_arxiv_id":null,"evidence_quote":"Provides the noisy-image contrastive decoding baseline and the evaluation protocol (accuracy, F1, yes ratio) that the paper follows."},{"cited_title":"Halc: Object hallucination reduc- tion via adaptive focal-contrast decoding","cited_arxiv_id":null,"evidence_quote":"Provides the adaptive-contrast decoding baseline compared on POPE, MME, and MMVP."},{"cited_title":"Eyes wide shut? exploring the visual shortcomings of multimodal llms","cited_arxiv_id":null,"evidence_quote":"Provides the MMVP benchmark with CLIP-blind image pairs used to test visually grounded responses."},{"cited_title":"Object hallucination in image cap- tioning","cited_arxiv_id":null,"evidence_quote":"Defines the CHAIR metrics used to verify that open-ended captioning does not degrade."}],"review_version":1}