{"id":"328f45bd-4283-4ea4-adcd-5efd771f21b1","arxiv_id":"2411.19187","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Middle-layer contextual embeddings, not logit-lens readings, improve hallucination detection in VLMs and enable bounding-box grounding for visual question answering.","lead":"The paper shows that using contextual token embeddings from middle layers of vision-language models detects hallucinations and grounds answers better than the logit lens method on categories like actions, OCR, attributes, and relations. It also introduces a bounding-box grounding technique for training-free grounded visual question answering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-tuned layer selection plus missing error bars leave the headline 'significant improvement' unsubstantiated; several Table 1 margins are within noise.","rationale":"The reader's weakest assumption focuses on the geometric reliability of cosine similarity at a fixed layer pair and whether those layers remain optimal. My concern is adjacent but more specific: the paper selects layers per category on a 250-example validation split and then reports test mAP without any uncertainty quantification. This makes it impossible to tell whether the observed improvements, many of which are a few points or less, are real or noise. The adversarial validation partially addresses layer-choice stability for InternLM-VL, but not for Qwen2-VL and not for the magnitude of the score differences. I do not see a fatal flaw; the method is plausible and the qualitative examples are helpful. However, the central claim of 'significant improvement' requires either confidence intervals or a pre-registered fixed layer choice. The reader's conditional verdict already captures this need, so I recommend no change to the verdict. A bootstrap and layer-selection sensitivity analysis would settle whether the concern lands.","tokens_in":12319,"tokens_out":6310,"duration_ms":64604,"concrete_test":"Compute paired bootstrap 95% confidence intervals for ContextualLens minus logit-lens and ContextualLens minus output-probabilities mAP on the 250 test examples per category, and repeat the full per-category layer-selection procedure on 100 bootstrap resamples of the validation split for both Qwen2-VL-7B and InternLM-XComposer2-VL-7B. If any confidence interval for Action, OCR, or Relation includes zero, or if the selected layer pair changes across resamples for Qwen2-VL, the claim of significant improvement across diverse categories is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for ContextualLens is Table 1, but each per-category mAP is computed on only 250 test examples after choosing the layer pair on a 250-example validation split. No error bars, bootstrap confidence intervals, or significance tests are reported. Several claimed improvements are tiny or inconsistent: on InternLM-VL, Action is 0.796 vs 0.795 for logit lens (+0.001), OCR is 0.772 vs 0.769 (+0.003) and below output probabilities (0.856), and Environment is 0.811 vs 0.835 for output probabilities; on both models, Count is worse than output probabilities. The broad claim of significant improvement 'including actions and OCR' thus rests mainly on Qwen2-VL for those categories. With 250 test samples, mAP differences of 0.01-0.05 are within plausible noise. The adversarial validation in Appendix C checks only InternLM-VL, only whether layer pair (13,27) ranks highly, and not whether the score differences are stable across resamples or models. Therefore the load-bearing assumption that the validation-selected layers generalize and that the observed margins reflect a real signal is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ContextualLens, a training-free method for hallucination detection and grounding in vision-language models. Instead of the logit lens, which maps intermediate activations through the unembedding matrix, ContextualLens uses cosine similarity between average contextual embeddings of answer tokens at a middle text layer and image patch embeddings at a middle image layer. A high maximum patch similarity is interpreted as the answer being visually grounded. The paper also proposes a bounding-box variant that searches over all rectangular boxes for the one most similar to the answer embedding, enabling grounded visual question answering. Experiments on the HQH hallucination benchmark compare ContextualLens with output probabilities and the logit lens across seven categories on InternLM-VL and Qwen2-VL, and grounding is evaluated with PR curves on TextVQA-X and VizWiz-G. The paper claims significant improvements over the logit lens, especially in attribute, comparison, and relation categories, and also demonstrates a capability for grounding answers in images.","tokens_in":12678,"tokens_out":4933,"duration_ms":43843,"significance":"If the empirical claims are reliable, the work is a useful step toward training-free, model-internal verification and grounding for VLMs. The failure analysis of the logit lens on contextual categories is plausible and the idea of using middle-layer contextual embeddings is well motivated by prior work. The bounding-box grounding technique is a reasonable extension and the qualitative examples are suggestive. However, the central quantitative evidence is currently undermined by the absence of uncertainty quantification, the small per-category test sets, and validation-based layer selection whose cross-model robustness is not established. The paper's contribution is potentially valuable, but the evidence offered does not yet support the headline claim of 'significant improvement'.","major_comments":[{"comment":"The central claim that ContextualLens 'significantly improves' hallucination detection is not substantiated by the reported statistics. Each mAP is computed on only 250 test examples after validation-based layer selection, yet no error bars, bootstrap confidence intervals, or significance tests are provided. Several headline margins are within noise: on InternLM-VL, Action is 0.796 vs. 0.795 (logit lens) and OCR is 0.772 vs. 0.769, while output probabilities reach 0.856; on both models, Count is worse than output probabilities (0.885 vs. 0.946 and 0.889 vs. 0.956). The Table 1 caption's bold/underline criterion for 'significantly better' is therefore undefined and unverified.","section":"§6.1, Table 1"},{"comment":"The method's free parameters are the layer indices lT and lI for detection and lb for grounding. Table 1 reports test-split results after selecting these layers on 250-example per-category validation splits. The adversarial validation in Appendix C is performed only for InternLM-VL and only checks that the layer pair (13,27) ranks highly; it does not test whether the chosen layers are stable under resampling or whether the same choice transfers to Qwen2-VL. The layer lb used for the bounding-box technique in §4.2.2 is not specified at all. Without a fixed-layer analysis or cross-model layer-transfer results, the generalization of the validation-selected layers remains an unverified assumption.","section":"§4.2.1 and Appendix C"},{"comment":"The grounding evaluation is reported only as PR curves, with no aggregate metric or uncertainty quantification. The text claims that ContextualLens 'consistently leads to better grounding' and that the bounding-box method's precision is 'often significantly higher,' but these claims cannot be assessed from the figures alone. The authors should report a summary statistic such as AUC or precision at a fixed recall, with bootstrap confidence intervals, for both datasets and both models.","section":"§6.2, Figures 4 and 5"}],"minor_comments":[{"comment":"The adversarial validation set size is given as 1,500 examples, stated as 250 × 6, but Table 1 lists seven categories (Action, Attribute, Comparison, Count, Environment, Relation, OCR); either a category is omitted from the count or the arithmetic is incorrect.","section":"Appendix C"},{"comment":"The caption defines bold and underline as 'significantly better' and 'marginal,' respectively, but no significance test is described anywhere in the paper; please define the criterion or remove the implication.","section":"Table 1 caption"},{"comment":"The abstract's claim of improvement 'across diverse categories' is too broad given that Count is explicitly worse than output probabilities in both models; the authors should temper the wording to reflect the category-dependent results.","section":"Abstract and §6.1"},{"comment":"The bounding-box search enumerates all W^2 × H^2 boxes, but the patch grid resolution for the two tested models is not stated; please provide the grid sizes and a note on computational cost.","section":"§4.2.2"},{"comment":"The qualitative examples are helpful, but the top-k patch selection thresholds used in Figures 6–9 are not specified, which makes it hard to interpret the visualizations.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the core idea is reasonable. The main gap is statistical: the headline claims rest on small test sets with no error bars, and the layer-selection procedure is not shown to generalize across models. These issues are fixable with additional experiments and reporting, so I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a genuine, simple idea with real promise: swap the unembedding logits for middle-layer contextual embeddings when comparing answer tokens to image patches, and you get better hallucination detection on categories like attributes, comparisons, and relations, plus a bounding-box grounding method that actually localizes evidence. Second, the quantitative support is thinner than the abstract suggests: the headline \"significant improvements\" mostly rests on a single validation-tuned layer pair per model, 250 test examples per category, and no error bars or significance tests. The stress-test note is right about that.\n\nWhat's actually new: the paper identifies a clear failure mode of Jiang et al.'s logit lens — it can verify presence but not contextual relationships — and fixes it with a parameter change that is almost trivial to implement. The bounding-box version that averages patch embeddings within candidate boxes and picks the max cosine is a sensible, training-free way to do grounded VQA. The qualitative examples look good, and the adversarial validation in Appendix C, at least for InternLM, suggests the layer choice is not fragile across categories. That's credit where it's due.\n\nThe load-bearing issue is inferential. Table 1 margins like InternLM Action 0.796 vs 0.795 and OCR 0.772 vs 0.769 are inside noise; the broad claim for actions and OCR leans heavily on Qwen2-VL. With 250 test points, a 0.02 mAP gap is not meaningful without a bootstrap or a test. The layer indices are selected on a validation split, and the robustness check covers only one model and checks rank, not margin stability. Also, \"significantly\" in the abstract is doing too much work. The Count category is honestly conceded as a failure. These are fixable: report bootstrap CIs, run a proper significance test, release the code and exact layer choices. The method is so simple that reproducibility is a realistic ask.\n\nWho's it for: people working on training-free interpretability for VLMs, hallucination detection, or grounded VQA. It deserves a serious referee — the idea is worth engaging, and the weaknesses are empirical, not conceptual. I'd send it to peer review with a clear request for more statistical rigor. I wouldn't cite the numbers yet, but I'd cite the approach as a useful baseline and a clean extension of the logit lens.","headline":"A simple, training-free extension of the logit lens that plausibly improves VLM hallucination detection and grounding, but the headline significance claims outrun the statistics.","tokens_in":13095,"tokens_out":2042,"would_cite":true,"duration_ms":17614,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ContextualLens replaces logit-lens probing with middle-layer contextual embeddings to detect VLM hallucinations, including attributes, relations, comparisons, actions, and OCR, and to ground answers as bounding boxes.","keywords":["hallucination detection","vision-language models","contextual embeddings","logit lens","grounded visual question answering","multimodal attribution","training-free interpretability","embedding similarity"],"falsifier":"On a curated set of images where the answer word appears somewhere in the image but not as the property of the object the question asks about, ContextualLens should give a low Confidence_max for a hallucinated answer; if Confidence_max stays high in that setting, the cosine-similarity signal is not actually checking the relation between the answer and the correct object.","tokens_in":1672,"feed_emoji":"🖼️","tokens_out":2015,"duration_ms":85472,"temperature":0.7,"pith_summary":"The paper tries to establish that a training-free method using contextual token embeddings from middle layers of a vision-language model can detect hallucinations and ground answers more reliably than the logit lens, which reads uncontextualized logits from the unembedding layer. On the HQH benchmark, ContextualLens is reported to move hallucination detection in attribute, comparison, and relation categories from near-random to meaningful accuracy, and to outperform the logit lens on actions and OCR. The same similarity signal supports a grounding technique that returns precise bounding boxes, extending zero-shot object segmentation to grounded visual question answering. If the central claim holds, the model internals already used to generate an answer can also verify it and locate its visual evidence without any added training.","feed_headline":"Contextual embeddings catch VLM hallucinations logit lens misses","feed_subtitle":"It catches attribute, relation, and OCR hallucinations the logit lens misses, then locates precise bounding boxes.","key_machinery":"The central object is the contextual embedding vector of answer tokens and image patches taken from intermediate transformer layers. Given answer tokens, let $h_{\\mathrm{Ans}, l_T}$ be the average of token embeddings at text layer $l_T$; for each image patch $p_j$, let $h_{l_I}(p_j)$ be its embedding at image layer $l_I$; the patch score is $\\mathrm{CosineSim}(h_{\\mathrm{Ans}, l_T}, h_{l_I}(p_j))$, and $\\mathrm{Confidence}_{\\max} = \\max_j \\mathrm{Score}(p_j)$ is the hallucination score. For grounding, the paper averages patch embeddings inside every candidate bounding box and selects the box maximizing cosine similarity to the answer embedding. The mechanism works by replacing uncontextualized vocabulary-space logits with contextual embeddings that are known to represent multi-token concepts and relations.","core_discovery":"ContextualLens detects hallucinations by averaging the contextual embeddings of the generated answer tokens at a chosen text layer, then computing the cosine similarity of that average against each image patch embedding at a chosen image layer; the maximum over patches, $\\mathrm{Confidence}_{\\max}$, is taken as the model's confidence that the answer is visually supported. The paper argues that because these middle-layer embeddings are contextual, they capture multi-token concepts and relations that the logit lens cannot, which is why the method succeeds on attributes, spatial relations, comparisons, actions, and OCR. For grounding, ContextualLens either resizes per-patch cosine scores to the image or searches over all candidate bounding boxes for the box whose average patch embedding is most similar to the answer embedding, yielding high-precision regions for grounded visual question answering. The paper's central claim is that this contextual-similarity reading of the model's own representations is a general, training-free route to hallucination detection and multimodal attribution.","pith_inferences":["A natural extension, left implicit by the paper, is to decompose longer or multi-span answers and ground each span separately, which would move the method beyond short factual VQA.","The bounding-box search over all candidate boxes suggests a cheap proposal mechanism for trained segmentation models, using ContextualLens scores as a prior rather than a final mask.","The reported robustness of the chosen layer pair implies the method might transfer across models of similar depth, but models with different layer counts or vision encoders would likely need a lightweight per-model calibration step; this is directly testable.","The same confidence signal could plausibly serve as an abstention criterion, telling the system when not to answer at all, which the paper does not explore."],"forward_implications":["Hallucination detection in VLM categories that used to be near random—attribute, comparison, and relation—becomes feasible without fine-tuning, using only the model's internal embeddings.","Grounding is not limited to object segmentation: answers involving actions, OCR text, and chart content can be localized to bounding boxes in a training-free way.","Because the method uses the same model that generated the answer, it can run at inference time without external detectors or retrievers.","The method is still weaker than output probabilities on counting, a limitation the paper explicitly acknowledges.","Layer selection is part of the procedure: validation-based or adversarial validation can pick the image/text layer pair, with only small reported drops when the tested category is excluded."],"supporting_citations":[{"why":"It introduces the logit-lens training-free hallucination detection and segmentation baseline that ContextualLens extends and compares against.","marker":"(Jiang et al., 2024)"},{"why":"It originally defines the logit lens technique of mapping intermediate activations to vocabulary space.","marker":"(nostalgebraist, 2020)"},{"why":"It supplies the method of averaging token embeddings and using cosine similarity for attribution that ContextualLens adapts from text to vision.","marker":"(Phukan et al., 2024)"},{"why":"It supports the claim that middle layers represent multi-token concepts better than unembedding-layer logits.","marker":"(Wendler et al., 2024)"},{"why":"It provides the HQH benchmark with eight hallucination categories used for detection evaluation.","marker":"(Yan et al., 2024)"},{"why":"It provides the TextVQA-X dataset with grounding masks for OCR-focused grounded VQA evaluation.","marker":"(Rao et al., 2021)"},{"why":"It provides the VizWiz-G dataset with answer grounding masks for grounded VQA evaluation.","marker":"(Chen et al., 2022)"}],"fun_headline_variants":["Middle-layer context beats logit lens on VLM hallucinations","ContextualLens: use middle-layer embeddings to expose VLM hallucinations","No logit lens needed: middle-layer context catches VLM hallucination","From logits to context: a training-free fix for VLM grounding and hallucination","Contextual embeddings: the middle layer's answer to VLM hallucination"],"cache_read_input_tokens":15232,"weakest_assumption_plain":"The load-bearing premise is that cosine similarity between an averaged answer-token embedding and image-patch embeddings, at one fixed pair of middle layers chosen on validation data, reliably indicates whether the answer is visually grounded.","fun_headline_variants_meta":{"raw":{"variants":["Middle-layer context beats logit lens on VLM hallucinations","ContextualLens: use middle-layer embeddings to expose VLM hallucinations","No logit lens needed: middle-layer context catches VLM hallucination","From logits to context: a training-free fix for VLM grounding and hallucination","Contextual embeddings: the middle layer's answer to VLM hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000807,"raw_usage":{"total_tokens":3538,"prompt_tokens":933,"completion_tokens":2605,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2509}},"tokens_in":549,"tokens_out":2605,"duration_ms":18420,"temperature":1.0,"reasoning_tokens":2509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:25:35.450779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a curated set of images where the answer word appears somewhere in the image but not as the property of the object the question asks about, ContextualLens should give a low Confidence_max for a hallucinated answer; if Confidence_max stays high in that setting, the cosine-similarity signal is not actually checking the relation between the answer and the correct object.","supporting_citations":[{"cited_title":"A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations","cited_arxiv_id":"2105.02626","evidence_quote":"It provides the TextVQA-X dataset with grounding masks for OCR-focused grounded VQA evaluation."}],"review_version":1}