Pith. sign in

REVIEW 2 major objections 7 minor 38 references

Attention Inside the Box: A Training-Free Score That Nearly Doubles MLLM Detection Accuracy

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 19:27 UTC pith:564EZXKG

load-bearing objection MTLA: a simple, training-free attention-based confidence score for MLLM localization that works across image, video, and audio — the headline AP number over-attributes gains to MTLA by bundling self-consistency voting, but the core method is sound and genuinely novel. the 2 major comments →

arxiv 2607.05978 v1 pith:564EZXKG submitted 2026-07-07 cs.CV cs.AI

Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention

Daniel Shalam , Emanuel Ben Baruch , Avi Ben Cohen , Tal Remez This is my paper
classification cs.CV cs.AI
keywords multimodal large language modelsobject detectionhallucination detectionattention mechanismstemporal groundingtraining-free confidence estimationzero-shot localization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal large language models can emit bounding boxes and temporal windows, but between 58% and 68% of their localized predictions are hallucinations—regions that do not correspond to any real object or event. The model's own token probabilities are nearly useless for telling good predictions from bad ones, because coordinate tokens become near-deterministic once the model commits to a location. This paper proposes Multi-Token Localized Attention (MTLA), a post-hoc, training-free confidence score that works by reading the model's internal attention maps and checking whether the tokens that encode a predicted region actually attend to the input patches inside that same region. The core insight is twofold: grounded predictions concentrate their attention within their own proposed region, while hallucinated predictions spread attention across contextual distractors elsewhere; and this signal is distributed across all the prediction's tokens (coordinates and label), so aggregating across them yields a far more robust score than reading any single token. Used to re-rank predictions, MTLA lifts an 8B generalist model's zero-shot COCO detection AP from 20.4 to 37.0, approaching supervised specialist detectors, and produces similar gains across video temporal grounding and audio event localization—all without any task-specific training or tuning.

Core claim

The paper's central discovery is that a multimodal LLM's decoder self-attention, when restricted to the modality tokens inside the model's own predicted region and averaged across all tokens comprising that prediction, functions as a reliable grounding confidence signal. Prior attention-based methods summed attention globally across the entire input and read a single response token; the paper shows these are weaker special cases. The localized restriction is what makes the signal discriminative—especially on longer inputs like video, where global attention is nearly uninformative (AUROC 41.5 for the global baseline vs. 80.0 for the localized version on QVHighlights). The multi-token聚合 is the

What carries the argument

Multi-Token Localized Attention (MTLA): a scalar score computed by (1) identifying the response tokens that encode a prediction's region coordinates and label, (2) masking the decoder's attention weights to keep only those directed at input-modality tokens falling inside the predicted region, (3) summing the masked attention per token, (4) averaging across all prediction tokens, and (5) averaging across attention heads and a fixed band of middle transformer layers.

Load-bearing premise

The method assumes that the mass of decoder self-attention from prediction tokens onto input-modality tokens inside the predicted region is a faithful, unbiased indicator of whether the model actually grounded its prediction in the input. This requires extracting raw attention maps (bypassing optimized attention kernels), and the paper acknowledges a mathematical bias toward larger regions from softmax normalization that it does not fully resolve, relying instead on the empir

What would settle it

If grounded and hallucinated predictions showed equal attention mass inside their proposed regions—e.g., if the model attended inside its claimed box regardless of whether a real object was there—MTLA would collapse to chance and provide no re-ranking benefit.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Generalist MLLMs can serve as practical localization systems without task-specific training, provided a reliable post-hoc confidence signal is available to filter hallucinated predictions.
  • The same attention-localization principle transfers across image, video, and audio with no modality-specific modifications beyond constructing the region-to-token mask, suggesting the mechanism is a general property of how grounding MLLMs process spatial and temporal information.
  • Token log-probabilities are fundamentally inadequate as confidence signals for structured coordinate outputs, because they reflect decode-order artifacts and coarse-to-fine digit refinement rather than grounding quality.
  • Standard detection metrics (AP, Recall@1, PSDS1) should become a routine part of MLLM evaluation, not just captioning or VQA benchmarks, to expose the hallucination rates these models exhibit in localization tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If MTLA is reading a real grounding mechanism rather than a correlate, then models trained with explicit attention-alignment losses (forcing prediction tokens to attend inside their proposed regions) should show both higher base localization accuracy and reduced MTLA gains—testing whether MTLA is measuring a cause or a symptom.
  • The region-size bias from softmax normalization (larger regions get more attention mass by construction) suggests that a non-softmax or entropy-normalized attention variant might recover the small-object performance that MTLA currently sacrifices, even though naive area normalization hurt in the paper's experiments.
  • The method's reliance on eager-attention forward passes (bypassing fused kernels like FlashAttention) creates a practical deployment tension: the very inference optimizations that make MLLMs fast also destroy the signal MTLA needs, which may motivate hardware-software co-design for attention map extraction.
  • If the grounding signal lives in middle-layer attention patterns regardless of modality, this raises the question of whether it emerges from the grounding training data itself or from a more general property of how transformers integrate spatial/temporal information—a question testable by applying MTLA to non-grounding-trained models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This paper proposes Multi-Token Localized Attention (MTLA), a training-free, post-hoc confidence score for localized predictions from multimodal large language models (MLLMs). The core idea is to measure how strongly a prediction's tokens attend to the input-modality tokens within the predicted region, aggregating across all prediction tokens. The method is evaluated across image detection (COCO), video temporal grounding (Charades-STA, QVHighlights), and audio localization (AudioSet-Strong), showing consistent improvements in hallucination AUROC and re-ranking accuracy over prior training-free baselines such as SVAR. The ablations in Section 5 cleanly decompose the contributions of the region mask and multi-token aggregation. The central claim—that MTLA is an effective, general confidence score for MLLM localization—is well-supported by the experiments.

Significance. The paper addresses a practical and timely problem: MLLMs hallucinate localized predictions prolifically, and token log-probabilities are nearly uninformative for grounding quality. The proposed MTLA score is parameter-free (no fitted weights, no external models), architecture-agnostic, and applies uniformly across image, video, and audio modalities with no per-benchmark tuning. The layer band (L8-21) is borrowed from prior work (SVAR) and reused without per-benchmark adjustment. The code is released. The falsifiable prediction—that attention mass inside the claimed region separates grounded from hallucinated predictions more sharply than global attention or token probabilities—is confirmed across four benchmarks and three model families. The nearly doubling of zero-shot COCO detection AP (20.4 to 37.0) is a notable result, though the framing of this headline number requires clarification (see major comments).

major comments (2)
  1. The headline claim that MTLA 'nearly doubles' COCO detection AP from 20.4 to 37.0 (Abstract, Sec. 1, Table 3) bundles MTLA re-ranking with N=16 self-consistency voting. Table 3 reports MTLA at N=1 as 32.12 AP, so MTLA's ranking contribution alone is +11.7 AP (a 57% increase). The remaining ~5 points come from self-consistency voting (N=1 to N=16). The critical missing baseline is: N=16 rollouts merged with vote-weighted NMS but without MTLA (i.e., vote count only as confidence). Fig. 6a shows SVAR at N=16 reaches 34.6 AP, suggesting that self-consistency with any reasonable ranking captures much of the gain. If a vote-count-only baseline at N=16 reaches 33-35 AP, then MTLA's marginal contribution at N=16 is only 2-4 points, and the 'nearly doubles' framing substantially over-attributes the gain to MTLA rather than to the 16x oversampling. The paper is transparent about reporting N=1 (32.
  2. 1 AP) separately, so this is a framing concern rather than a hidden flaw. The N=1 result already supports the core claim. However, the abstract and headline number attribute the full 37.0 to MTLA without isolating the self-consistency contribution. Adding the vote-count-only baseline at N=16 and adjusting the framing accordingly would resolve this.
minor comments (7)
  1. Abstract: the AUROC improvement range is stated as '+7 to +38' in the reader's report but as '+7 to +21' in the abstract text provided. Please reconcile the exact range with the reported numbers in Table 2 and Table 10.
  2. Table 3: the Gemini API runs (Gemini 2.5 Flash, Pro, 3.0 Flash) are listed alongside training-free post-hoc scores but use a different model family and single-sample (N=1) API calls. A footnote or separator clarifying that these are not head-to-head comparisons with the Qwen3-VL re-ranking results would improve readability.
  3. Supp. C: the region-size attention bias is acknowledged but not fully resolved. The paper states that normalization schemes degraded performance and that MTLA outperforms SVAR across all size categories. This is a reasonable empirical defense, but a brief discussion of why the bias does not invalidate cross-prediction comparisons (since AP is threshold-dependent) would strengthen the argument.
  4. Sec. 3.2, self-consistency voting: the fusion rule differs between COCO (sum of cluster MTLA scores) and other modalities (max of cluster MTLA scores). The rationale for this difference is stated briefly but could be elaborated, as it affects the comparability of results across modalities.
  5. Table 6: the 'label mean' row is marked as undefined (—) for QVHighlights because predictions are bare [start, end] timestamps. This is clear, but the table caption could note that the dash indicates the token set is not applicable to the output format, not that the experiment was not run.
  6. Supp. A.4: the eager-attention forward pass requirement (bypassing FlashAttention) is a practical limitation. The paper notes this in Sec. 6, but the supplementary could briefly discuss the computational overhead of the eager pass relative to the standard fused-attention inference.
  7. Sec. 4.4 / Table 5: the audio localization pipeline (Supp. A.3) involves a propose-and-localize cascade with an auxiliary language model for label normalization. This is more complex than the image/video setup. A brief note in the main text that the audio pipeline differs structurally would set appropriate expectations.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful reading and the constructive framing suggestion. The referee's single major comment is well-taken: the headline number of 37.0 AP bundles MTLA re-ranking with N=16 self-consistency voting, and the missing baseline (vote-count-only at N=16) is needed to isolate MTLA's marginal contribution at that setting. We agree this should be added and the framing adjusted.

read point-by-point responses

  1. Referee: The headline claim that MTLA 'nearly doubles' COCO detection AP from 20.4 to 37.0 bundles MTLA re-ranking with N=16 self-consistency voting. Table 3 reports MTLA at N=1 as 32.12 AP, so MTLA's ranking contribution alone is +11.7 AP (a 57% increase). The remaining ~5 points come from self-consistency voting (N=1 to N=16). The critical missing baseline is: N=16 rollouts merged with vote-weighted NMS but without MTLA (i.e., vote count only as confidence). Fig. 6a shows SVAR at N=16 reaches 34.6 AP, suggesting that self-consistency with any reasonable ranking captures much of the gain. If a vote-count-only baseline at N=16 reaches 33-35 AP, then MTLA's marginal contribution at N=16 is only 2-4 points, and the 'nearly doubles' framing substantially over-attributes the gain to MTLA rather than to the 16x oversampling. The paper is transparent about reporting N=1 separately, so this is a framing

    Authors: The referee is correct on both counts: (1) the headline number of 37.0 AP bundles MTLA with N=16 self-consistency voting, and (2) the vote-count-only baseline at N=16 is missing and is needed to isolate MTLA's marginal contribution at that setting. We will add this baseline and adjust the framing accordingly. We agree that the N=1 result (32.12 AP, a +11.7 / +57% improvement over raw 20.43) is the cleaner demonstration of MTLA's contribution as a confidence score, and we will foreground it in the abstract and introduction. We will also reframe the 37.0 AP figure to explicitly attribute the components (MTLA re-ranking at N=1 vs. the additional gain from N=16 self-consistency voting), rather than presenting the combined number as if it were solely due to MTLA. We note that Figure 6a already shows that SVAR at N=16 reaches 34.6 AP, which is consistent with the referee's expectation that a vote-count-only baseline would land in the 33–35 range; the vote-count-only baseline will make this explicit. We will add it to Table 3 and/or Figure 6a. We will retain the 37.0 AP result as the best achievable number under our full pipeline, but will no longer describe it as MTLA 'nearly doubling' AP without qualification. Instead, we will state that MTLA at N=1 improves AP by 57% (20.4 → 32.1) and that combining it with self-consistency voting yields 37.0 AP, with the vote-count-only baseline reported alongside to show the marginal contribution of each component. revision: yes

Circularity Check

0 steps flagged

No circularity: MTLA is a training-free post-hoc score derived from model attention maps, evaluated against external benchmarks with no fitted parameters or self-citation chains.

full rationale

The paper proposes MTLA, a training-free confidence score computed directly from decoder self-attention weights restricted to the model's own predicted regions (Eqs. 1-4). The derivation chain is straightforward: attention weights are extracted from a forward pass, masked to tokens inside the proposed region R_p, averaged across prediction tokens Q_p, heads, and a fixed layer band (L8-21). No parameters are fitted to the target benchmarks. The layer band is borrowed from SVAR [19] (an external citation by different authors) and reused without per-benchmark tuning. The self-consistency voting (N=16 rollouts) is a standard technique that enlarges the candidate pool; the paper transparently reports N=1 results (32.12 AP) separately from N=16 (37.01 AP), and ablates N in Fig. 6. The headline 'nearly doubles' framing (20.4→37.0) bundles MTLA ranking with self-consistency voting, which is a framing/attribution concern rather than circularity—the N=1 result independently supports MTLA's effectiveness. All baselines (SVAR, GLSim, ContextualLens, token log-prob) are externally defined methods by different author groups. The evaluation uses standard external benchmarks (COCO, Charades-STA, QVHighlights, AudioSet) with standard metrics (AP, AUROC, PSDS1). No step in the derivation reduces to its inputs by construction, no prediction is a renamed fit, and no self-citation chain is load-bearing for the central claim.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The method introduces no new entities, particles, or forces. It operates entirely on existing model components (attention weights, predicted regions). The free parameters are minimal and mostly borrowed from prior work or determined by sweeping. The axioms are standard domain assumptions about the interpretability of attention weights.

free parameters (3)
  • Layer band L = L8-21 (for Qwen3-VL and Gemma-4)
    The set of transformer layers over which attention is averaged. Chosen following prior work (SVAR) and fixed across benchmarks, not tuned per task. Audio Flamingo 3 uses all layers (L0-27).
  • Audio mask dilation = ±2 tokens (±80ms)
    Dilation of the inside-region token mask for audio temporal localization. Swept from 0 to ±200ms to maximize PSDS1, moving the metric by <0.002. Used only for audio localization accuracy results.
  • Self-consistency rollout count N = 16
    Number of stochastic rollouts generated per input for re-ranking. Reported as the headline setting, with scaling analysis provided in Supp. E.
axioms (3)
  • domain assumption Decoder self-attention weights from prediction tokens onto input modality tokens accurately reflect the model's evidence usage for grounding.
    The entire method relies on reading these attention weights as a proxy for grounding quality. This is assumed, not proven from first principles.
  • domain assumption Grounded predictions attend more strongly to modality evidence within their proposed region than hallucinated predictions.
    This is the core empirical observation motivating MTLA (Sec. 1, Sec. 3.2). The paper provides visual evidence (Fig. 3) and empirical validation but does not derive it theoretically.
  • domain assumption Middle transformer layers (e.g., L8-21 for Qwen3-VL) are where grounding-relevant attention is concentrated.
    Borrowed from prior work (SVAR [19]) and used to select the layer band. The ablation in Table 7 shows the band is beneficial but not strictly necessary (full layers also work).

pith-pipeline@v1.1.0-glm · 21837 in / 2590 out tokens · 787206 ms · 2026-07-08T19:27:48.936857+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention." pith.science (2026). https://pith.science/paper/564EZXKG

@misc{pith2026260705978,
  author       = {Pith},
  title        = {Pith review of: Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/564EZXKG}},
  note         = {Machine review of arXiv:2607.05978}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prolifically. The model's own token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate tokens become near-deterministic once the model commits. We propose Multi-Token Localized Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim. Prior attention-based detectors, which sum attention over the entire input modality and read a single response token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in video and audio. Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.

Figures

Figures reproduced from arXiv: 2607.05978 by Avi Ben Cohen, Daniel Shalam, Emanuel Ben Baruch, Tal Remez.

Figure 1
Figure 1. Figure 1: MTLA enables reliable localization from MLLMs. Top: MLLM detection outputs contain many hallucinated predic￾tions (red); MTLA uses the model’s own attention to estimate lo￾calization confidence and suppress them, retaining grounded pre￾dictions (blue). Bottom: applied post hoc, MTLA (N=16 self￾consistency) lifts each base MLLM’s localization accuracy across image, video, and audio, narrowing the gap to sup… view at source ↗
Figure 2
Figure 2. Figure 2: Multi-Token Localized Attention (MTLA). A pre-trained MLLM localizes objects, emitting per prediction a token sequence of box coordinates and a class label. We read the decoder’s attention from these prediction tokens Qp onto the input, then restrict it to the patches inside the model’s own proposed region Rp. The training-free score s(p) is the mean over prediction tokens of the attention mass inside Rp: … view at source ↗
Figure 3
Figure 3. Figure 3: Grounding attention is a multi-token signal. Each column shows where the tokens of one of the coordinates (x1, y1, x2, y2) or the label attend; the box marks the proposal region Rp, and the mean column (right of the dashed rule) averages them. Any single token gives a partial, noisy view, but jointly the tokens of a grounded prediction (zebra, bird) concentrate their attention inside Rp, whereas those of a… view at source ↗
Figure 4
Figure 4. Figure 4: Per-prediction score distributions on COCO val2017 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: shows that MTLA substantially improves hallu￾cination detection on both video benchmarks. On Charades￾STA, MTLA improves over SVAR from 51.1 to 66.8 AU￾ROC; on QVHighlights, the gain is even larger, from 41.5 to 80.0. This gap indicates that global attention becomes unreliable on longer videos. MTLA recovers a more local￾ized and discriminative signal. Token-confidence baselines remain weak (52.7–59.5 AURO… view at source ↗
Figure 6
Figure 6. Figure 6: Self-consistency scaling across modalities. (a) COCO Detection AP (Qwen3-VL-8B). (b) Charades-STA R@1@0.5 and (c) QVHighlights AP (Qwen3-VL-8B) as a function of the number of rollouts N, comparing MTLA (blue) vs. SVAR (orange) selection. summed MTLA (resp. SVAR) scores of its cluster. Most of the gain from additional rollouts is realized by N=5 (32.1 → 36.1 AP), after which MTLA continues to im￾prove more … view at source ↗
Figure 7
Figure 7. Figure 7: Per-token attention on additional COCO predictions (Qwen3-VL, zero-shot). Top rows are grounded predictions, bottom rows hallucinations; columns are the coordinate sub-tokens, the label, and their mean. Grounded predictions concentrate attention inside the proposed box, whereas hallucinations spread it across the scene. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 6 internal anchors

  1. [1]

    Qwen2.5-VL Technical Report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-VL technical repor...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. InECCV, 2020. 1, 5, 6

    work page 2020

  3. [3]

    Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal LLM’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    BEATs: Audio pre-training with acoustic tok- enizers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tok- enizers. InInternational Conference on Machine Learning (ICML), 2023. 7

    work page 2023

  5. [5]

    Microsoft COCO Captions: Data Collection and Evaluation Server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  6. [6]

    Lookback lens: De- tecting and mitigating contextual hallucinations in large lan- guage models using only attention maps

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Kr- ishna, Yoon Kim, and James Glass. Lookback lens: De- tecting and mitigating contextual hallucinations in large lan- guage models using only attention maps. InProceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), 2024. 2, 3

    work page 2024

  7. [7]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R´e. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InNeurIPS, 2022. 12

    work page 2022

  8. [8]

    coco-gemini: Zero-shot COCO detec- tion with Gemini.https://github.com/simedw/ coco-gemini, 2025

    Simon Edwardsson. coco-gemini: Zero-shot COCO detec- tion with Gemini.https://github.com/simedw/ coco-gemini, 2025. 6

    work page 2025

  9. [9]

    Multi-modal hallucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In CVPR, 2024. 3

    work page 2024

  10. [10]

    TALL: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: Temporal activity localization via language query. In ICCV, 2017. 5

    work page 2017

  11. [11]

    Gemma 3 Technical Report

    Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025. 1, 3, 5

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In2017 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 2017. 5, 7

    work page 2017

  13. [13]

    Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models

    Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ra- mani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.arXiv preprint arXiv:2507.08128, 2025. 1, 3, 5, 7, 11

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    DAMRO: Dive into the attention mechanism of LVLM to re- duce object hallucination

    Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. DAMRO: Dive into the attention mechanism of LVLM to re- duce object hallucination. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Process- ing (EMNLP), 2024. 3

    work page 2024

  15. [15]

    Making the V in VQA matter: El- evating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: El- evating the role of image understanding in visual question answering. InCVPR, 2017. 2

    work page 2017

  16. [16]

    Shawn Hershey, Daniel P. W. Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R. Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classi- fication. In2021 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), 2021. 5, 7

    work page 2021

  17. [17]

    OPERA: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. InCVPR, 2024. 3

    work page 2024

  18. [18]

    Interpreting and editing vision-language rep- resentations to mitigate hallucinations

    Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language rep- resentations to mitigate hallucinations. InICLR, 2025. 6, 12, 14

    work page 2025

  19. [19]

    Devils in middle layers of large vision- language models: Interpreting, detecting and mitigating ob- ject hallucinations via attention lens

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision- language models: Interpreting, detecting and mitigating ob- ject hallucinations via attention lens. InCVPR, 2025. 2, 3, 4, 6, 7, 8, 13, 14

    work page 2025

  20. [20]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations.IJCV, 2017. 1

    work page 2017

  21. [21]

    Berg, and Mohit Bansal

    Jie Lei, Tamara L. Berg, and Mohit Bansal. QVHighlights: Detecting moments and highlights in videos via natural lan- guage queries. InNeurIPS, 2021. 5, 7

    work page 2021

  22. [22]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. InCVPR, 2024. 2, 3

    work page 2024

  23. [23]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022. 1

    work page 2022

  24. [24]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), 2023. 3

    work page 2023

  25. [25]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 5, 6 9

    work page 2014

  26. [26]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In ECCV, 2024. 1

    work page 2024

  27. [27]

    Paying more atten- tion to image: A training-free method for alleviating halluci- nation in LVLMs

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying more atten- tion to image: A training-free method for alleviating halluci- nation in LVLMs. InECCV, 2024. 3

    work page 2024

  28. [28]

    MMBench: Is your multi-modal model an all-around player? InECCV, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? InECCV, 2024. 2

    work page 2024

  29. [29]

    Simple open-vocabulary object detection with vi- sion transformers

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection with vi- sion transformers. InECCV, 2022. 1

    work page 2022

  30. [30]

    Query-Dependent Video Represen- tation for Moment Retrieval and Highlight Detection

    WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-Dependent Video Represen- tation for Moment Retrieval and Highlight Detection. In CVPR, 2023. 7

    work page 2023

  31. [31]

    Verjans, Phi Le Nguyen, and Vu Minh Hieu Phan

    Tuan Dung Nguyen, Minh Khoi Ho, Qi Chen, Yutong Xie, Cam-Tu Nguyen, Minh Khoi Nguyen, Dang Huy Pham Nguyen, Anton van den Hengel, Johan W. Verjans, Phi Le Nguyen, and Vu Minh Hieu Phan. Beyond the global scores: Fine-grained token grounding as a robust detector of LVLM hallucinations. InCVPR, 2026. 2, 3

    work page 2026

  32. [32]

    GLSim: Detecting ob- ject hallucinations in LVLMs via global-local similarity

    Seongheon Park and Sharon Li. GLSim: Detecting ob- ject hallucinations in LVLMs via global-local similarity. In NeurIPS, 2025. 3, 6, 13

    work page 2025

  33. [33]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. InICLR,

    work page

  34. [34]

    Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in VLMs

    Anirudh Phukan, Divyansh, Harshit Kumar Morj, Vaishnavi, Apoorv Saxena, and Koustava Goswami. Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in VLMs. InProceedings of the 2025 Conference of the North American Chapter of the Association for Com- putational Linguistics (NAACL), 2025. 3, 6, 13

    work page 2025

  35. [35]

    Effective pre- training of audio transformers for sound event detection

    Florian Schmid, Tobias Morocutti, Francesco Foscarin, Jan Schl¨uter, Paul Primus, and Gerhard Widmer. Effective pre- training of audio transformers for sound event detection. In ICASSP 2025 - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025. 7

    work page 2025

  36. [36]

    InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expres- sions. InECCV, 2016. 1 10 Supplementary Material A. Implementation Details A.1. Models and decoding We evaluate three MLLMs.Qwen3-VL-8B-Instruct (L=36layers,H=32heads, full attention) is evalu- ated with the middle-layer bandl∈[8,21];Gemma-...

    work page 2016

  38. [38]

    bbox_2d": [x1,y1,x2,y2],

    split numbers into single-digit tokens, so a single coor- dinate value such as421occupies three token positions; this is why each coordinate contributes several tokens to the prediction-token setQ p. We note that digit-by-digit tok- enization is common but not universal across LLMs (e.g., some BPE tokenizers group runs of digits), so the exact number of t...

    work page