arxiv: 2605.10815 · v2 · submitted 2026-05-11 · 💻 cs.AI · eess.AS

Recognition: no theorem link

Probing Cross-modal Information Hubs in Audio-Visual LLMs

Jihoo Jung , Chaeyoung Jung , Ji-Hoon Kim , Joon Son Chung

Authors on Pith no claims yet

Pith reviewed 2026-05-13 02:59 UTC · model grok-4.3

classification 💻 cs.AI eess.AS

keywords audio-visual LLMssink tokenscross-modal informationhallucination mitigationmultimodal modelsinformation flowprobing LLMs

0 comments

The pith

Audio-visual LLMs encode integrated cross-modal information primarily in a subset of sink tokens.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how audio-visual large language models combine and store data from audio and video inputs. It shows that this integrated information concentrates in sink tokens, but only a specific group among them handles the cross-modal fusion. This concentration reveals an internal structure for multimodal reasoning that was not previously mapped. Because the pattern holds across several models, it points to a practical way to guide model outputs toward more reliable integrated knowledge. The authors demonstrate this by introducing a training-free adjustment that boosts reliance on the key tokens to reduce hallucinations.

Core claim

Through an analysis of multiple recent AVLLMs, we uncover two common findings. First, AVLLMs primarily encode integrated audio-visual information in sink tokens. Second, sink tokens do not uniformly hold cross-modal information. Instead, a distinct subset of sink tokens, which we term cross-modal sink tokens, specializes in storing such information. Based on these findings, we further propose a simple training-free hallucination mitigation method by encouraging reliance on integrated cross-modal information within cross-modal sink tokens.

What carries the argument

Cross-modal sink tokens: the distinct subset of sink tokens that specializes in storing and serving as the hub for integrated audio-visual information.

If this is right

A simple training-free method can reduce hallucinations by directing the model to rely more on the cross-modal sink tokens.
The same pattern of specialization appears across the different AVLLM architectures examined.
Targeting these tokens provides a direct handle on cross-modal information flow without changing model weights.
Not all sink tokens perform equivalent roles in multimodal integration.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The emergence of specialized cross-modal hubs may reflect a general organizational tendency in multimodal models when fusing sensory streams.
Probing techniques similar to those used here could locate analogous integration points in other multimodal LLMs.
If the tokens remain stable across inputs, they could become reliable sites for monitoring or editing fused knowledge in deployed systems.

Load-bearing premise

The concentration of cross-modal information in a particular subset of sink tokens is representative across AVLLMs and that intervening on these tokens will reduce hallucinations without introducing new errors.

What would settle it

Analysis of additional AVLLMs showing cross-modal information distributed evenly across all sink tokens rather than concentrated in a subset, or the hallucination-mitigation intervention producing no consistent improvement or new errors when applied to the identified tokens.

Figures

Figures reproduced from arXiv: 2605.10815 by Chaeyoung Jung, Jihoo Jung, Ji-Hoon Kim, Joon Son Chung.

**Figure 1.** Figure 1: Cross-modal information is primarily stored in crossmodal sink tokens. Consider an audiovisual clip of a barking sea lion. Cross-modal sink tokens aggregate cues from both modalities, whereas unimodal sink tokens encode information solely from their native modality. diverse modalities (Yu et al., 2024; Weng et al., 2024; Huang et al., 2024). Among these, audio-visual LLMs (AVLLMs), which integrate auditor… view at source ↗

**Figure 2.** Figure 2: Causal Tracing under the Unimodal Dominance Framework. For the audio-dominant case, the corrupt run is constructed by corrupting the audio modality, and restoration is conducted by patching hidden states of video tokens from the clean run. Conversely, for the video-dominant case, the video modality is corrupted, and hidden states of audio tokens are patched from the clean run. We expect patching the non-do… view at source ↗

**Figure 3.** Figure 3: visualizes MDS values for audio and video sink tokens for a representative example. This visualization reveals that sink tokens may diverge into two groups: some receive incoming attention primarily from their own modality, while others from the other modality. See Appendix B.1.6 for further analysis on MDS [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 5.** Figure 5: Averaged attention mass to cross-modal and unimodal sink tokens across 70 genuine and 70 hallucinated samples. Genuine object maintain dominant attention on cross-modal sinks across all layers. Conversely, hallucinated object reveal a significant surge in attention to unimodal sinks, occasionally surpassing that of cross-modal sinks. Delving deeper into these hallucinations, we investigated the model’s at… view at source ↗

**Figure 4.** Figure 4: Example of object hallucination in AVLLMs. While the video modality correctly recognizes the object as a zebra, the audio modality misinterprets the zebra’s braying as a dog’s bark, causing the hallucinated object dog to appear in the caption. monized with the correct modality. Failure to fully suppress the erroneous modality leads to a leakage of incorrect semantic cues, resulting in the model generating… view at source ↗

**Figure 6.** Figure 6: Parameter sensitivity of α with CHAIR metrics. lucinations. Notably, our improvements are maximized on the VGGSound-Animal benchmark-where AVLLM-specific hallucinations induced by audio-visual disagreement are most prevalent-validating our targeted approach. Crucially, ASD maintains this superiority on the more general VGGSound-All and AudioSet datasets, underscoring its robustness. Ablation study on α … view at source ↗

**Figure 7.** Figure 7: Layer-wise average number of sink tokens across 100 samples in Qwen2.5-Omni (7B). The shaded region denotes one standard deviation. The number of sink tokens vary substantially across layers. Motivated by this observation, we introduce a global sink token definition that accounts for cross-layer variability. Instead of using layer-wise sink sets Iˆl , we measure how frequently each token is classified as … view at source ↗

**Figure 8.** Figure 8: Dimension-wise RMSNorm magnitudes for BOS, sink, and non-sink tokens. Top: Qwen-based models. Bottom: SALMONNbased models. For video-SALMONN-o1, which is based on Qwen2-7B, we directly adopt the sink-dimension set Dsink = {458, 2570} reported in (Kang et al., 2025) [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

**Figure 9.** Figure 9: Comparison of attention weights on object tokens during the generation of genuine versus hallucinated objects. Unlike the unimodal sink, object tokens do not show a significant disparity between the two scenarios. Ground-truth object sets. Following (Petryk et al., 2024), all video clips are truncated to a maximum duration of 10 seconds. We uniformly sample 10 frames per video and perform object detection … view at source ↗

**Figure 10.** Figure 10: Layer-wise causal patching results for the audio-dominant setting on Qwen2.5-Omni (7B). We patch all tokens from the non-dominant modality using a sliding window of 10 consecutive layers. cross-modal information exchange is most pronounced in the middle layers, indicating that these layers act as primary integration hubs. B.2.5. LATENCY OVERHEAD OF ASD Although ASD proves effective, it comes at the cost o… view at source ↗

**Figure 11.** Figure 11: Qualitative results on AV conflict cases (2/2). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗

**Figure 11.** Figure 11: Qualitative results of our method. Text highlighted in red indicates hallucinated objects, while unhighlighted text denotes genuine objects. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗

**Figure 11.** Figure 11: Qualitative results of our method. Continued. Text highlighted in red indicates hallucinated objects, while unhighlighted text denotes genuine objects. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗

read the original abstract

Audio-visual large language models (AVLLMs) have recently emerged as a powerful architecture capable of jointly reasoning over audio, visual, and textual modalities. In AVLLMs, the bidirectional interaction between audio and video modalities introduces intricate processing dynamics, necessitating a deeper understanding of their internal mechanisms. However, unlike extensively studied text-only or large vision language models, the internal workings of AVLLMs remain largely unexplored. In this paper, we focus on cross-modal information flow between audio and visual modalities in AVLLMs, investigating where information derived from one modality is encoded within the token representations of the other modality. Through an analysis of multiple recent AVLLMs, we uncover two common findings. First, AVLLMs primarily encode integrated audio-visual information in sink tokens. Second, sink tokens do not uniformly hold cross-modal information. Instead, a distinct subset of sink tokens, which we term cross-modal sink tokens, specializes in storing such information. Based on these findings, we further propose a simple training-free hallucination mitigation method by encouraging reliance on integrated cross-modal information within cross-modal sink tokens. Our code is available at https://github.com/kaistmm/crossmodal-hub.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper analyzes internal mechanisms of audio-visual large language models (AVLLMs), claiming that integrated cross-modal information is primarily encoded in sink tokens rather than uniformly across representations. It identifies a specialized subset of these, termed cross-modal sink tokens, that store audio-visual information, and proposes a simple training-free intervention to mitigate hallucinations by encouraging reliance on these tokens. The claims are supported by empirical analysis across multiple recent AVLLMs.

Significance. If the findings hold, this provides new mechanistic insight into how AVLLMs integrate audio and visual modalities, an area less explored than text-only or vision-language models. The identification of cross-modal sink tokens and the associated training-free hallucination mitigation method could offer a practical, low-cost way to improve reliability in multimodal systems, with potential for broader application if the patterns generalize.

major comments (2)

[Methods] Methods section: The analysis of sink tokens and cross-modal specialization lacks explicit controls for confounding factors such as token position, sequence length, or modality-specific attention patterns. Without these, it is unclear whether the observed encoding is due to cross-modal integration or other architectural biases, undermining the two common findings.
[Experiments] Experiments on hallucination mitigation: The proposed intervention's evaluation does not sufficiently demonstrate robustness across diverse tasks or models, nor does it quantify potential degradation in other capabilities or introduction of new errors, as noted in the weakest assumption. This makes the practical utility of the method hard to assess.

minor comments (2)

[Figures] Figure captions and legends could be expanded to include exact definitions of 'sink tokens' and 'cross-modal sink tokens' for clarity.
[Introduction] The abstract mentions 'multiple recent AVLLMs' but the specific models and selection criteria should be listed earlier in the introduction for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive feedback on our paper. We address the major comments point by point below, and indicate where revisions will be made to the manuscript.

read point-by-point responses

Referee: [Methods] Methods section: The analysis of sink tokens and cross-modal specialization lacks explicit controls for confounding factors such as token position, sequence length, or modality-specific attention patterns. Without these, it is unclear whether the observed encoding is due to cross-modal integration or other architectural biases, undermining the two common findings.

Authors: We thank the referee for pointing this out. Our analyses were conducted across multiple AVLLMs to demonstrate consistency, which helps control for model-specific biases. However, we agree that explicit controls for token position and sequence length would strengthen the claims. In the revision, we will include additional experiments that ablate these factors, such as position-normalized attention maps and fixed-length sequence comparisons, to better isolate cross-modal effects. revision: yes
Referee: [Experiments] Experiments on hallucination mitigation: The proposed intervention's evaluation does not sufficiently demonstrate robustness across diverse tasks or models, nor does it quantify potential degradation in other capabilities or introduction of new errors, as noted in the weakest assumption. This makes the practical utility of the method hard to assess.

Authors: We acknowledge the need for more comprehensive evaluation. The original experiments focused on key hallucination benchmarks in AVLLMs, but to address this, we will expand the evaluation to additional tasks and models in the revised manuscript. We will also report metrics on general capability preservation and any introduced errors to better assess the method's utility and trade-offs. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical observational claims with no derivations or self-referential reductions

full rationale

The paper reports observational findings from analyzing multiple AVLLMs (sink tokens encoding integrated audio-visual information, with a specialized subset termed cross-modal sink tokens) and proposes a training-free intervention based on those observations. No equations, fitted parameters, uniqueness theorems, or self-citations are invoked as load-bearing steps in any derivation chain. The claims reduce directly to the empirical patterns observed across tested models rather than to any input by construction, making the analysis self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is an empirical probing study with no explicit mathematical axioms, free parameters, or newly postulated physical entities; the term 'cross-modal sink tokens' is introduced as a descriptive label for an observed subset rather than an invented theoretical construct.

pith-pipeline@v0.9.0 · 5513 in / 1098 out tokens · 89844 ms · 2026-05-13T02:59:37.197858+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · 3 internal anchors

[1]

Some modalities are more equal than others: Decoding and architecting multi- modal integration in mllms.arXiv preprint arXiv:2511.22826,

Chen, T., Chakka, C., Akula, A. R., Thomas, X., and Ghadi- yaram, D. Some modalities are more equal than oth- ers: Decoding and architecting multimodal integration in mllms.arXiv:2511.22826,

work page arXiv
[2]

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

Cheng, Z., Leng, S., Zhang, H., Xin, Y ., Li, X., Chen, G., Zhu, Y ., Zhang, W., Luo, Z., Zhao, D., et al. Vide- oLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv:2406.07476,

work page internal anchor Pith review Pith/arXiv arXiv
[3]

Bridg- ing ears and eyes: Analyzing audio and visual large lan- guage models to humans in visible sound recognition and reducing their sensory gap via cross-modal distillation

Jiang, X., Wu, J., Choudhari, V ., and Mesgarani, N. Bridg- ing ears and eyes: Analyzing audio and visual large lan- guage models to humans in visible sound recognition and reducing their sensory gap via cross-modal distillation. arXiv:2505.06803,

work page arXiv
[4]

Jung, C., Jang, Y ., Choi, J., and Chung, J. S. Fork- merge decoding: Enhancing multimodal understanding in audio-visual large language models.arXiv preprint arXiv:2505.20873,

work page arXiv
[5]

arXiv preprint arXiv:2306.09093 , year=

Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z., Shi, S., and Tu, Z. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv:2306.09093,

work page arXiv
[6]

On the au- dio hallucinations in large audio-video language models

Nishimura, T., Nakada, S., and Kondo, M. On the au- dio hallucinations in large audio-video language models. arXiv:2401.09774,

work page arXiv
[7]

video-salmonn 2: Caption-enhanced audio-visual large language models.arXiv preprint arXiv:2506.15220, 2025

Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. InProc. COLM, 2024b. Tang, C., Li, Y ., Yang, Y ., Zhuang, J., Sun, G., Li, W., Ma, Z., and Zhang, C. video-SALMONN 2: Captioning-enhanced audio-visual large language models. arXiv:2506.15220,

work page arXiv
[8]

LLaMA: Open and Efficient Foundation Language Models

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models.arXiv:2302.13971,

work page internal anchor Pith review Pith/arXiv arXiv
[9]

Causal mediation analysis for in- terpreting neural nlp: The case of gender bias.arXiv preprint arXiv:2004.12265, 2020

Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Sakenis, S., Huang, J., Singer, Y ., and Shieber, S. Causal mediation analysis for interpreting neural nlp: The case of gender bias.arXiv:2004.12265,

work page arXiv 2004
[10]

Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y ., Dang, K., et al. Qwen2. 5-omni techni- cal report.arXiv:2503.20215, 2025a. Xu, J., Guo, Z., Hu, H., Chu, Y ., Wang, X., He, J., Wang, Y ., Shi, X., He, T., Zhu, X., Lv, Y ., Wang, Y ., Guo, D., Wang, H., Ma, L., Zhang, P., Zhang, X., Hao, H., Guo, Z., Yang, B., Zhang, B., Ma, ...

work page internal anchor Pith review Pith/arXiv arXiv
[11]

The structure is organized as follows: A

11 Probing Cross-modal Information Hubs in Audio-Visual LLMs Appendix Outline The appendix provides detailed implementations, additional analyses, and qualitative examples supporting the main paper. The structure is organized as follows: A. Implementation Details •A.1. Sink Tokens –A.1.1. Definition of Sink Tokens –A.1.2. Selection of Sink Dimension •A.2....

work page 2025
[12]

Specifically, visual sink tokens in VLMs exhibit massive activation along the same dimensions as the BOS token in the base LLM

reported that, even after multimodal fine-tuning, the dimensions exhibiting massive activations remain largely consistent with those of the base LLM. Specifically, visual sink tokens in VLMs exhibit massive activation along the same dimensions as the BOS token in the base LLM. Following this observation, we select sink dimensions based on BOS-token activa...

work page 2025
[13]

While the original VCD applies noise solely to the image modality, we extend this approach to the audio-visual domain

reduces statistical priors and hallucinations by contrasting the original output logits with logits derived from distorted visual inputs. While the original VCD applies noise solely to the image modality, we extend this approach to the audio-visual domain. Specifically, we apply noise to both audio and video inputs to generate the distorted logits for con...

work page 2022
[14]

Consistent with the main results, we observe that cross-modal information is predominantly concentrated in sink tokens, with cross-modal sink tokens exhibiting the strongest effects. B.1.4. LAYERWISEANALYSIS Fig. 10 shows layer-wise causal patching results for the audio-dominant setting on Qwen2.5-Omni (7B), where we patch all tokens from the non-dominant...

work page 2025
[15]

Table 12.Distribution statistics of the Modality Dominance Score (MDS) for video and audio sink tokens, computed over 100 samples across five backbones. Metric Modality Qwen2.5-Omni(7B) Qwen2.5-Omni(3B) video-SALMONN-o1(7B) video-SALMONN2+(7B) video-SALMONN2+(3B) Median Video 0.45 0.49 0.56 0.84 0.79 Audio−0.49−0.50−0.77−0.18−0.59 IQR Video 0.34 0.34 0.39...

work page 2025
[16]

As shown in Tab

and FMD (Jung et al., 2025). As shown in Tab. 16, ASD consistently achieves the strongest overall performance across the three datasets, attaining the bestC s andC i scores on every benchmark and the highest ALOHa on the two VGGSound subsets. B.2.5. LATENCYOVERHEAD OFASD Although ASD proves effective, it comes at the cost of increased inference latency (3...

work page 2025