Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning

Chenxing Li; Dong Yu; Le Xu; Ruibo Fu; Shan Yang; Yong Ren; Yu Gu; Yujie Chen

REVIEW 5 major objections 4 minor 35 references

Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-guided audio captioning can withstand mismatched video by gating on attention entropy, with a roughly 6x speedup.

desk verdict Sensible incremental method for audiovisual mismatch in audio captioning, but the gating mechanism's core premise isn't directly validated and the evaluation is thin. read the letter →

arxiv 2505.22045 v1 pith:RKFTRFEZ submitted 2025-05-28 cs.MM cs.CVcs.SDeess.AS

Le Xu , Chenxing Li , Yong Ren , Yujie Chen , Yu Gu , Ruibo Fu , Shan Yang , Dong Yu This is my paper

classification cs.MMcs.CVcs.SDeess.AS

keywords audiocaptioningaudiovisualmismatchentropy-awaregatingcross-modalfusionstochasticmodalityshufflingdataaugmentationCapsmultimodalrobustness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-guided audio captioning assumes that what you see matches what you hear, but real videos often violate this through dubbing, off-screen sounds, or stock footage. The paper argues that this mismatch can be handled at fusion time by reading the entropy of the cross-attention distribution between audio and visual tokens: when attention is spread widely over visual candidates, a learned sigmoid gate lowers the visual weight and lets audio carry the caption. It also introduces Stochastic Modality Shuffling, which randomly swaps video tracks within a training batch at a small probability, so the model learns not to over-rely on visual evidence. On the AudioCaps benchmark the combined system beats the AVCap baseline on most metrics, degrades far less under test-time shuffles, and cuts inference time on long video inputs by roughly a factor of six.

What carries the argument

The load-bearing object is the entropy-aware gate placed inside cross-attention fusion. Given the softmax attention weights $P_{att} = \mathrm{softmax}(QK^\top/\sqrt{d})$ from an audio query to visual tokens, the gate normalizes the entropy of that distribution and passes it through a sigmoid, producing a scalar in $[0,1]$ that weights the visual contribution. The second component, Stochastic Modality Shuffling, generates synthetic mismatched pairs by applying a random permutation matrix to the visual features within a training batch, so the model sees deliberately wrong audio-video pairings. Together they carry the argument: one mechanism adaptively suppresses bad visual cues at inference, and the other makes the representation robust when such cues appear.

What would settle it

Take a set of matched audio-video clips, corrupt the video track in controlled ways (swap frames with a different video, add noise, or use random frames), and compare the gate's output to the true alignment label. If the gate does not systematically close when the video is wrong, or if a fixed non-entropy gate performs equally under these corruptions, the entropy-reliability premise fails.

Watch

Extended reading notes

Core claim

EVACap's central claim is that cross-attention entropy is a usable, self-supervised measure of whether visual information is trustworthy for audio captioning. In each cross-attention layer the audio query attends to visual key-value pairs, and the paper computes the entropy $E_{att}$ of the normalized attention weights over the visual sequence. A learnable gate $g = \sigma(w_g E_{att} + b_g)$ then blends the audio feature $A$ with the attended visual feature $F$ through $F = (1-g)A + gF$, suppressing vision when attention is dispersed and keeping it when attention is concentrated. Paired with Stochastic Modality Shuffling, a batch-wise random permutation of visual features applied at a low probability during training, the model becomes resilient to audiovisual mismatch without extra supervision. The AudioCaps experiments are designed to show that this combination outperforms the AVCap baseline, retains more performance under shuffled test conditions, and runs much faster because irrelevant visual processing is effectively gated off.

Load-bearing premise

The paper assumes that high entropy in the cross-attention distribution reliably marks misleading visual input, so the learned gate can use it to decide when to suppress vision; this connection is asserted rather than independently validated.

Editorial extensions

If this is right

Attention entropy acts as a cheap, label-free reliability signal for cross-modal fusion, so no supervision or external alignment model is needed to decide when vision should be ignored.
Training with a small mismatch probability (5%) improves n-gram captioning metrics while preserving most semantic scores, whereas large probabilities (50–100%) degrade quality.
Under test-time shuffling, the proposed model keeps SPICE at 98.8% of its aligned performance, versus 92.4% for AVCap, showing the augmentation transfers to unseen mismatch levels.
Long-video inference drops from about 15.0 to 2.7 seconds at 18 frames, a roughly 6x speedup that makes the approach more practical for real-time captioning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence the paper leaves implicit: the same gate could serve as an explainability tool, flagging video segments the model distrusts, which may correspond to dubbing errors or off-screen audio in real footage.
Because shuffle probability is a sensitive hyperparameter, an adaptive curriculum that gradually raises mismatch exposure during training could yield stronger robustness than the fixed 5% setting, and would be a straightforward extension of the paper's method.
The speedup suggests a more aggressive design: when the gate closes, the model could skip computing visual tokens entirely rather than merely reweighting them, which would extend the gains to even longer videos.
If attention entropy reliably tracks modality mismatch, the same gating scheme could transfer to other alignment-sensitive multimodal tasks, such as video question answering or audio-visual speech recognition.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The concrete new thing here is the combination of an attention-entropy gate with a batch-wise modality-shuffling augmentation for visually guided audio captioning. That combination is not in the prior work they cite, and the robustness evaluation is the right idea: they actually shuffle audio-video pairs at test time and show the full model degrades less than the AVCap baseline on most metrics. That is a legitimate, if modest, result. The shuffle augmentation is a simple data trick, but it seems to work as a regularizer. I also appreciate that they are explicit about the missing VACT baseline and use the audio-only model as a sanity floor.

The soft spots are real, though. First, the gating mechanism's central claim — that high attention entropy signals misleading visual cues and the gate suppresses them — is never directly tested. Table 3 compares the full proposed system against AVCap; it does not isolate the gate under mismatched inputs. Table 4, which does ablate the gate, is run only on the aligned test set, and there removing the gate actually improves SPICE and ROUGE-L. So the paper never shows that entropy gating is what drives the robustness under mismatch. That is a load-bearing gap, not a minor quibble. Second, Eq. (2) is sloppy: the sum index and the normalization over alpha are not coherently defined, which matters because the gate is a monotone function of that scalar. The equation needs fixing before the method can be reproduced. Third, the inference speed claim is under-explained. A 5.5x gap over AVCap at 18 frames is surprising, and the paper does not say why AVCap is so slow or what implementation choices produce the speedup. The effect could be real but it is not accounted for.

On the evidence side: one baseline, no significance tests, no variance, and the headline gains are small (BLEU4 0.258 vs 0.240). That is honest progress within a narrow subfield, and I do not think the method is wrong — I think it is under-validated. The authors clearly know the relevant literature and the experiments are organized around the right question.

Who should read this? Researchers working on audio captioning or multimodal robustness might find the test-time shuffling protocol useful. I would bring it to a reading group as a case study in how to (and how not to) validate a gating mechanism. It deserves peer review, but a referee should push hard for a mismatch-condition ablation of the gate and a corrected Eq. (2).

Referee Report

5 major / 4 minor

Summary. The paper proposes EVACap, an audio captioning system that augments vision-guided fusion with an entropy-aware gate (Section 2.2, Eqs. 2-3) and a Stochastic Modality Shuffling (SMS) training augmentation (Section 2.3). The gate is intended to suppress misleading visual cues when audio and video are misaligned, and SMS simulates mismatched training pairs. Experiments on AudioCaps report favorable or comparable results against a single baseline (AVCap) on aligned and shuffled test sets, plus an inference-time comparison claiming roughly 6x speedup. The core claims are that the gate and SMS improve robustness under audiovisual mismatch and that the method is efficient.

Significance. If substantiated, the work addresses a real and under-studied problem: audiovisual mismatch in video-guided audio captioning. The proposed entropy-based gating is a lightweight, supervision-free idea, and the test-time shuffling evaluation is a useful protocol for robustness assessment. The paper also deserves credit for reporting inference-time measurements and for acknowledging the unavailability of a competing baseline's weights. However, the evidence currently falls short of supporting the central mechanism: the gate's benefit under mismatch is never isolated, Eq. (2) contains an undefined normalization, and the single-baseline comparison limits the generality of the stated claims.

major comments (5)

[Section 2.2, Eq. (2) and Eq. (3)] Equation (2) is internally inconsistent: the attention distribution P_att is defined over visual key-value pairs of length T_v, but the entropy sum runs over i = 1 to T_a, and the normalization constant alpha in R^N is not defined. If alpha is a vector, log alpha is undefined; if alpha is a scalar, its value and upper bound are unspecified, and log alpha can be negative for alpha < 1, which would flip the sign of the entropy. Moreover, the variable A in Eq. (3) is not defined; it presumably denotes the audio-side feature, but this is never stated. These ambiguities make the gate's input signal, E_att, ill-defined and prevent the reader from verifying the claimed entropy-relevance mapping.
[Section 3.4, Table 4; Section 3.3.2, Table 3] The central claim that the entropy-aware gate suppresses misleading visual cues under mismatch is not directly tested. Table 3 compares the full Proposed system (gate + SMS) against AVCap under test-time shuffling, so it cannot attribute robustness to the gate. Table 4 ablates the gate only on the aligned test set, where removing the gate (w/o E) actually improves SPICE (0.175 vs. 0.170) and ROUGE-L (0.475 vs. 0.474). No experiment ablates the gate under shuffled or mismatched test inputs, which is precisely the condition the gating mechanism is designed for. Please add an ablation such as 'Proposed w/o E' under 50% and 100% test shuffling, and report the learned gate values g on matched versus mismatched inputs to validate that high attention entropy correlates with mismatch.
[Section 3.1.2 and abstract] The abstract states that the system shows 'superior performance over existing baselines,' but the experiments include only one baseline, AVCap; the VACT baseline is mentioned but not compared because its weights are unavailable. With a single baseline, claims of superiority over 'existing baselines' in general are not supported. Please either add more baselines (e.g., LAVCap, if accessible) or revise the abstract and conclusion to specify that comparisons are against AVCap only, avoiding the plural and absolute phrasing.
[Section 3.5 and Figure 2] The approximately 6x inference speedup claim is not adequately supported. The parameter counts of Proposed (235M) and AVCap (233M) are nearly identical, yet the inference time differs by a factor of about 5.5 at 18 frames (2.733s vs. 15.029s). The paper does not explain the architectural reason for this large difference, nor does it report hardware, batch size, or whether the timing includes any warm-up or framework-specific overhead. Please provide an explanation of the computational path (e.g., whether the gate reduces the number of visual tokens processed or changes the cross-attention complexity), and report speedups with standard deviations across runs.
[Section 3.2, Tables 1 and 3] No variance or significance testing is reported anywhere. Several headline differences are small (e.g., BLEU-1 0.670 vs. 0.666; SPICE 0.170 vs. 0.170 in Table 1), and robustness comparisons in Table 3 are based on single runs. Without standard deviations over multiple random seeds or bootstrap confidence intervals, the reader cannot assess whether the reported improvements and degradation differences are statistically reliable. Please add such measures or clearly state the number of runs and, if only one run was performed, temper the claims accordingly.

minor comments (4)

[Section 3.4] The text says the baseline cross-attention (w/o G) 'exhibits a decline across all metrics' compared to the full model, but Table 4 shows w/o G has ROUGE-L 0.475, which is higher than the Proposed model's 0.474. Please rephrase to avoid an inaccurate claim, e.g., 'a decline on most metrics.'
[Algorithm 1, Section 2.3] The pseudocode does not show the 'probability p_mix' that the text says controls the shuffling; the algorithm as written always shuffles the entire batch. Please clarify whether the shuffled batch is used only with probability p_mix or whether each sample is independently re-paired with probability p_mix.
[Section 2.1] There is a typo: 'same struct of ViT encoder' should be 'same structure of the ViT encoder.'
[Reference [14]] Reference [14] lists only the first author 'A. Vaswani' rather than the full citation for 'Attention is all you need'; please expand to the complete author list and publication venue.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: gate parameters are learned from captioning loss, the shuffle probability is chosen on unshuffled data, and robustness is evaluated on shuffled test conditions not used for tuning.

full rationale

The paper's central claims do not reduce to their inputs by construction. The entropy gate in Eq. (3) has trainable parameters w_g and b_g optimized with the standard captioning loss; the entropy E_att is computed from cross-attention weights, not from the target captions or from the evaluation metrics. The Stochastic Modality Shuffling probability p_mix=5% is selected by comparing models on the original unshuffled test set (Table 2), while the mismatch-robustness claim is evaluated on test-time shuffles (Table 3) that were not part of that selection. Thus the robustness result is not a fitted parameter renamed as a prediction. The approximately 6x inference speedup is an architectural timing comparison (Figure 2) independent of learned parameters. The only load-bearing external components are CAV-MAE [15] and AVCap [12], both from other research groups; references to the authors' own prior work appear only as background and are not load-bearing. The paper does contain a notational ambiguity in Eq. (2), where the entropy sums over T_a while P_att is described over visual positions, and the factor log(alpha) is not explicitly defined; this is a correctness or clarity concern, not circularity. The entropy-relevance premise (that high cross-attention entropy signals misleading visual cues) is asserted but not independently validated; this is an empirical validity concern, not a circularity concern, because the gate's effect is measured on data that was not used to fit it. Overall, the derivation chain is self-contained with respect to its inputs, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two untested behavioral assumptions (entropy equals unreliability; shuffling equals real-world mismatch) plus one hyperparameter (p_mix=5%). No new physical or conceptual entities are introduced.

free parameters (1)

shuffle probability p_mix = 0.05 (5%)
Selected as best on validation from {0, 5, 50, 100}% (Table 2); the main robustness results use this value.

assumptions (4)

standard math Softmax attention and Shannon entropy are valid for quantifying attention dispersion
Used in Eq. (1) and Eq. (2) to compute attention weights and entropy.
domain assumption High cross-attention entropy indicates unreliable or misleading visual cues
The gate in Eq. (3) suppresses visual features when entropy is high; no independent validation of this mapping is provided (Section 2.2).
domain assumption Randomly shuffling visuals within a batch approximates the distribution of real-world audiovisual mismatches
SMS assumes synthetic mismatches created by permutation are representative of dubbing or off-screen sounds (Section 2.3).
domain assumption CAV-MAE pretrained multimodal features transfer to the captioning task without additional alignment training
The model builds on the CAV-MAE foundation (Section 2.1) and assumes its contrastively aligned features are sufficient for caption generation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning." pith.science (2026). https://pith.science/paper/RKFTRFEZ

@misc{pith2026250522045,
  author       = {Pith},
  title        = {Pith review of: Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKFTRFEZ}},
  note         = {Machine review of arXiv:2505.22045}
}

read the original abstract

Current vision-guided audio captioning systems frequently fail to address audiovisual misalignment in real-world scenarios, such as dubbed content or off-screen sounds. To bridge this critical gap, we present an entropy-aware gated fusion framework that dynamically modulates visual information flow through cross-modal uncertainty quantification. Our novel approach employs attention entropy analysis in cross-attention layers to automatically identify and suppress misleading visual cues during modal fusion. Complementing this architecture, we develop a batch-wise audiovisual shuffling technique that generates synthetic mismatched training pairs, greatly enhancing model resilience against alignment noise. Evaluations on the AudioCaps benchmark demonstrate our system's superior performance over existing baselines, especially in mismatched modality scenarios. Furthermore, our solution demonstrates an approximately 6x improvement in inference speed compared to the baseline.

Figures

Figures reproduced from arXiv: 2505.22045 by the authors.

**Figure 1.** (a) Overview of the proposed EVACap Framework. (b) Detail of the Entropy-aware Gated Fusion module. that enhances model robustness. The overall architecture is based on the CAV-MAE [15] pretrained foundation, with careful adaptations tailored for caption generation. 2.1. Audiovisual modal alignment As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Inference time comparison (seconds) across different length of input video frames. 0 Input Frames denotes a baseline with no video modal inputs. timed inference runs, with the final results averaged for stability. Our proposed method demonstrates superior computational efficiency (2.369-2.733 seconds) compared to AVCap (3.416- 15.029 seconds), mainly showing more significant advantages as video context expands. The… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 22 canonical work pages

[1]

This task is applicable in ar- eas such as multimedia retrieval [2, 3], assistive technologies for the hearing-impaired [4, 5], and intelligent video analysis systems [6]

Introduction Automatic audio captioning (AAC), which aims to generate nat- ural language descriptions for unconstrained audio clips [1], has emerged as a critical technology bridging machine percep- tion and human communication. This task is applicable in ar- eas such as multimedia retrieval [2, 3], assistive technologies for the hearing-impaired [4, 5], ...

work page
[2]

An adaptive gating mechanism based on attention entropy dynamics, achieving context-aware modality weighting with- out additional supervision

work page
[3]

A practical learning paradigm through randomized intra- batch re-pairing, effectively simulating diverse mismatch scenarios

work page
[4]

First systematic investigation of audiovisual mismatch in video-assisted AAC, identifying its detrimental effects on conventional fusion approaches

work page
[5]

Video accessibility enhancement for hearing-impaired users,

R. Hong, M. Wang, X.-T. Yuan, M. Xu, J. Jiang, S. Yan, and T.- S. Chua, “Video accessibility enhancement for hearing-impaired users,”ACM Transactions on Multimedia Computing, Communi- cations, and Applications (TOMM), vol. 7, no. 1, pp. 1–19, 2011

work page 2011
[6]

NowYouSee Me: Context-Aware Automatic Audio Description

S.-H. Lee, J. Wang, D. Fan, Z. Zhang, L. Liu, X. Hao, V . Bhat, and X. Li, “Nowyousee me: Context-aware automatic audio de- scription,”arXiv preprint arXiv:2412.10002, 2024

work page Pith review arXiv 2024
[7]

Moreover, our system achieves an approximately 6x im- provement in inference speed compared to the baseline while still maintaining competitive accuracy

Comprehensive evaluations across multiple metrics demon- strate superior performance under mismatched conditions. Moreover, our system achieves an approximately 6x im- provement in inference speed compared to the baseline while still maintaining competitive accuracy

work page
[8]

Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning

Methods Our proposed framework,Entropy-awareVisual-guideAudio Captioning (calledEV ACap), addresses audiovisual mis- matches with two key innovations: (1) an entropy-guided gated fusion mechanism that dynamically controls the flow of visual information, and (2) a probabilistic data augmentation strategy arXiv:2505.22045v1 [cs.MM] 28 May 2025 Audio Encoder...

work page Pith review arXiv 2025

Show all 35 references

[9]

Experimental settings 3.1.1

Experiments 3.1. Experimental settings 3.1.1. Dataset and metrics We conduct experiments on the AudioCaps dataset [17], the most extensive audio captioning corpus containing 51,308 Table 1:Performance on the AudioCaps test set with the context of long video inputs (18 frames)....
[10]

Conclusion In this paper, we present a novel framework to address au- diovisual mismatch in video-assisted audio captioning through adaptive gating and mismatch-aware training. By leveraging cross-attention entropy as a self-supervised gate, our model dynamically suppresses ir...
[11]

Automated audio captioning: An overview of recent progress and new challenges,

X. Mei, X. Liu, M. D. Plumbley, and W. Wang, “Automated audio captioning: An overview of recent progress and new challenges,” EURASIP journal on audio, speech, and music processing, vol. 2022, no. 1, p. 26, 2022

2022
[12]

Building on this, LA VCap

concatenates raw audiovisual features, assuming inherent compatibility between modalities. Building on this, LA VCap
[13]

V ACT [11] proposes an Adap- tive Audio-Visual Attention method that integrates audio and visual information through confidence scores derived from the †Corresponding author

introduces an optimal transport-based loss function to align audiovisual modalities better. V ACT [11] proposes an Adap- tive Audio-Visual Attention method that integrates audio and visual information through confidence scores derived from the †Corresponding author. textual mo...
[14]

On met- ric learning for audio-text cross-modal retrieval,

X. Mei, X. Liu, J. Sun, M. D. Plumbley, and W. Wang, “On met- ric learning for audio-text cross-modal retrieval,”arXiv preprint arXiv:2203.15537, 2022

2022 arXiv
[15]

Separate what you describe: Language-queried audio source separation,

X. Liu, H. Liu, Q. Kong, X. Mei, J. Zhao, Q. Huang, M. D. Plumbley, and W. Wang, “Separate what you describe: Language-queried audio source separation,”arXiv preprint arXiv:2203.15147, 2022

2022 arXiv
[16]

Dynamic captioning: video accessibility enhancement for hearing impair- ment,

R. Hong, M. Wang, M. Xu, S. Yan, and T.-S. Chua, “Dynamic captioning: video accessibility enhancement for hearing impair- ment,” inProceedings of the 18th ACM international conference on Multimedia, 2010, pp. 421–430

2010
[17]

Towards diverse and efficient audio captioning via diffusion models,

M. Xu, C. Li, X. Tu, Y . Ren, R. Fu, W. Liang, and D. Yu, “Towards diverse and efficient audio captioning via diffusion models,”arXiv preprint arXiv:2409.09401, 2024

2024 arXiv
[18]

Audio captioning transformer,

X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, “Audio captioning transformer,”arXiv preprint arXiv:2107.09817, 2021

2021 arXiv
[19]

Clotho: An audio cap- tioning dataset,

K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” inICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 736–740

2020
[20]

Enhancing automated audio captioning via large language models with optimized audio encoding,

J. Liu, G. Li, J. Zhang, H. Dinkel, Y . Wang, Z. Yan, Y . Wang, and B. Wang, “Enhancing automated audio captioning via large language models with optimized audio encoding,”arXiv preprint arXiv:2406.13275, 2024

2024 arXiv
[21]

Visually-aware audio captioning with adaptive audio-visual attention,

X. Liu, Q. Huang, X. Mei, H. Liu, Q. Kong, J. Sun, S. Li, T. Ko, Y . Zhang, L. H. Tanget al., “Visually-aware audio captioning with adaptive audio-visual attention,”arXiv preprint arXiv:2210.16428, 2022

2022 arXiv
[22]

Avcap: Leveraging audio-visual fea- tures as text tokens for captioning,

J. Kim, J. Shin, and J. Kim, “Avcap: Leveraging audio-visual fea- tures as text tokens for captioning,” inInterspeech 2024, 2024, pp. 1915–1919

2024
[23]

Lavcap: Llm-based audio-visual captioning using optimal transport,

K. Rho, H. Lee, V . Iverson, and J. S. Chung, “Lavcap: Llm-based audio-visual captioning using optimal transport,”arXiv preprint arXiv:2501.09291, 2025

2025 arXiv
[24]

Attention is all you need,

A. Vaswani, “Attention is all you need,”Advances in Neural In- formation Processing Systems, 2017

2017
[25]

Contrastive audio-visual masked au- toencoder,

Y . Gong, A. Rouditchenko, A. H. Liu, D. Harwath, L. Karlinsky, H. Kuehne, and J. Glass, “Contrastive audio-visual masked au- toencoder,”arXiv preprint arXiv:2210.07839, 2022

2022 arXiv
[26]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

2010 arXiv
[27]

Audiocaps: Generat- ing captions for audios in the wild,

C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generat- ing captions for audios in the wild,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers...

2019
[28]

Audio set: An ontology and human-labeled dataset for audio events,

J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in2017 IEEE inter- national conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, ...

2017
[29]

Bleu: a method for automatic evaluation of machine translation,

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

2002
[30]

Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic eval- uation measures for machine translation and/or summarization, 2005, pp. 65–72

2005
[31]

Rouge: A package for automatic evaluation of sum- maries,

C.-Y . Lin, “Rouge: A package for automatic evaluation of sum- maries,” inText summarization branches out, 2004, pp. 74–81

2004
[32]

Cider: Consensus-based image description evaluation,

R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” inProceedings of the IEEE conference on computer vision and pattern recogni- tion, 2015, pp. 4566–4575

2015
[33]

Spice: Semantic propositional image caption evaluation,

P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” inComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. Springer, 2016, pp. 382–398

2016
[34]

Improved image captioning via policy gradient optimization of spider,

S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 873–881

2017
[35]

Microsoft coco: Common objects in context,

T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inComputer Vision–ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, ...

2014

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

This task is applicable in ar- eas such as multimedia retrieval [2, 3], assistive technologies for the hearing-impaired [4, 5], and intelligent video analysis systems [6]

Introduction Automatic audio captioning (AAC), which aims to generate nat- ural language descriptions for unconstrained audio clips [1], has emerged as a critical technology bridging machine percep- tion and human communication. This task is applicable in ar- eas such as multimedia retrieval [2, 3], assistive technologies for the hearing-impaired [4, 5], ...

work page

[2] [2]

An adaptive gating mechanism based on attention entropy dynamics, achieving context-aware modality weighting with- out additional supervision

work page

[3] [3]

A practical learning paradigm through randomized intra- batch re-pairing, effectively simulating diverse mismatch scenarios

work page

[4] [4]

First systematic investigation of audiovisual mismatch in video-assisted AAC, identifying its detrimental effects on conventional fusion approaches

work page

[5] [5]

Video accessibility enhancement for hearing-impaired users,

R. Hong, M. Wang, X.-T. Yuan, M. Xu, J. Jiang, S. Yan, and T.- S. Chua, “Video accessibility enhancement for hearing-impaired users,”ACM Transactions on Multimedia Computing, Communi- cations, and Applications (TOMM), vol. 7, no. 1, pp. 1–19, 2011

work page 2011

[6] [6]

NowYouSee Me: Context-Aware Automatic Audio Description

S.-H. Lee, J. Wang, D. Fan, Z. Zhang, L. Liu, X. Hao, V . Bhat, and X. Li, “Nowyousee me: Context-aware automatic audio de- scription,”arXiv preprint arXiv:2412.10002, 2024

work page Pith review arXiv 2024

[7] [7]

Moreover, our system achieves an approximately 6x im- provement in inference speed compared to the baseline while still maintaining competitive accuracy

Comprehensive evaluations across multiple metrics demon- strate superior performance under mismatched conditions. Moreover, our system achieves an approximately 6x im- provement in inference speed compared to the baseline while still maintaining competitive accuracy

work page

[8] [8]

Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning

Methods Our proposed framework,Entropy-awareVisual-guideAudio Captioning (calledEV ACap), addresses audiovisual mis- matches with two key innovations: (1) an entropy-guided gated fusion mechanism that dynamically controls the flow of visual information, and (2) a probabilistic data augmentation strategy arXiv:2505.22045v1 [cs.MM] 28 May 2025 Audio Encoder...

work page Pith review arXiv 2025