Pith. sign in

REVIEW 4 major objections 7 minor 41 references

Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes a chain-of-thought fine-tuning method that lets a vision-language model write audio descriptions from silent videos, replacing manual annotation in video-to-audio generation.

desk verdict A practical SVAD pipeline that likely helps VT2A inference, but the 'reasoning' claim needs a lot more evidence than the paper provides. read the letter →

arxiv 2505.13062 v3 pith:UKBLY2UB submitted 2025-05-19 cs.MM cs.SDeess.AS

Yong Ren , Chenxing Li , Le Xu , Hao Gu , Duzhen Zhang , Yujie Chen , Manjie Xu , Ruibo Fu
show 2 more authors
Shan Yang Dong Yu
This is my paper · ORCID
classification cs.MMcs.SDeess.AS
keywords audiodescriptionsilentvideovision-languagemodelmodal-mismatchreasoningchain-of-thoughtsupervisedfine-tuningvideo-to-audioCoT-AudioCaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a task called SVAD: given only a silent video, a model must infer and describe the sounds that plausibly occurred. It claims that off-the-shelf vision-language models do this poorly, because they describe what they see rather than reason about implied audio. The authors build a dataset of reasoning chains that lead from visual objects to sound events to audio captions, and fine-tune a VLM on those chains. They report that this chain-of-thought fine-tuning raises the semantic match between predicted and real audio, and that the resulting captions work as text prompts for video-to-audio generators, reducing the need for human annotation. If right, the method turns an ordinary VLM into a usable source of audio descriptions for sound synthesis.

What carries the argument

The load-bearing mechanism is CoT-based supervised fine-tuning (CoT-SFT) on a constructed dataset, CoT-AudioCaps. A reasoning chain is a three-step template: visible objects, then plausible sound events, then an audio caption. During training the VLM is shown the silent video and asked each step in sequence; during inference the outputs of earlier steps are fed into the later steps. This decomposes a hard cross-modal inference into smaller steps and is what the paper credits for the measured gains.

What would settle it

Take the AudioCaps evaluation clips, have human annotators mark each ground-truth sound as visually inferable or not, then run the CoT-SFT model on the two subsets. If the model's CLAP score on the not-inferable subset is no better than the base VLM's, while the inferable subset shows the reported gain, the claimed reasoning ability reduces to visual association rather than genuine inference.

Watch

Extended reading notes

Core claim

The central claim is that modal-mismatch reasoning can be taught to a vision-language model by supervised fine-tuning on explicit reasoning chains. On the SVAD task, pre-trained VLMs tend to recite visual details and miss implicit sound events, whereas the same VLM fine-tuned on the constructed CoT-AudioCaps data decomposes the problem into three steps: identify objects visible in the video, infer the sound events those objects would produce, and then write an audio caption from the inferred events. This single-stage CoT-SFT raises the CLAP text-audio similarity from 0.252 for the base model to 0.424, and the resulting captions improve downstream video-to-audio generation, narrowing the gap to ground-truth captions on both tested systems.

Load-bearing premise

The whole pipeline assumes that the visual content of an AudioCaps video carries enough information to infer the ground-truth audio events; when a sound has no visible source, the model cannot reason it out and can only memorize training captions.

Editorial extensions

If this is right

  • Video-to-audio systems can be run from silent footage alone, since the fine-tuned VLM supplies the text prompt that currently requires a human annotator.
  • SVAD becomes a testbed for whether VLMs can reason about a modality they never observe, rather than only align modalities they have seen.
  • The single-stage CoT-SFT outperforms the two-stage pipeline, suggesting that letting the VLM see the raw video during reasoning beats routing through an intermediate video caption.
  • The reported gains are metric-based: CLAP rises, longer-phrase text metrics improve, and downstream video-to-audio metrics approach those obtained with ground-truth prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not measure how many AudioCaps sounds are visually inferable, so the practical ceiling of SVAD is unknown; performance may plateau at the fraction of captions whose sound sources actually appear on screen.
  • The same three-step decomposition could transfer to other modality-mismatch tasks, such as inferring tactile or olfactory descriptions from video, if paired human annotations can be obtained.
  • A direct test of the weakest assumption would be to split the evaluation set into sounds with visible sources and sounds without them; CLAP near chance on the latter would indicate the model is associating rather than reasoning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces the task of Reasoning Audio Descriptions from Silent Videos (SVAD), in which a vision-language model must generate an audio caption from a silent video without access to the audio. To improve VLM performance on this task, the authors construct a CoT-AudioCaps dataset: for each AudioCaps video, they use VideoLLaMA2 to produce a video caption, then use GPT-4 to parse the video caption and the ground-truth audio caption into structured video objects and sound events, which are assembled into three subtask datasets (visual objects, sound events, audio captions). They fine-tune VideoLLaMA2 with LoRA under two strategies (two-stage and single-stage) and a chain-of-thought supervised fine-tuning (CoT-SFT) variant. They evaluate on AudioCaps using CLAP and text-text captioning metrics, and they feed the generated captions as text prompts into two VT2A models (STA-V2A and FoleyCraft) to assess downstream video-to-audio generation. The reported results show that CoT-SFT improves CLAP from 0.404 to 0.424 in the single-stage setting and generally improves VT2A metrics relative to captions from the base VLM.

Significance. If the reported gains reflect genuine modal-mismatch reasoning, the paper offers a practical solution to the missing-text-prompt problem in VT2A inference and a new benchmark task for evaluating VLMs' cross-modal reasoning abilities. The construction of a CoT dataset from an established corpus and the systematic comparison of two-stage versus single-stage SFT strategies are useful contributions. The experimental comparisons are clearly described and cover multiple pre-trained VLMs, two SFT strategies, and two downstream VT2A systems. However, the central interpretive claim is not yet fully supported because the supervision for the reasoning steps (video objects and sound events) is derived from the ground-truth audio captions rather than from visual evidence, and no statistical significance analysis accompanies the claimed improvements.

major comments (4)
  1. [Section 2.2, Algorithm 1, Table 2] The CoT-AudioCaps construction derives S_event from ground-truth audio captions via GPT-4 (Algorithm 1, line 3), not from visual evidence. For any sound event without a visual correlate in the video (e.g., off-screen noise, room tone, or an ambiguous source), the training pair does not provide a reasoning example; the model can only memorize object-sound co-occurrences. The paper never quantifies what fraction of AudioCaps sound events is visually inferable, so the improvements in Table 2 (e.g., CLAP from 0.404 to 0.424) are equally consistent with caption-pattern memorization as with the claimed 'modal-mismatch reasoning'. To support the central claim, please provide either (a) a human-judgment study of how many AudioCaps sound events are inferable from silent video, (b) an oracle upper bound where the model is given the correct video objects, or (c) a held-out split separating visually inferable from non-inferable sound events and showing that CoT-SFT's advantage is concentrated in the inferable subset.
  2. [Section 3.1.2 and Tables 2 and 3] No error bars, confidence intervals, or significance tests are reported anywhere. The abstract and Section 3.3 use the word 'significantly', but with single runs the differences between VL2-SFT and VL2-CoT-SFT (CLAP 0.404 vs 0.424, CIDEr 0.550 vs 0.578) may be within run-to-run noise, particularly for CLAP and the VT2A metrics, which are known to be noisy. Please report results over multiple seeds or provide bootstrap or paired significance tests, especially for the headline CLAP numbers and the FD/FAD improvements in Table 3.
  3. [Section 3.1.1 and Table 2] The AVCap-V and DALI baselines are cited from their original papers rather than re-run under the same evaluation protocol. Because the SVAD task definition, video preprocessing, and metric implementations may differ, the claimed 'significant improvements' over these baselines may reflect protocol differences rather than model capability. Please either re-run these baselines with the same video encoder and evaluation harness or explicitly justify comparability (e.g., same test split, same metric code, same caption decoding settings).
  4. [Section 3.4 and Table 3] Table 3 evaluates only the best SVAD model (VL2-CoT-SFT) as a text-prompt source; it does not include VL2-SFT or the two-stage CoT-SFT model. As a result, the table does not demonstrate that VT2A performance tracks SVAD caption quality, which is the paper's second central claim. Please include at least one intermediate model (e.g., VL2-SFT) so the reader can see that better SVAD captions monotonically improve VT2A metrics. In addition, for FoleyCraft the 'w/o AD' row has AV-Align 0.233 versus GT 0.232, so the practical benefit for that model appears marginal and deserves explicit discussion.
minor comments (7)
  1. [Table 3] The row label 'VL2-CoT-SFT23.43' in Table 3 is missing a space before the numeric value; it should read 'VL2-CoT-SFT 23.43'.
  2. [Section 3.1.2] The phrase 'Fréchet distance distance' contains a duplicated word, and 'ROUGEl' should be 'ROUGE-L'.
  3. [Sections 2.1 and 3.3] The text uses both 'Single-Stage' and 'One-Stage' for the same strategy; please standardize the terminology throughout.
  4. [Section 3.1.2] The metric 'A V-Align' should be written as 'AV-Align' for consistency with the standard name of the metric.
  5. [Section 3.1.1] The paper states that AudioCaps contains 43,941 training instances but does not specify the size of the constructed CoT-AudioCaps dataset or how many instances were used for SFT; please clarify the dataset statistics.
  6. [Section 2.3] The training procedure for CoT-SFT does not state whether the intermediate targets V_object and S_event are used as teacher forcing during training or whether model predictions are fed back during training; please clarify the training protocol.
  7. [Section 3.1.3] The implementation details report the LoRA hyperparameters and GPU type but not the number of training steps, batch size, or validation-based model selection procedure; adding these details would improve reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the CoT-AudioCaps label provenance raises an external-validity concern but does not make any test prediction equal to a training input by construction.

full rationale

The SVAD evaluation is not circular: CoT-AudioCaps is built from AudioCaps training pairs (V, C_audio) via Algorithm 1, while Tables 2 and 3 evaluate on the standard held-out AudioCaps test split; no test caption is used to construct the CoT labels or to fit the LoRA adapters. The final objective C_audio = F(V) is trained against the same ground-truth captions used in ordinary SFT, so the CoT-SFT gains are measured against independent targets. The fact that the intermediate sound-event targets S_event are extracted by GPT-4 from the training C_audio (Algorithm 1 line 3: '<V_object, S_event> = LLM(P^user_reason(C_video, C_audio))') is a supervision-construction choice: the sound events are training labels, not test-time predictions, and the paper does not feed test captions into this extraction. The skeptic's worry that many AudioCaps sounds may lack visual correlates is a legitimate external-validity limitation, because it would make the 'modal-mismatch reasoning' claim hard to distinguish from memorized object-sound correlations, but it is not a circularity: no evaluation metric reduces to a fitted parameter or to a self-citation by construction. The self-citations (e.g., STA-V2A as a downstream VT2A model) are not load-bearing since FoleyCraft provides an independent downstream check. Overall, the central derivation is self-contained.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No scientific free parameters are fitted to data; the reported LoRA hyperparameters are standard training choices, not constants the central claim is derived from. The claim rests on dataset quality and metric validity assumptions rather than on fitted constants. The CoT-AudioCaps construction uses GPT-4 to derive intermediate reasoning steps from ground-truth captions, which is the main unexamined dependency. No new physical or formal entities are introduced; CoT-AudioCaps is a dataset artifact, not a postulated entity.

assumptions (4)
  • domain assumption AudioCaps ground-truth audio captions accurately describe the audible events in the videos.
    Used as supervision for SFT and as evaluation targets (Section 3.1.1). If captions are noisy, reported gains may reflect caption noise rather than reasoning improvements.
  • domain assumption The visual content of AudioCaps videos contains sufficient cues to infer the corresponding audio events.
    The whole SVAD task assumes sounds are inferable from silent frames (Section 1, Figure 1). Off-screen or visually ambiguous sounds would make the task ill-posed.
  • domain assumption GPT-4 and VideoLLaMA2 generate sufficiently accurate video captions, video objects, and sound events for CoT supervision.
    Algorithm 1 uses these outputs to construct CoT-AudioCaps without manual verification (Section 2.2). Errors propagate into training labels.
  • domain assumption CLAP similarity is a valid proxy for semantic alignment between generated text and target audio.
    Used as the key metric for SVAD (Section 3.1.2). CLAP can be insensitive to some caption differences and may favor generic captions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model." pith.science (2026). https://pith.science/paper/UKBLY2UB

@misc{pith2026250513062,
  author       = {Pith},
  title        = {Pith review of: Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKBLY2UB}},
  note         = {Machine review of arXiv:2505.13062}
}
read the original abstract

Humans can intuitively infer sounds from silent videos, but whether multimodal large language models can perform modal-mismatch reasoning without accessing target modalities remains relatively unexplored. Current text-assisted-video-to-audio (VT2A) methods excel in video foley tasks but struggle to acquire audio descriptions during inference. We introduce the task of Reasoning Audio Descriptions from Silent Videos (SVAD) to address this challenge and investigate vision-language models' (VLMs) capabilities on this task. To further enhance the VLMs' reasoning capacity for the SVAD task, we construct a CoT-AudioCaps dataset and propose a Chain-of-Thought-based supervised fine-tuning strategy. Experiments on SVAD and subsequent VT2A tasks demonstrate our method's effectiveness in two key aspects: significantly improving VLMs' modal-mismatch reasoning for SVAD and effectively addressing the challenge of acquiring audio descriptions during VT2A inference.

Figures

Figures reproduced from arXiv: 2505.13062 by the authors.

Figure 1
Figure 1. Good sound descriptions from humans. vs. auditory￾irrelevant hallucination from VLMs when reason audio descrip￾tions from silent videos. V2A VT2A temporal-aligned semantic-aligned high-quality temporal-aligned semantic-aligned high-quality (a). video-to-audio (b). text-assisted-video-to-audio Training Inference Audio description ❌ (c). Difficulties when VT2A inference 🤔 How can I get the correct audio description du… view at source ↗
Figure 2
Figure 2. Two primary technical paradigms of video foley and challenges faced by VT2A. this challenge, we introduce the Reasoning Audio Descriptions from Silent Videos (SVAD) task. Unlike existing caption tasks such as audio caption [15,16], video caption [17,18], and audio￾visual caption [19,20], SVAD challenges on reasoning informa￾tion related to a modality (audio) that does not match the input modality (visual). AVCap [19… view at source ↗
Figure 3
Figure 3. Overview of our methods for SVAD task, including two SFT strategies, the SFT training for VLM by LoRA, the CoT-Audiocaps Dataset construction process, and the CoT-based SFT method for SVAD. descriptions as ground truth (GT). SFT has shown a significant improvement in SVAD tasks, with the single-stage strategy per￾forming better. Chain-of-thought (CoT) is a specialized tool de￾signed for the task of multi-step reason… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The templates for constructing the CoT-Audiocaps Dataset, the direct prompt template for video and audio caption from video for VLMs, and the CoT prompt template for VLMs (For LLMs, replace the video with the video caption) [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 20 canonical work pages

  1. [1]

    Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model

    Introduction Human cognition inherently integrates multimodal information, allowing us to infer auditory experiences from purely visual stimuli like silent videos as shown in Figure 1. This remarkable ability stems from our brain’s capacity to associate visual pat- terns with corresponding sounds through learned experiences and cognitive reasoning [1]. Al...

    work page Pith review arXiv 2025

  2. [2]

    SFT for SV AD Given a silent video V = I T t=1 withT frames, the SV AD task aims to generate a corresponding audio descriptionCaudio

    Methods 2.1. SFT for SV AD Given a silent video V = I T t=1 withT frames, the SV AD task aims to generate a corresponding audio descriptionCaudio. Caudio =F (V ), (1) whereF denotes the vision understanding and reasoning mod- els like VLMs. Utilizing pre-trained VLMs for zero-shot infer- ence often results in suboptimal performance. Existing VLMs are typi...

    work page

  3. [3]

    Experiments In this section, we conduct detailed experiments to evaluate the performance of VLMs in SV AD, and the effectiveness of the proposed method on SV AD and VT2A tasks. Our experiments seek to answer the following research questions (RQs): • RQ1: How do different pre-training VLMs perform in modal-mismatch reasoning for the SV AD task? • RQ2: Is S...

    work page

  4. [4]

    Conclusion This paper introduces a new SV AD task that reasons audio de- scriptions from silent videos, tackling the challenge of audio descriptions missing in VT2A inference. Through evaluation of the SV AD task, we reveal VLMs’ inherent limitations in modal- mismatch reasoning when target modalities are absent, and pro- pose an innovative CoT-SFT strate...

    work page

  5. [5]

    Minigpt4-video: Advancing mul- timodal llms for video understanding with interleaved visual- textual tokens,

    K. Ataallah, X. Shen, E. Abdelrahman, E. Sleiman, D. Zhu, J. Ding, and M. Elhoseiny, “Minigpt4-video: Advancing mul- timodal llms for video understanding with interleaved visual- textual tokens,” arXiv preprint arXiv:2404.03413, 2024

    arXiv 2024

  6. [6]

    Multisensory- guided associative learning enhances multisensory representation in primary auditory cortex,

    X. Han, J. Xu, S. Chang, L. Keniston, and L. Yu, “Multisensory- guided associative learning enhances multisensory representation in primary auditory cortex,” Cerebral Cortex, vol. 32, no. 5, pp. 1040–1054, 2022

    work page 2022

  7. [7]

    Mm-llms: Recent advances in multimodal large language mod- els,

    D. Zhang, Y . Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu, “Mm-llms: Recent advances in multimodal large language mod- els,” arXiv preprint arXiv:2401.13601, 2024

    arXiv 2024

  8. [8]

    Videochat: Chat-centric video under- standing,

    K. Li, Y . He, Y . Wang, Y . Li, W. Wang, P. Luo, Y . Wang, L. Wang, and Y . Qiao, “Videochat: Chat-centric video under- standing,” arXiv preprint arXiv:2305.06355, 2023

    arXiv 2023

Show all 41 references
  1. [9]

    Video- llava: Learning united visual representation by alignment before projection,

    B. Lin, Y . Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan, “Video- llava: Learning united visual representation by alignment before projection,” arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  2. [10]

    Sta-v2a: Video-to-audio generation with semantic and temporal alignment,

    Y . Ren, C. Li, M. Xu, W. Liang, Y . Gu, R. Chen, and D. Yu, “Sta-v2a: Video-to-audio generation with semantic and temporal alignment,” arXiv preprint arXiv:2409.08601, 2024

    2024 arXiv

  3. [11]

    Video-to-audio generation with hidden alignment,

    M. Xu, C. Li, X. Tu, Y . Ren, R. Chen, Y . Gu, W. Liang, and D. Yu, “Video-to-audio generation with hidden alignment,”arXiv preprint arXiv:2407.07464, 2024

    2024 arXiv

  4. [12]

    Frieren: Efficient video-to-audio generation with rectified flow matching,

    Y . Wang, W. Guo, R. Huang, J. Huang, Z. Wang, F. You, R. Li, and Z. Zhao, “Frieren: Efficient video-to-audio generation with rectified flow matching,” arXiv preprint arXiv:2406.00320, 2024

    2024 arXiv

  5. [13]

    V2a- mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,

    H. Wang, J. Ma, S. Pascual, R. Cartwright, and W. Cai, “V2a- mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 14, 2024, pp. 15 492–15 501

    2024

  6. [14]

    Foleygen: Visually-guided audio generation,

    X. Mei, V . Nagaraja, G. Le Lan, Z. Ni, E. Chang, Y . Shi, and V . Chandra, “Foleygen: Visually-guided audio generation,” in 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2024, pp. 1–6

    2024

  7. [15]

    Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumb- ley, Y . Zou, and W. Wang, “Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 2024

    2024

  8. [16]

    Text-to-audio generation synchronized with videos,

    S. Mo, J. Shi, and Y . Tian, “Text-to-audio generation synchronized with videos,” arXiv preprint arXiv:2403.07938, 2024

    2024 arXiv

  9. [17]

    Foleycrafter: Bring silent videos to life with lifelike and synchro- nized sounds,

    Y . Zhang, Y . Gu, Y . Zeng, Z. Xing, Y . Wang, Z. Wu, and K. Chen, “Foleycrafter: Bring silent videos to life with lifelike and synchro- nized sounds,” arXiv preprint arXiv:2407.01494, 2024

    2024 arXiv

  10. [18]

    Read, watch and scream! sound generation from text and video,

    Y . Jeong, Y . Kim, S. Chun, and J. Lee, “Read, watch and scream! sound generation from text and video,” arXiv preprint arXiv:2407.05551, 2024

    2024 arXiv

  11. [19]

    Stable-v2a: Syn- thesis of synchronized sound effects with temporal and semantic controls,

    R. F. Gramaccioni, C. Marinoni, E. Postolache, M. Comunit `a, L. Cosmo, J. D. Reiss, and D. Comminiello, “Stable-v2a: Syn- thesis of synchronized sound effects with temporal and semantic controls,” arXiv preprint arXiv:2412.15023, 2024

    2024 arXiv

  12. [20]

    Lavcap: Llm-based audio-visual captioning using optimal transport,

    K. Rho, H. Lee, V . Iverson, and J. S. Chung, “Lavcap: Llm-based audio-visual captioning using optimal transport,” arXiv preprint arXiv:2501.09291, 2025

    2025 arXiv

  13. [21]

    Conette: An efficient au- dio captioning system leveraging multiple datasets with task em- bedding,

    E. Labb, T. Pellegrini, J. Pinquier et al., “Conette: An efficient au- dio captioning system leveraging multiple datasets with task em- bedding,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 2024

    2024

  14. [22]

    Automatic video captioning using tree hierarchical deep convolutional neu- ral network and asrnn-bi-directional lstm,

    N. Kavitha, K. R. Soundar, R. Karthick, and J. Kohila, “Automatic video captioning using tree hierarchical deep convolutional neu- ral network and asrnn-bi-directional lstm,” Computing, vol. 106, no. 11, pp. 3691–3709, 2024

    2024

  15. [23]

    Streaming dense video captioning,

    X. Zhou, A. Arnab, S. Buch, S. Yan, A. Myers, X. Xiong, A. Na- grani, and C. Schmid, “Streaming dense video captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 243–18 252

    2024

  16. [24]

    Avcap: Leveraging audio- visual features as text tokens for captioning,

    J. Kim, J. Shin, and J. Kim, “Avcap: Leveraging audio- visual features as text tokens for captioning,” arXiv preprint arXiv:2407.07801, 2024

    2024 arXiv

  17. [25]

    LoRA: Low-Rank Adaptation of Large Language Models,

    E. J. Hu, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “LoRA: Low-Rank Adaptation of Large Language Models,” in International Conference on Learning Representa- tions, 2022

    2022

  18. [26]

    To further enhance the reasoning capabilities in SV AD tasks, we propose a Chain-of-Thought-based Supervised Fine- Tuning (CoT-SFT) strategy and construct the CoT-AudioCaps dataset for it, which provides explicit reasoning chains con- necting visual scenes to their correspondi...

  19. [27]

    An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching,

    H. Malard, M. Olvera, S. Lathuili `ere, and S. Essid, “An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching,” Advances in Neu- ral Information Processing Systems , vol. 37, pp. 38 720–38 743, 2025

    2025

  20. [28]

    Videollama 2: Advanc- ing spatial-temporal modeling and audio understanding in video- llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhao et al. , “Videollama 2: Advanc- ing spatial-temporal modeling and audio understanding in video- llms,” arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  21. [29]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolu- tion,

    Z. Liu, Y . Dong, Z. Liu, W. Hu, J. Lu, and Y . Rao, “Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolu- tion,” arXiv preprint arXiv:2409.12961, 2024

    2024 arXiv

  22. [30]

    Internvl: Scaling up vision foun- dation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu et al., “Internvl: Scaling up vision foun- dation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2024

  23. [31]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information pro- cessing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  24. [32]

    Audiocaps: Generat- ing captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generat- ing captions for audios in the wild,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Paper...

    2019

  25. [33]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  26. [34]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023, 2023

    2023

  27. [35]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002

    2002

  28. [36]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic eval- uation measures for machine translation and/or summarization , 2005

    2005

  29. [37]

    Rouge: A package for automatic evaluation of sum- maries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of sum- maries,” in Text summarization branches out, 2004

    2004

  30. [38]

    Cider: Consensus-based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in Proceedings of the IEEE conference on computer vision and pattern recogni- tion, 2015

    2015

  31. [39]

    Spice: Se- mantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Se- mantic propositional image caption evaluation,” in ECCV 2016. Springer, 2016

    2016

  32. [40]

    Diverse and aligned audio-to-video generation via text-to-video model adaptation,

    G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz, and Y . Adi, “Diverse and aligned audio-to-video generation via text-to-video model adaptation,” in Proceedings of the AAAI Conference on Ar- tificial Intelligence, vol. 38, no. 7, 2024, pp. 6639–6647

    2024

  33. [41]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Let- man, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.