Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video-to-audio model posts best VGGSound benchmark results

desk verdict Solid industrial V2A with strong VGGSound numbers, but the few-shot headline is asserted, not measured — no k-shot eval, no CoT ablation, no human studies. read the letter →

arxiv 2412.09168 v1 pith:BS33F62Z submitted 2024-12-12 cs.SD cs.CVcs.MMeess.AS

Zihao Chen , Haomin Zhang , Xinhan Di , Haoyu Wang , Sizhe Shan , Junjie Zheng , Yunming Liang , Yihan Fan
show 5 more authors
Xinfa Zhu Wenjie Tian Yihua Wang Chaofan Ding Lei Xie
This is my paper · ORCID
classification cs.SDcs.CVcs.MMeess.AS
keywords video-to-audiogenerationsoundeffectssynthesisflowmatchingmulti-modalchain-of-thoughtfew-shotlearningaudio-visualalignmentdiffusiontransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

YingSound is a proposed foundation model that turns video, optionally with a text prompt, into synchronized sound effects, aimed at industrial settings where labeled audio-video data is scarce. The paper's central claim is that the model produces high-quality, semantically and temporally aligned audio, and that a multi-modal chain-of-thought module extends it to few-shot scenarios such as game character motions, animation, and short videos. The reported evaluations on the VGGSound test set place the text-conditioned version ahead of the compared systems on audio fidelity, semantic alignment, and temporal alignment, with the video-only version close behind.

What carries the argument

The load-bearing mechanism is a two-tower transformer in which an audio-visual aggregator computes, at each DiT layer, a linear fusion of audio and video stream outputs, letting visual semantics and timing guide audio generation. Around it, the paper assembles a multi-stage training schedule (text-to-audio, then text-plus-video, then video with random text dropout) and a chain-of-thought refinement loop: coarse audio and video signal embeddings go through transformer modules that emit multi-modal reasoning, reward models score the result, and an expert module regenerates finer audio, trained first in a few-shot supervised setting and then through iterative preference learning and intrinsic self-correction.

What would settle it

Take a held-out set of industrial scene types never seen in training, provide 0, 1, 5, and 10 labeled clips per scene, and measure generated-audio quality and semantic match with and without the chain-of-thought module; if the module adds no measurable improvement, the few-shot claim is contradicted.

Watch

Extended reading notes

Core claim

The discovery presented is the design and benchmarking of YingSound, an end-to-end learned video-to-audio generator. It combines a conditional flow matching transformer with a learnable audio-visual aggregator that injects multi-resolution visual features into every audio DiT layer, and a multi-modal visual-audio chain-of-thought module that produces refined audio from coarse audio, generated reasoning, and reward-model feedback. On the VGGSound-test benchmark, the paper reports the text-conditioned version achieves FAD of 0.78, FD of 6.28, KL-sigmoid of 3.97, IS of 14.02, CLIP of 16.86, and AV of 0.25, while the video-only version reaches 0.80 FAD and 16.14 CLIP; these are the lowest FAD, FD, and KL-sigmoid and the highest IS and CLIP among Diff-Foley, FoleyCrafter, V2A-Mapper, and Frieren, with AV tied at the top value.

Load-bearing premise

The central claim rests on the assumption that the chain-of-thought training recipe, a few-shot supervised start, iterative preference learning, and intrinsic self-correction, actually transfers to new industrial scenes from a few labeled examples, an effect that is currently shown only through qualitative application samples.

Editorial extensions

If this is right

  • The same trained model can generate audio from text only, video only, or both, because text conditioning is randomly dropped during the final training stage.
  • For game-character motions and animation clips up to roughly 30 seconds, the paper claims synchronized sound effects with coarse temporal alignment to the action.
  • For short videos, the model is claimed to adapt sound effects to the video's rhythm toward zero-shot use.
  • The reported VGGSound-test numbers give future video-to-audio systems concrete comparison points to beat on audio fidelity, semantic alignment, and temporal alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates few-shot settings only with qualitative samples, so a k-shot protocol on unseen scene categories (1/5/10 labeled clips per scene) would be the natural extension to make the few-shot claim quantitative.
  • Running the model with the chain-of-thought module disabled on the same few-shot industrial scenes would isolate whether the few-shot benefit comes from the CoT training or from the underlying flow-matching generator.
  • If the CoT module does generalize, video-to-audio systems could inherit the reasoning-style transfer seen in language models, with the generated intermediate thoughts acting as a bridge between seen and unseen scenes.
  • The industry video-to-audio dataset is described but not released with metrics, so reproducing the pipeline would require building a comparable filtered dataset; a public release would let others verify the reported gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces YingSound, a video-guided sound-effects generation system built on a flow-matching transformer with a learnable audio-visual aggregator (AVA) and a multi-modal chain-of-thought (CoT) module. The authors claim that the CoT module enables high-quality audio generation in few-shot settings, and they also claim to present a new industry-standard V2A dataset. The system is evaluated on the VGGSound test set, where it reports state-of-the-art or competitive results on FAD, FD, KL-sigmoid, IS, CLIP, and AV metrics against Diff-Foley, FoleyCrafter, V2A-Mapper, and Frieren. Section 5 presents three qualitative application samples (game motions, animation, short videos). The paper concludes with claims about few-shot capability, the V2A dataset, and future directions.

Significance. If the few-shot generation claim were supported by experiments, YingSound would be a notable contribution to video-to-audio generation for low-resource industrial scenarios. The reported VGGSound results show that the model is competitive on a standard benchmark. However, the central few-shot contribution is not measured anywhere, the CoT module is never ablated, no human study is reported despite being claimed in the abstract, and the proposed V2A dataset is not actually described with concrete statistics or availability. These gaps substantially weaken the paper's stated significance.

major comments (5)
  1. [Abstract, Section 1, Section 5] The central claim of few-shot sound generation is never experimentally evaluated. No few-shot benchmark, no k-shot experiments, and no adaptation protocol are presented; Section 5 offers only three qualitative application samples with no statement of how many annotated samples were used, how the few-shot setting was defined, or how outputs compare to a baseline without the CoT module. Since few-shot generation is the headline contribution, the paper's title and abstract are not supported by the evidence.
  2. [Section 3.4, Section 4.2] The multi-modal CoT module's training recipe (few-shot supervised setting, iterative preference learning, intrinsic self-correction) is described only qualitatively, without hyperparameters, reward-model specifications, or loss formulations. More importantly, there is no ablation isolating the CoT module from the base flow-matching + AVA model; therefore, the performance gains in Table 2 cannot be attributed to the proposed CoT component.
  3. [Abstract, Section 4] The abstract states that YingSound is validated 'through automated evaluations and human studies,' but the paper contains no human evaluation protocol, no listening test, no MOS scores, and no comparisons based on human judgments. This is a direct mismatch between the claimed validation and the reported experiments.
  4. [Introduction, Section 3.2, Table 1] The paper repeatedly claims to present a new 'industry-standard V2A dataset,' yet Section 3.2 only describes a data processing pipeline (collection, annotation, filtering, cutting) without reporting dataset size, duration, scene composition, or release details. Table 1 lists only existing public datasets (AudioCaps, WavCaps, TangoPromptBank, MusicCaps, AF-AudioSet, VGGSound), so the claimed new dataset is not substantiated in the manuscript.
  5. [Section 4.1, Evaluation Metrics] The CLIP score used for semantic alignment is computed with Wav2CLIP as the audio encoder and CLIP as the video encoder, while the model's video conditioning uses CLIP features (clip-vit, clip-vit-large-patch14, clip-convnext-XXL). Because the model is trained to align with CLIP visual features, the evaluation metric shares a representation space with the model's own conditioning, which can bias the semantic-alignment results in the model's favor. The paper should acknowledge this limitation and include an independent semantic metric.
minor comments (4)
  1. [Throughout] There are several typos and spacing inconsistencies, e.g., 'video-to-auido' in the introduction, 'A V A' with spaces, and inconsistent use of 'YingSound w. text' versus 'YingSound w/ text'.
  2. [Section 3.3] The description of the AVA module and the audio-video mapping (AVMM) is brief; a figure or a more detailed schematic would improve reproducibility.
  3. [Section 4.1] The paper does not report the number of samples used for V2A-Mapper evaluation (the text mentions 7,590 clips but the table includes full comparison); clarifying the evaluation subset for each baseline would strengthen the comparison.
  4. [Section 5] The application samples are described only in text; providing the actual generated audio (e.g., on the project page mentioned) is essential for verifying the qualitative claims.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the CLIP semantic metric is computed in the same representation space the model is conditioned on and that the data pipeline filters with, so the reported CLIP score is partly self-referential; the few-shot claim is unmeasured but that is an evidence gap, not a circular reduction.

  1. other [Section 3.2 (Data-Pipeline filtering); Section 3.3 (Video Foundation Model Cluster); Section 4.1 (Evaluation Metrics)]
    "For video conditions, we construct a Video Foundation Model Cluster inspired by [86]. We use different input resolutions and different visual encoders pre-trained, including clip-vit, clip-vit-large-patch14, clip-convnext-XXL [87], dinov2-giant [88] to get frame-level features. ... Next, we employ CLIP and CLAP [78] techniques to filter out semantically misaligned data. ... The CLIP measures the similarity between the input video and the generated audio embeddings within the same representation space."

    The paper's headline semantic-alignment result (CLIP score 16.86) is not an independent confirmation of alignment: the model's video-conditioning stack is built from CLIP encoders, the training-data pipeline uses CLIP/CLAP to discard semantically misaligned pairs, and the evaluation metric measures similarity in the CLIP/Wav2CLIP representation space. The model is explicitly trained to map CLIP video features toward audio, so high CLIP similarity is partially a constructed consequence of the chosen conditioning and filtering, not an external test. Because FAD, FD, KL-sigmoid, IS, and AV metrics do not share that representation, the core V2A quality claim retains independent support; hence this is partial, not total, circularity.

full rationale

No load-bearing self-citation chain was found; the paper builds on external T2A, flow-matching, CoT, and reward-model literature. The central few-shot contribution is not measured by any k-shot or ablation experiment, and the abstract's 'human studies' have no corresponding section; however, an unmeasured claim is an evidence gap, not a circular reduction, so it does not by itself raise the circularity score. The one identifiable circular element is the CLIP semantic metric: the same CLIP representation family appears on the conditioning side (clip-vit, clip-vit-large-patch14, clip-convnext-XXL), the data-filtering side (CLIP/CLAP), and the evaluation side (CLIP video encoder with Wav2CLIP audio encoder). That makes the reported advantage in CLIP partially self-referential by construction. The remaining objective metrics (FAD, FD, KL-sigmoid, AV) are representation-independent enough to support the model's general V2A quality, so the score is 4 rather than higher.

Assumptions & free parameters 6 free parameters · 5 assumptions · 3 invented entities

No theory is derived; this is an empirical ML system. The central claim relies on assumptions about the effectiveness of flow matching, the AVA connector, the CoT/reward-module training, and the validity of the chosen metrics. None of these receive ablations or external validation.

free parameters (6)
  • Training-stage data mixing ratio (T2A:T&V2A:V2A) = 1:1:2
    Set by hand for the final training stage (Section 3.3); no sweep or sensitivity analysis is reported.
  • Text condition keep probabilities = 1.0, 1.0, 0.5
    Chosen for the three training stages (Section 3.3); no justification or ablation.
  • Video condition keep probabilities = 0.0, 0.5, 0.75
    Chosen for the three training stages (Section 3.3); no justification or ablation.
  • Inference NFE with sway sampling = 64
    Set in implementation details (Section 4.1); no comparison of NFE values.
  • Training step budgets per stage = 250k, 50k, 230k
    Used for T2A, T&V2A, and V2A stages (Section 4.1); no evidence they are optimal.
  • Optimizer hyperparameters = lr=3e-5, grad clip=0.2, batch=128
    Implementation details (Section 4.1); no sensitivity analysis is provided.
assumptions (5)
  • domain assumption Flow matching and DiT yield high-quality audio generation
    The whole generator is built on prior results [82,83]; the paper provides no derivation or independent validation.
  • ad hoc to paper The linear AVMM mapping at each DiT layer improves audio-visual alignment
    Eq. (1) in Section 3.3 defines the mapping; no ablation separates AVA from the rest of the model.
  • ad hoc to paper The multi-modal CoT module with reward models improves few-shot audio generation
    Asserted in Section 3.4; no few-shot benchmark, no ablation, no human-study results.
  • domain assumption Objective metrics FAD, FD, KL-sigmoid, IS, CLIP, and AV are valid proxies for generation quality and alignment
    Section 4.1 adopts these metrics without validating them on V2A tasks; CLIP shares representation space with the model's visual encoder.
  • domain assumption VGGSound training/test split supports the reported generalization
    The paper uses 173k VGGSound clips for training and 15,446 clips for testing (Sections 4.1, 4.2); standard practice, but no leakage check is described.
invented entities (3)
  • learnable audio-visual aggregator (AVA)
    purpose: integrates high-resolution visual features with corresponding audio features at multiple DiT stages
    No ablation or external measurement isolates the AVA's contribution to the reported gains.
  • multi-modal visual-audio chain-of-thought (CoT) module
    purpose: generates finer-level audio in few-shot settings via a signal module, generation module, reward models, and expert module
    The module is described qualitatively in Section 3.4 but is never measured in a few-shot experiment or compared against a model without CoT.
  • industry-standard V2A dataset
    purpose: training data covering movies, games, and commercials for the claimed industrial applicability
    The dataset is presented in Sections 3.2 and 5 but not released; its size, statistics, and access are not given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls." pith.science (2026). https://pith.science/paper/BS33F62Z

@misc{pith2026241209168,
  author       = {Pith},
  title        = {Pith review of: YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BS33F62Z}},
  note         = {Machine review of arXiv:2412.09168}
}
read the original abstract

Generating sound effects for product-level videos, where only a small amount of labeled data is available for diverse scenes, requires the production of high-quality sounds in few-shot settings. To tackle the challenge of limited labeled data in real-world scenes, we introduce YingSound, a foundation model designed for video-guided sound generation that supports high-quality audio generation in few-shot settings. Specifically, YingSound consists of two major modules. The first module uses a conditional flow matching transformer to achieve effective semantic alignment in sound generation across audio and visual modalities. This module aims to build a learnable audio-visual aggregator (AVA) that integrates high-resolution visual features with corresponding audio features at multiple stages. The second module is developed with a proposed multi-modal visual-audio chain-of-thought (CoT) approach to generate finer sound effects in few-shot settings. Finally, an industry-standard video-to-audio (V2A) dataset that encompasses various real-world scenarios is presented. We show that YingSound effectively generates high-quality synchronized sounds across diverse conditional inputs through automated evaluations and human studies. Project Page: \url{https://giantailab.github.io/yingsound/}

Figures

Figures reproduced from arXiv: 2412.09168 by the authors.

Figure 1
Figure 1. The data collection and processing pipeline with human-in-the-loop. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overview of the YingSound. It comprises two key components: Conditional Flow Matching with Transformers and a Multi-modal Chain-of-Thought Based Audio Generation. 3.3 Conditional Flow Matching with Transformers We perform audio generation based on flow matching [82] and a scalable diffusion transformer (DiT) architecture [83], which have been proven to be both high-quality and efficient in image and audio genera… view at source ↗
Figure 3
Figure 3. Temporal Alignment comparison. field estimator and channel-level cross-modal feature fusion, enabling efficient audio generation with superior synchronization and quality. 4.2 Video-to-Audio Experiments We present a quantitative comparison of semantic alignment and audio quality in the VGGSound-test dataset. The VGGSound-test dataset comprises 15,446 videos collected from YouTube, covering a diverse range of genres.… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Application visualization results of YingSound. 5 Applications In few-shot settings, it is promising to apply industrial-level applications for long-duration scenes, including synchronized sound effects generation for the motions of game characters, animation videos, a…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    The paper introduces an event-level hierarchical control benchmark and a training-free agentic pipeline for video-to-audio generation, claiming 40.7% better controllability and 12.5% better perceptual quality than pri...

  2. Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks

    cs.SD 2025-05 reject novelty 4.0 of 10

    The paper proposes an open-source 10-hour video-to-piano benchmark with four-level Chain-of-Perform annotations, but supplies only preliminary, incomplete baseline results.

Reference graph

Works this paper leans on

119 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    Fastspeech 2: Fast and high-quality end-to-end text to speech

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558, 2020

    arXiv 2006

  2. [2]

    Tacotron: Towards end-to-end speech synthesis

    Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135, 2017

    arXiv 2017

  3. [3]

    Conditional variational autoencoder with ad- versarial learning for end-to-end text-to-speech

    Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with ad- versarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530–5540. PMLR, 2021

    2021

  4. [4]

    Naturalspeech: End-to-end text-to-speech synthesis with human-level quality

    Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  5. [5]

    Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis

    Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International conference on machine learning, pages 5180–5189. PMLR, 2018

    2018

  6. [6]

    Robust and fine-grained prosody control of end-to-end speech synthesis

    Younggun Lee and Taesu Kim. Robust and fine-grained prosody control of end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5911–5915. IEEE, 2019

    2019

  7. [7]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

    Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116, 2023

    arXiv 2023

  8. [8]

    Char2wav: End-to-end speech synthesis

    Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. Char2wav: End-to-end speech synthesis. 2017. 9

    2017

Show all 119 references
  1. [9]

    Flashspeech: Efficient zero-shot speech synthesis

    Zhen Ye, Zeqian Ju, Haohe Liu, Xu Tan, Jianyi Chen, Yiwen Lu, Peiwen Sun, Jiahao Pan, Weizhen Bian, Shulin He, et al. Flashspeech: Efficient zero-shot speech synthesis. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6998–7007, 2024

    2024

  2. [10]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000, 2023

    2023 arXiv

  3. [11]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  4. [12]

    Llama-omni: Seamless speech interaction with large language models

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  5. [13]

    Mini-omni: Language models can hear, talk while thinking in streaming

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725, 2024

    2024 arXiv

  6. [14]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024

    arXiv

  7. [15]

    Lauragpt: Listen, attend, understand, and regenerate audio with gpt

    Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023

    2023 arXiv

  8. [16]

    Audiochatllama: Towards general-purpose speech abilities for llms

    Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Audiochatllama: Towards general-purpose speech abilities for llms. In Proceedings of the 2024 Conference of the North American C...

    2024

  9. [17]

    Vita: Towards open-source interactive omni multimodal llm

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024

    2024 arXiv

  10. [18]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023

    2023

  11. [19]

    Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment

    Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  12. [20]

    Multitrack music transformer

    Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg-Kirkpatrick. Multitrack music transformer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  13. [21]

    Symbolic music generation with diffusion models

    Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.16091, 2021

    2021 arXiv

  14. [22]

    Generating chord progression from melody with flexible harmonic rhythm and controllable harmonic density

    Shangda Wu, Yue Yang, Zhaowen Wang, Xiaobing Li, and Maosong Sun. Generating chord progression from melody with flexible harmonic rhythm and controllable harmonic density. EURASIP Journal on Audio, Speech, and Music Processing, 2024(1):4, 2024

    2024

  15. [23]

    Getmusic: Generating any music tracks with a unified representation and diffusion framework

    Ang Lv, Xu Tan, Peiling Lu, Wei Ye, Shikun Zhang, Jiang Bian, and Rui Yan. Getmusic: Generating any music tracks with a unified representation and diffusion framework. arXiv preprint arXiv:2305.10841, 2023

    2023 arXiv

  16. [24]

    Midinet: A convolutional generative adversarial network for symbolic-domain music generation

    Li-Chia Yang, Szu-Yu Chou, and Yi-Hsuan Yang. Midinet: A convolutional generative adversarial network for symbolic-domain music generation. arXiv preprint arXiv:1703.10847, 2017. 10

    2017 arXiv

  17. [25]

    Musiclm: Generating music from text

    Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023

    2023 arXiv

  18. [26]

    Simple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  19. [27]

    Video background music generation with controllable music transformer

    Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan. Video background music generation with controllable music transformer. In Proceedings of the 29th ACM International Conference on Multimedia , pages 2037–2045, 2021

    2021

  20. [28]

    Jukebox: A generative model for music

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020

    2005 arXiv

  21. [29]

    Efficient neural music generation

    Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, et al. Efficient neural music generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [30]

    Lafma: A latent flow matching model for text-to-audio generation

    Wenhao Guan, Kaidi Wang, Wangjin Zhou, Yang Wang, Feng Deng, Hui Wang, Lin Li, Qingyang Hong, and Yong Qin. Lafma: A latent flow matching model for text-to-audio generation. arXiv preprint arXiv:2406.08203, 2024

    2024 arXiv

  23. [31]

    Audiolcm: Text-to-audio generation with latent consistency models

    Huadai Liu, Rongjie Huang, Yang Liu, Hengyuan Cao, Jialei Wang, Xize Cheng, Siqi Zheng, and Zhou Zhao. Audiolcm: Text-to-audio generation with latent consistency models. arXiv preprint arXiv:2406.00356, 2024

    2024 arXiv

  24. [32]

    Audiolm: a language modeling approach to audio generation

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and ...

    2023

  25. [33]

    I hear your true colors: Image guided audio generation

    Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1–5. IEEE, 2023

    2023

  26. [34]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. In British Machine Vision Conference (BMVC), 2021

    2021

  27. [35]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  28. [36]

    Imagen video: High definition video generation with diffusion models

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022

    arXiv

  29. [37]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022

    2022 arXiv

  30. [38]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  31. [39]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  32. [40]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 11

    2024 arXiv

  33. [41]

    Uniaudio: An audio foundation model toward universal audio generation

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023

    arXiv

  34. [42]

    Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner, 2024

    Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjie Huang, Xiang Li, Xu Tan, Xixin Wu, and Helen Meng. Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner, 2024

    2024

  35. [43]

    Plumbley

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:...

    2024

  36. [44]

    Auffusion: Leveraging the power of diffu- sion and large language models for text-to-audio generation

    Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Auffusion: Leveraging the power of diffu- sion and large language models for text-to-audio generation. arXiv preprint arXiv:2401.01044, 2024

    2024 arXiv

  37. [45]

    Audiolm: a language modeling approach to audio generation, 2023

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation, 2023

    2023

  38. [46]

    M2ugen: Multi-modal music understanding and generation with the power of large language models, 2024

    Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. M2ugen: Multi-modal music understanding and generation with the power of large language models, 2024

    2024

  39. [47]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

    Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916–13932. PMLR, 2023

    2023

  40. [48]

    Picoaudio: Enabling precise timestamp and frequency controllability of audio events in text-to-audio generation

    Zeyu Xie, Xuenan Xu, Zhizheng Wu, and Mengyue Wu. Picoaudio: Enabling precise timestamp and frequency controllability of audio events in text-to-audio generation. arXiv preprint arXiv:2407.02869, 2024

    2024 arXiv

  41. [49]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pages 609–617, 2017

    2017

  42. [50]

    Audio-visual instance discrimination with cross-modal agreement

    Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12475–12486, 2021

    2021

  43. [51]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023

    2023

  44. [52]

    Objects that sound

    Relja Arandjelovic and Andrew Zisserman. Objects that sound. InProceedings of the European conference on computer vision (ECCV), pages 435–451, 2018

    2018

  45. [53]

    Mix and localize: Localizing sound sources in mixtures

    Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10483–10492, 2022

    2022

  46. [54]

    Audio-visual segmentation with semantics

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, et al. Audio-visual segmentation with semantics. International Journal of Computer Vision, pages 1–21, 2024

    2024

  47. [55]

    Audio-visual synchronisation in the wild

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Audio-visual synchronisation in the wild. arXiv preprint arXiv:2112.04432, 2021

    2021 arXiv

  48. [56]

    Self-supervised video forensics by audio-visual anomaly detection

    Chao Feng, Ziyang Chen, and Andrew Owens. Self-supervised video forensics by audio-visual anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10491–10503, 2023. 12

    2023

  49. [57]

    Gramaccioni, Emilian Postolache, Emanuele Rodolà, Danilo Comminiello, and Joshua D

    Marco Comunità, Riccardo F. Gramaccioni, Emilian Postolache, Emanuele Rodolà, Danilo Comminiello, and Joshua D. Reiss. Syncfusion: Multimodal onset-synchronized video-to- audio foley synthesis, 2023

    2023

  50. [58]

    Video-to-audio generation with hidden alignment, 2024

    Manjie Xu, Chenxing Li, Xinyi Tu, Yong Ren, Rilin Chen, Yu Gu, Wei Liang, and Dong Yu. Video-to-audio generation with hidden alignment, 2024

    2024

  51. [59]

    TiV A: Time-aligned video-to-audio generation

    Xihua Wang, Yuyue Wang, Yihan Wu, Ruihua Song, Xu Tan, Zehua Chen, Hongteng Xu, and Guodong Sui. TiV A: Time-aligned video-to-audio generation. InACM Multimedia 2024, 2024

    2024

  52. [60]

    Lova: Long-form video-to-audio generation, 2024

    Xin Cheng, Xihua Wang, Yihan Wu, Yuyue Wang, and Ruihua Song. Lova: Long-form video-to-audio generation, 2024

    2024

  53. [61]

    Sta-v2a: Video-to-audio generation with semantic and temporal alignment, 2024

    Yong Ren, Chenxing Li, Manjie Xu, Wei Liang, Yu Gu, Rilin Chen, and Dong Yu. Sta-v2a: Video-to-audio generation with semantic and temporal alignment, 2024

    2024

  54. [62]

    Diff-foley: Synchronized video-to- audio synthesis with latent diffusion models, 2023

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to- audio synthesis with latent diffusion models, 2023

    2023

  55. [63]

    Frieren: Efficient video-to-audio generation with rectified flow matching

    Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audio generation with rectified flow matching. arXiv preprint arXiv:2406.00320, 2024

    2024 arXiv

  56. [64]

    Temporally aligned audio for video with autoregression

    Ilpo Viertola, Vladimir Iashin, and Esa Rahtu. Temporally aligned audio for video with autoregression. arXiv preprint arXiv:2409.13689, 2024

    2024 arXiv

  57. [65]

    Taming visually guided sound generation, 2021

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation, 2021

    2021

  58. [66]

    I hear your true colors: Image guided audio generation, 2023

    Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation, 2023

    2023

  59. [67]

    Conditional generation of audio from video via foley analogies, 2023

    Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. Conditional generation of audio from video via foley analogies, 2023

    2023

  60. [68]

    Foleygen: Visually-guided audio generation, 2023

    Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually-guided audio generation, 2023

    2023

  61. [69]

    Sonicvisionlm: Playing sound with vision language models

    Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. Sonicvisionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26866–26875, 2024

    2024

  62. [70]

    Masked generative video-to-audio transformers with enhanced synchronicity

    Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, and Joan Serrà. Masked generative video-to-audio transformers with enhanced synchronicity. arXiv preprint arXiv:2407.10387, 2024

    2024 arXiv

  63. [71]

    Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494, 2024

    2024 arXiv

  64. [72]

    Read, watch and scream! sound generation from text and video

    Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound generation from text and video. arXiv preprint arXiv:2407.05551, 2024

    2024 arXiv

  65. [73]

    Draw an audio: Leveraging multi-instruction for video-to-audio synthesis

    Qi Yang, Binjie Mao, Zili Wang, Xing Nie, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. Draw an audio: Leveraging multi-instruction for video-to-audio synthesis. arXiv preprint arXiv:2409.06135, 2024

    2024 arXiv

  66. [74]

    Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners, 2024

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners, 2024

    2024

  67. [75]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models, 2023

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models, 2023

    2023

  68. [76]

    Efficient video to audio mapper with visual scene detection, 2024

    Mingjing Yi and Ming Li. Efficient video to audio mapper with visual scene detection, 2024. 13

    2024

  69. [77]

    Diverse and aligned audio-to-video generation via text-to-video model adaptation

    Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video generation via text-to-video model adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6639–6647, 2024

    2024

  70. [78]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  71. [79]

    Funasr: A fundamental end-to-end speech recognition toolkit

    Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. Funasr: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305.11013, 2023

    2023 arXiv

  72. [80]

    Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit

    Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit. arXiv preprint arXiv:2102.01547, 2021

    2021 arXiv

  73. [81]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  74. [82]

    Consistency flow matching: Defining straight flows with velocity consistency

    Ling Yang, Zixiang Zhang, Zhilong Zhang, Xingchao Liu, Minkai Xu, Wentao Zhang, Chenlin Meng, Stefano Ermon, and Bin Cui. Consistency flow matching: Defining straight flows with velocity consistency. arXiv preprint arXiv:2407.02398, 2024

    2024 arXiv

  75. [83]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  76. [84]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  77. [85]

    Text-to- audio generation using instruction tuned llm and latent diffusion model

    Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to- audio generation using instruction tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731, 2023

    2023 arXiv

  78. [86]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024

    2024 arXiv

  79. [87]

    LAION-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev....

    2022

  80. [88]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  81. [89]

    High fidelity neural audio compression

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  82. [90]

    Audiogen: Textually guided audio generation

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352, 2022

    2022 arXiv

  83. [91]

    Audiobox: Unified audio genera- tion with natural language prompts

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio genera- tion with natural language prompts. arXiv preprint arXiv:2312.15821, 2023

    2023 arXiv

  84. [92]

    Multi-stream transformers

    Mikhail Burtsev and Anna Rumshisky. Multi-stream transformers. arXiv preprint arXiv:2107.10342, 2021. 14

    2021 arXiv

  85. [93]

    Cantor: Inspiring mul- timodal chain-of-thought of mllm

    Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, Xing Sun, Liujuan Cao, et al. Cantor: Inspiring mul- timodal chain-of-thought of mllm. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9096–...

    2024

  86. [94]

    Iw-bench: Evaluating large multimodal models for converting image-to-web

    Hongcheng Guo, Wei Zhang, Junhao Chen, Yaonan Gu, Jian Yang, Junjia Du, Binyuan Hui, Tianyu Liu, Jianxin Ma, Chang Zhou, et al. Iw-bench: Evaluating large multimodal models for converting image-to-web. arXiv preprint arXiv:2409.18980, 2024

    2024

  87. [95]

    R-cot: Reverse chain-of-thought problem generation for geometric reasoning in large multimodal models

    Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, et al. R-cot: Reverse chain-of-thought problem generation for geometric reasoning in large multimodal models. arXiv preprint arXiv:2410.17885, 2024

    arXiv

  88. [96]

    Visual sketchpad: Sketching as a visual chain of thought for multimodal language models

    Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.09403, 2024

    2024 arXiv

  89. [97]

    Vision- language models are zero-shot reward models for reinforcement learning

    Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310.12921, 2023

    2023 arXiv

  90. [98]

    Furl: Visual-language models as fuzzy rewards for reinforcement learning

    Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet. Furl: Visual-language models as fuzzy rewards for reinforcement learning. arXiv preprint arXiv:2406.00645, 2024

    2024 arXiv

  91. [99]

    Rovrm: A robust visual reward model optimized via auxiliary textual preference data

    Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, et al. Rovrm: A robust visual reward model optimized via auxiliary textual preference data. arXiv preprint arXiv:2408.12109, 2024

    2024 arXiv

  92. [100]

    Rl-vlm-f: Reinforcement learning from vision language foundation model feedback

    Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. Rl-vlm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.03681, 2024

    2024 arXiv

  93. [101]

    in-the-wild

    Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103.16817, 2021

    2021 arXiv

  94. [102]

    Vision- language models as a source of rewards

    Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan, Gheorghe Comanici, Sebastian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, et al. Vision- language models as a source of rewards. arXiv preprint arXiv:2312.09187, 2023

    2023 arXiv

  95. [103]

    Auto mc-reward: Automated dense reward design with large language models for minecraft

    Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hong- sheng Li, Lewei Lu, and Jifeng Dai. Auto mc-reward: Automated dense reward design with large language models for minecraft. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  96. [104]

    Kam-cot: Knowledge augmented multimodal chain-of-thoughts reasoning

    Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cot: Knowledge augmented multimodal chain-of-thoughts reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18798–18806, 2024

    2024

  97. [105]

    Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168–5191, 2023

    2023

  98. [106]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023

    2023

  99. [107]

    Audiocaps: Gen- erating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume...

    2019

  100. [108]

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Lang...

    2024

  101. [109]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (...

    2017

  102. [110]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE, 2020

    2020

  103. [111]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15492–15501, 2024

    2024

  104. [112]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024

    2024 arXiv

  105. [113]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016

    2016

  106. [114]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  107. [115]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint arXiv:2110.08791, 2021

    2021 arXiv

  108. [116]

    Wav2clip: Learning robust audio representations from clip

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4563–4567. IEEE, 2022

    2022

  109. [117]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020

    2020

  110. [118]

    Comparison and analysis of deep audio embeddings for music emotion recognition

    Eunjeong Koh and Shlomo Dubnov. Comparison and analysis of deep audio embeddings for music emotion recognition. arXiv preprint arXiv:2104.06517, 2021

    2021 arXiv

  111. [119]

    Cavp: A context-aware vulnerability prioritization model

    Bill Jung, Yan Li, and Tamir Bechor. Cavp: A context-aware vulnerability prioritization model. Computers & Security, 116:102639, 2022. 16

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.