Pith. sign in

REVIEW 4 major objections 7 minor 66 references

HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HarmoniDPO claims that preference-optimized diffusion with dual video conditioning produces better-synchronized and more natural-sounding video-to-audio output than existing models.

desk verdict Solid dual-feature V2A model with a circular test-time search and an over-claimed 'human preference' story; deserves review, but not acceptance as is. read the letter →

arxiv 2608.11913 v1 pith:PH7ZJU7F submitted 2026-08-12 cs.CV

Wenshuo Peng , Kaipeng Zhang This is my paper · ORCID
classification cs.CV
keywords video-to-audiogenerationdiffusionmodelsdirectpreferenceoptimizationaudio-visualalignmentreinforcementlearningfromhumanfeedbacktest-timescalingdualvideorepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-to-audio generation usually struggles because single visual embeddings lose temporal detail and because reconstruction losses do not track what listeners like. The paper argues that both problems can be addressed in one framework: a diffusion model conditioned on both a global video embedding and per-frame visual features, then fine-tuned with an online, automated preference-optimization loop. The loop scores candidate audios with a composite reward built from audiovisual, audio-text, and intrinsic-quality measures, and it adds an inference-time search that refines candidate latents. If the claims hold, generated audio for a silent video becomes simultaneously better synchronized and more natural-sounding than output from previous approaches.

What carries the argument

The central mechanism is a two-stage pipeline. In the first stage, a pre-trained text-to-audio latent diffusion model is kept frozen and adapted to video by injecting a fused visual condition: a global video embedding and a set of frame-wise image embeddings are projected, concatenated, and processed by a self-attention layer with rotary position embeddings (position encodings that preserve frame order) before being fed into the denoising network through cross-attention. In the second stage, online-DPO repeatedly samples candidate audio clips from the current policy, scores them with the composite reward $R(y) = w_{\mathrm{av}}R'_{\mathrm{av}}(y) + w_{\mathrm{at}}R'_{\mathrm{at}}(y) + w_{\mathrm{quality}}R'_{\mathrm{quality}}(y)$ built from audiovisual, audio-text, and acoustic-quality scorers, selects the best and worst candidates, and updates the model with a value-aware DPO loss that includes the reward margin. At inference, Dual-scale Diffusion Search (DDS) maintains a population of latent candidates and proposes both small and large mixing steps, accepting whichever candidate scores higher. This machinery is what carries the paper's reported gains in synchronization and quality.

What would settle it

A held-out paired listening study comparing the aligned model with the unaligned base model would settle the claim: if human raters do not significantly prefer the aligned outputs, or if the outputs they prefer score lower on the composite reward, then the online-DPO stage is optimizing the wrong objective. An even more direct check is to compute the composite reward on human-chosen winners and losers and see whether the reward ordering matches the human ordering.

Watch

Extended reading notes

Core claim

On its own terms, HarmoniDPO's discovery is that preference-based optimization transfers to video-to-audio generation without human preference labels: the model generates candidate sounds itself, ranks them by an automated composite reward, and updates the policy with a Direct Preference Optimization loss that also includes the reward margin. The paper reports that on VGGSound the full configuration reaches MKL 1.82, CLIP 13.65, FID 6.42, FAD 1.59, and CLAP 32.57, beating every listed baseline on most metrics, and on AVSync15 it reports the best onset accuracy (32.53) and onset average precision (69.97) among the compared methods. The claim is that the dual video condition supplies the temporal synchronization while the online-DPO stage and Dual-scale Diffusion Search supply the perceptual quality.

Load-bearing premise

The preference stage rests on the assumption that a machine-computed score combining audiovisual alignment, text alignment, and acoustic quality is a faithful stand-in for human preference, because no human-annotated preference labels are used in training.

Editorial extensions

If this is right

  • The dual video representation alone, before any preference alignment, already outperforms all listed baseline systems on the reported VGGSound metrics.
  • Adding the online-DPO stage improves every reported automated metric and raises the human rating in the paper's user study, with eight candidate sounds giving the best balance.
  • Adding Dual-scale Diffusion Search at inference pushes the reported scores further, showing that test-time computation can substitute for some additional training.
  • The optional text prompt changes the generated sound in controllable ways, so the same video can be matched with different plausible audio styles.
  • The alignment gains also appear on AVSync15, the temporally hardest subset, where onset accuracy and onset average precision are highest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the automated reward design invites a test for reward hacking: if the audiovisual, audio-text, and quality scorers can be satisfied by audible artifacts, the alignment stage may improve scores without improving perceived audio.
  • Beyond the paper, DDS is a generic population-based search for latent diffusion; the same dual-scale accept-or-reject scheme could be applied to music generation, sound-effect synthesis, or other latent-diffusion modalities.
  • Beyond the paper, the paper's own limitation note about single-reference metrics implies that reference-based scores such as MKL and FID may systematically penalize valid alternative sounds, so paired human comparisons are the more decisive evaluation for one-to-many video-to-audio mapping.
  • Beyond the paper, a practical extension is to make the number of preference candidates adaptive: the reported trade-off between CLAP and FID/FAD suggests an automatic stopping rule based on reward-margin variance could replace the fixed eight-candidate setting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes HarmoniDPO, a video-to-audio (V2A) generation framework with three main components: (1) a dual video representation that combines a global InternVid video embedding with per-frame CLIP image embeddings, fused through RoPE-based self-attention and injected into a frozen Tango-2 latent diffusion model; (2) an "online-DPO" alignment stage that generates preference pairs from the current policy, ranks them with a composite automated reward R(y) built from normalized CAV-MAE, CLAP, and Audiobox-Aesthetics scores, and trains with a value-aware DPO loss; and (3) a test-time "Dual-scale Diffusion Search" (DDS) that evolves a population of candidates via two Gaussian perturbation step sizes and selects by CLIP/CLAP fitness. Experiments on VGGSound and AVSync15 report consistent improvements over SpecVQGAN, Diff-Foley, V2A-Mapper, FoleyCrafter, and Frieren across MKL, CLIP, FID, FAD, CLAP, and onset metrics, plus a small laboratory user study.

Significance. If the reported results are reproducible, the dual-conditioning design and the iterative self-training with automated rewards are useful contributions: the aligned model alone (without DDS) already beats the strongest baseline Frieren on all five VGGSound metrics in Table 1, and the paper reports a reasonably complete set of standard metrics plus onset-based synchronization measures. The paper does not ship code, but the algorithm descriptions are mostly concrete. However, the significance is reduced by two load-bearing issues: the training signal is not human preference but a weighted sum of automated metrics, so the RLHF-style claim in the abstract and introduction is overstated; and the DDS inference search directly optimizes CLIP/CLAP scores that are also reported as headline metrics, making those columns partially circular. The independent MKL/FID/FAD improvements and the user study provide some non-circular evidence, so the contribution is not void, but the paper requires substantial revision.

major comments (4)
  1. [§3.4, Eqs. (22)-(23), Algorithm 1] The DDS procedure is described as a "diffusion search," but the update equations are Gaussian perturbations, x_s = β_s x + sqrt(1-β_s^2) η and x_l = β_l x + sqrt(1-β_l^2) η, with no call to the denoising network ε_θ, no reverse-process update, and no VAE decoding step before evaluating the fitness F. As written, the candidates P_0 are drawn from N(0,1), and if these are raw latents the CLIP/CLAP objective cannot be computed without a decoding step; if they are waveforms, the equations are not diffusion updates. The reported DDS gains in Table 1 therefore cannot be reproduced from the manuscript, and the name "Dual-scale Diffusion Search" is not justified by the stated algorithm.
  2. [§3.4, Table 1] The DDS fitness function F uses "the CLIP score to measure the audio-visual correspondence and the CLAP score to assess the audio-text alignment," and Table 1 reports CLIP and CLAP as headline evaluation metrics. The row "HarmoniDPO (aligned + DDS)" reports CLIP 13.65 and CLAP 32.57 after a search that directly maximizes those quantities, so these two columns do not provide independent evidence of synchronization or semantic alignment. The simultaneous improvements in MKL, FID, and FAD are not explained by the stated algorithm, since F does not include those metrics. Please re-evaluate DDS with held-out or non-optimized metrics, or report the DDS results with CLIP/CLAP columns removed and explain how the non-circular gains arise.
  3. [§3.3.2, Eq. (19), abstract] The abstract and introduction claim that HarmoniDPO learns "from preference judgments" and generates "human-preferred audio," but Section 3.3.2 explicitly states that the method "uses no human-annotated preferences in the training loop." The reward R(y) in Eq. (19) is a weighted combination of automated scores from CAV-MAE, CLAP, and Audiobox-Aesthetics, and the preference pairs in Eq. (20) are selected by these automated scores. The RLHF-inspired framing therefore overstates the role of human feedback. The paper should either temper the language throughout or provide evidence that the automated composite reward tracks human perceptual judgments (for example, a correlation study between R(y) and human ratings, or a baseline comparison in the user study).
  4. [§3.3.2, Eq. (21)] The DPO derivation in Eqs. (15)-(18) assumes a fixed reference policy π_ref, because the reward reparameterization in Eq. (16) relies on a constant partition function Z(v). The paper states that "regarding the reference model π_ref, we update it periodically; specifically, at the end of each training epoch," but does not justify this choice or analyze its effect. Periodically replacing π_ref changes the implicit reward and means the loss in Eq. (21) is no longer the DPO objective. Please either provide a formal justification for the periodic update or compare it experimentally against a fixed π_ref.
minor comments (7)
  1. [Table 4] The row for 4 candidates is misformatted as "4 1.8312.95 7.76 1.73 30.54", making the values hard to read.
  2. [§4.2 and Algorithm 1] The sentence "We tested" is an incomplete fragment, and Algorithm 1 uses the abbreviation "DSS" in its title while the text and elsewhere use "DDS".
  3. [Title page and §3.3.2] There are typos: "Electornic" should be "Electronic," and "onlin-DPO" should be "online-DPO."
  4. [Table 1 and References] Table 1 lists "Fieren" while the reference list and text use "Frieren"; please standardize the spelling.
  5. [§2.1] The sentence "Diffusion models have achieve remarkable cross-model generation ability" should read "have achieved."
  6. [§4.2, User study] The phrase "comprehensive human evaluation" overstates the scale of the user study, which involved eight laboratory evaluators; moreover, the study compares candidate counts for the online-DPO alignment but does not compare the full HarmoniDPO model against the baselines in Table 1, so it provides only partial support for the subjective-quality claims.
  7. [Fig. 6 caption] The caption begins with a sentence fragment, "Using Online-DPO method can effectively attenuate ambient noise"; consider rewriting as a complete sentence.

Circularity Check

2 steps flagged · score 6.0 of 10

DDS directly maximizes the CLIP and CLAP scores that are later reported as headline evaluation metrics, and the online-DPO reward includes CLAP, so part of the claimed synchronization/alignment gains reduce by construction.

  1. self definitional [Section 3.4 "Test-time Scaling" and Algorithm 1; Section 4 "Metrics"; Table 1, row "HarmoniDPO (aligned + DDS)".]
    "To evaluate the quality of the generated audio, we employ a zero-shot evaluation approach. Specifically, we utilize the CLIP score [57] to measure the audio-visual correspondence and the CLAP score to assess the audio-text alignment. ... CLIP Similarity to evaluate the semantic coherence between the input video and the generated audio embeddings, using Wav2CLIP[57] as the audio encoder and CLIP as the video encoder ... CLAP Similarity to evaluate the cross-modal alignment between text and generated audio."

    Algorithm 1 selects candidates x_s or x_l solely by comparing F(x) to the current best score, and the text defines F using CLIP and CLAP scores. The headline row 'HarmoniDPO (aligned + DDS)' then reports CLIP 13.65 and CLAP 32.57 as evidence of synchronization and semantic alignment. Those two columns are exactly the quantities the search maximizes, so the DDS-induced gains are by construction rather than independent confirmation. MKL, FID, FAD, and onset metrics are not part of F, so their reported improvement is not explained by the stated algorithm.

  2. fitted input called prediction [Section 3.3.2 "Reward Modeling", Eq. (19); Section 4 "Metrics"; Table 1, row "HarmoniDPO (aligned)".]
    "audio-text consistency is assessed using the similarity score derived from a dedicated audio-text contrastive learning model CLAP. ... R(y) = w_av·R'_av(y) + w_at·R'_at(y) + w_quality·R'_quality(y). ... CLAP Similarity to evaluate the cross-modal alignment between text and generated audio."

    The online-DPO training reward explicitly includes a CLAP similarity term (R'_at), and the same CLAP similarity is later reported as a headline evaluation metric. The aligned model's CLAP improvement (Table 1: 28.35 to 31.75) is therefore partly forced by the training objective that selected preferred samples. This is partial circularity because the composite reward also contains CAV-MAE and Audiobox-Aesthetics terms, and the tables separately report independent MKL, FID, FAD, and onset numbers.

full rationale

The core architecture is self-contained: the dual video representation (InternVid global features plus CLIP frame features), the RoPE self-attention fusion, the Tango-2 base model, and the Diffusion-DPO loss are stated with explicit equations and do not depend on any self-citation for their content. The authors' only self-citation, reference [14], is a non-load-bearing related-work note. However, two evaluation claims partially reduce to the optimized objective. DDS is an inference-time search whose objective is defined by CLIP and CLAP scores, and the same scores are the headline CLIP and CLAP columns in Table 1; the 'HarmoniDPO (aligned + DDS)' row cannot use those columns as independent evidence of audio-video synchronization or text alignment. Likewise, the online-DPO reward in Eq. (19) includes a CLAP similarity term, so the CLAP gain from alignment is in part a fitted-input result. These circularities do not void the whole paper: MKL, FID, FAD, Onset Acc/AP, the AVSync15 results, and the lab user study are not part of F or the CLAP reward term and provide independent evidence; the dual-feature ablation in Table 5 also stands on its own. Separately, the DDS description is incomplete as written: the update in Eqs. (22)-(23) is a Gaussian perturbation of the candidate, with no call to the denoising network epsilon_theta, reverse-process step, or VAE decode before scoring, so the algorithm cannot be executed as described. That is a completeness/correctness defect, not an additional circularity. Overall partial circularity score 6.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; VA-DPO and DDS are algorithmic components, not entities. The free parameters are mostly unspecified or hand-chosen, and the key axiom is that automated metric scores can stand in for human preferences.

free parameters (6)
  • Reward weights (w_av, w_at, w_quality) = 1, 1, 1
    Chosen by hand as equal weighting in Eq. (19); the paper does not justify this balance.
  • VA-DPO margin coefficient λ = not reported
    Balances the DPO term and the reward-margin term in Eq. (21); value is not given.
  • DPO temperature β = not reported
    Temperature in Eq. (21); not specified.
  • DDS step sizes β_s, β_l = not reported
    Parameters of the dual-scale search in Eqs. (22)-(23); not reported.
  • DDS search iterations T and population D = not reported; candidate count N set to 4 in experiments
    Algorithm 1 parameters T, D, N; only N is mentioned in the experiments.
  • Number of DPO candidate samples = 8 (tuned)
    Table 4 scans 2, 4, 8, 12 candidates and picks 8 as the best balance; this is a hand-chosen hyperparameter, not a predicted quantity.
assumptions (5)
  • standard math Bradley-Terry model of preferences and KL-constrained optimal policy form base of DPO (Section 3.3.1).
    Standard DPO derivation reused without new proof.
  • domain assumption The composite automated reward R(y) is a valid proxy for human perceptual preference (Section 3.3.2, Eq. (19)).
    If CAV-MAE/CLAP/Audiobox-Aesthetics scores do not track human opinion, the alignment stage does not deliver the claimed subjective gains.
  • domain assumption Tango-2 serves as an effective frozen base for V2A, with the added visual cross-attention sufficient to condition audio generation on video (Section 3.1-3.2).
    No ablation of the frozen-base choice is provided.
  • ad hoc to paper Periodically replacing π_ref at the end of each epoch preserves the DPO objective (Section 3.3.2).
    The theoretical derivation in Eqs. (15)-(18) assumes a fixed reference policy; the paper does not justify the update.
  • ad hoc to paper The perturbation update in DDS, x_s = β_s x + sqrt(1-β_s^2) η, acts as a valid diffusion search direction (Eq. (22)-(23)).
    The update is a Gaussian convolution, not a reverse-diffusion denoising step; no formal connection is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion." pith.science (2026). https://pith.science/paper/PH7ZJU7F

@misc{pith2026260811913,
  author       = {Pith},
  title        = {Pith review of: HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PH7ZJU7F}},
  note         = {Machine review of arXiv:2608.11913}
}
read the original abstract

Video-to-audio (V2A) generation faces significant challenges in achieving precise temporal synchronization and high perceptual quality due to the complex, ambiguous relationship between visual and auditory cues. Existing methods typically compress video inputs into single feature representations, leading to significant loss of temporal dynamics and fine-grained visual information. These approaches also rely on reconstruction-based training objectives that poorly correlate with human perceptual judgments of audio quality and appropriateness. We propose HarmoniDPO, a novel framework that integrates preference-based optimization into diffusion-based V2A generation to address these limitations. (1) Our approach leverages a dual video representation: combining global context with frame-wise features to preserve temporal dynamics and semantic detail. (2) Inspired by reinforcement learning from human feedback (RLHF), HarmoniDPO employs online Direct Preference Optimization (online-DPO) to fine-tune a diffusion-based V2A model from preference judgments, enhancing perceptual quality and alignment. (3) Additionally, we introduce Dual-scale Diffusion Search (DDS), a test time scaling algorithm that adaptively optimizes output fidelity during inference. Experiments demonstrate that HarmoniDPO outperforms state-of-the-art methods in audio-video synchronization and subjective audio quality, offering a robust solution for generating realistic, human-preferred audio from video.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 38 canonical work pages

  1. [1]

    arXiv preprint arXiv:2209.15352 (2022)

    Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., D´ efossez, A., Copet, J., Parikh, D., Taigman, Y., Adi, Y.: Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352 (2022)

    arXiv 2022

  2. [2]

    In: International Conference on Machine Learning, pp

    Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., Zhao, Z.: Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In: International Conference on Machine Learning, pp. 13916– 13932 (2023). PMLR

    work page 2023

  3. [3]

    arXiv preprint arXiv:2301.12503 (2023)

    Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., Plumbley, M.D.: Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 (2023)

    arXiv 2023

  4. [4]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., Shi, H.: Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954–15964 (2023)

    work page 2023

  5. [5]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.- B., Liu, M.-Y., Balaji, Y.: Preserve your own correlation: A noise prior for video diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22930–22941 (2023)

    work page 2023

  6. [6]

    arXiv preprint arXiv:2407.01494 (2024)

    Zhang, Y., Gu, Y., Zeng, Y., Xing, Z., Wang, Y., Wu, Z., Chen, K.: Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494 (2024)

    arXiv 2024

  7. [7]

    Advances in Neural Information Processing Systems36(2024)

    Luo, S., Yan, C., Hu, C., Zhao, H.: Diff-foley: Synchronized video-to-audio syn- thesis with latent diffusion models. Advances in Neural Information Processing Systems36(2024)

    work page 2024

  8. [8]

    In: 25 Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Wang, H., Ma, J., Pascual, S., Cartwright, R., Cai, W.: V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In: 25 Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 15492–15501 (2024)

    work page 2024

Show all 66 references
  1. [9]

    arXiv preprint arXiv:2501.09732 (2025)

    Ma, N., Tong, S., Jia, H., Hu, H., Su, Y.-C., Zhang, M., Yang, X., Li, Y., Jaakkola, T., Jia, X., et al.: Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732 (2025)

    2025 arXiv

  2. [10]

    arXiv preprint arXiv:2005.00341 (2020)

    Dhariwal, P., Jun, H., Payne, C., Kim, J.W., Radford, A., Sutskever, I.: Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341 (2020)

    2020 arXiv

  3. [11]

    Advances in Neural Information Processing Systems36(2024)

    Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., D´ efossez, A.: Simple and controllable music generation. Advances in Neural Information Processing Systems36(2024)

    2024

  4. [12]

    Advances in neural information processing systems36, 14005–14034 (2023)

    Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J.,et al.: Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems36, 14005–14034 (2023)

    2023

  5. [13]

    arXiv preprint arXiv:2401.04577 (2024)

    Ziv, A., Gat, I., Lan, G.L., Remez, T., Kreuk, F., D´ efossez, A., Copet, J., Syn- naeve, G., Adi, Y.: Masked audio generation using a single non-autoregressive transformer. arXiv preprint arXiv:2401.04577 (2024)

    2024 arXiv

  6. [14]

    In: NAACL-HLT (Findings) (2024)

    Peng, W., Zhang, K., Zhang, S.Q.: T3m: Text guided 3d human motion synthesis from speech. In: NAACL-HLT (Findings) (2024)

    2024

  7. [15]

    arXiv preprint arXiv:2503.10634 (2025)

    Zhang, Y., Chen, J.-K., Lyu, J., Wang, Y.-X.: V2edit: Versatile video diffusion editor for videos and 3d scenes. arXiv preprint arXiv:2503.10634 (2025)

    2025 arXiv

  8. [16]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing31, 1720–1733 (2023)

    Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., Yu, D.: Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing31, 1720–1733 (2023)

    2023

  9. [17]

    In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp

    Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., Dubnov, S.: Large- scale contrastive language-audio pretraining with feature fusion and keyword-to- caption augmentation. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2023

  10. [18]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

    Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y., Wang, W., Wang, Y., Plumbley, M.D.: Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

    2024

  11. [19]

    In: Forty-first International Conference on Machine Learning (2024) 26

    Evans, Z., Carr, C., Taylor, J., Hawley, S.H., Pons, J.: Fast timing-conditioned latent audio diffusion. In: Forty-first International Conference on Machine Learning (2024) 26

    2024

  12. [20]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E.H., Freeman, W.T.: Visually indicated sounds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2405–2413 (2016)

    2016

  13. [21]

    In: British Machine Vision Conference (BMVC) (2021)

    Iashin, V., Rahtu, E.: Taming visually guided sound generation. In: British Machine Vision Conference (BMVC) (2021)

    2021

  14. [22]

    In: European Conference on Computer Vision, pp

    Gan, C., Huang, D., Chen, P., Tenenbaum, J.B., Torralba, A.: Foley music: Learn- ing to generate music from videos. In: European Conference on Computer Vision, pp. 758–775 (2020). Springer

    2020

  15. [23]

    arXiv preprint arXiv:2308.06721 (2023)

    Ye, H., Zhang, J., Liu, S., Han, X., Yang, W.: Ip-adapter: Text compati- ble image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 (2023)

    2023 arXiv

  16. [24]

    Advances in Neural Information Processing Systems37, 128118–128138 (2024)

    Wang, Y., Guo, W., Huang, R., Huang, J., Wang, Z., You, F., Li, R., Zhao, Z.: Frieren: Efficient video-to-audio generation network with rectified flow matching. Advances in Neural Information Processing Systems37, 128118–128138 (2024)

    2024

  17. [25]

    In: European Conference on Computer Vision, pp

    Pascual, S., Yeh, C., Tsiamas, I., Serr` a, J.: Masked generative video-to-audio transformers with enhanced synchronicity. In: European Conference on Computer Vision, pp. 247–264 (2024). Springer

    2024

  18. [26]

    In: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp

    Viertola, I., Iashin, V., Rahtu, E.: Temporally aligned audio for video with autore- gression. In: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5 (2025). IEEE

    2025

  19. [27]

    Expert Systems with Applications249, 123640 (2024)

    Kang, J., Poria, S., Herremans, D.: Video2music: Suitable music generation from videos using an affective multimodal transformer model. Expert Systems with Applications249, 123640 (2024)

    2024

  20. [28]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Tian, Z., Liu, Z., Yuan, R., Pan, J., Liu, Q., Tan, X., Chen, Q., Xue, W., Guo, Y.: Vidmuse: A simple video-to-music generation framework with long-short- term modeling. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18782–18793 (2025)

    2025

  21. [29]

    In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp

    Cui, C., Zhao, Z., Ren, Y., Liu, J., Huang, R., Chen, F., Wang, Z., Huai, B., Wu, F.: Varietysound: Timbre-controllable video to sound generation via unsu- pervised information disentanglement. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal ...

    2023

  22. [30]

    In: Proceedings of the 32nd ACM International Conference on Multimedia, pp

    Wang, X., Wang, Y., Wu, Y., Song, R., Tan, X., Chen, Z., Xu, H., Sui, G.: Tiva: Time-aligned video-to-audio generation. In: Proceedings of the 32nd ACM International Conference on Multimedia, pp. 573–582 (2024)

    2024

  23. [31]

    In: Pro- ceedings of the AAAI Conference on Artificial Intelligence, vol

    Li, B., Yang, F., Mao, Y., Ye, Q., Chen, H., Zhong, Y.: Tri-ergon: Fine-grained 27 video-to-audio generation with multi-modal conditions and lufs control. In: Pro- ceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 4616–4624 (2025)

    2025

  24. [32]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Jeong, Y., Kim, Y., Chun, S., Lee, J.: Read, watch and scream! sound genera- tion from text and video. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 17590–17598 (2025)

    2025

  25. [33]

    In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp

    Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio-visual dataset. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721–725 (2020). IEEE

    2020

  26. [34]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X.,et al.: Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp....

    2022

  27. [35]

    arXiv preprint arXiv:2406.14272 (2024)

    Sung-Bin, K., Chae-Yeon, L., Son, G., Hyun-Bin, O., Ju, J., Nam, S., Oh, T.-H.: Multitalk: Enhancing 3d talking head generation across languages with multilingual video dataset. arXiv preprint arXiv:2406.14272 (2024)

    2024 arXiv

  28. [36]

    arXiv preprint arXiv:2411.18953 (2024)

    Bai, J., Liu, H., Wang, M., Shi, D., Wang, W., Plumbley, M.D., Gan, W.-S., Chen, J.: Audiosetcaps: An enriched audio-caption dataset using automated generation pipeline with large audio and language models. arXiv preprint arXiv:2411.18953 (2024)

    2024 arXiv

  29. [37]

    Sudarsanam, P., Mart´ ın-Morat´ o, I., Hakala, A., Virtanen, T.: Avcaps: An audio- visual dataset with modality-specific captions

  30. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., Hu, D.: Learning to answer questions in dynamic audio-visual scenarios. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108–19118 (2022)

    2022

  31. [39]

    Advances in neural information processing systems30(2017)

    Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Advances in neural information processing systems30(2017)

    2017

  32. [40]

    Advances in neural information processing systems33, 3008–3021 (2020)

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P.F.: Learning to summarize with human feedback. Advances in neural information processing systems33, 3008–3021 (2020)

    2020

  33. [41]

    Advances in neural information processing systems35, 27730–27744 (2022)

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.,et al.: Training language models to follow instructions with human feedback. Advances in neural information processing systems35, 27730–27744 (2022)

    2022

  34. [42]

    Advances in Neural Information Processing Systems36, 53728–53741 (2023)

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: 28 Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems36, 53728–53741 (2023)

    2023

  35. [43]

    arXiv preprint arXiv:2304.06767 (2023)

    Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., Zhang, T.: Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 (2023)

    2023 arXiv

  36. [44]

    arXiv preprint arXiv:2304.05302 (2023)

    Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., Huang, F.: Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302 (2023)

    2023 arXiv

  37. [45]

    arXiv preprint arXiv:2212.08073 (2022)

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.: Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 (2022)

    2022 arXiv

  38. [46]

    Advances in Neural Information Processing Systems36, 15903–15935 (2023)

    Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: Imagere- ward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems36, 15903–15935 (2023)

    2023

  39. [47]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., Naik, N.: Diffusion model alignment using direct preference optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228–8...

    2024

  40. [48]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    2022

  41. [49]

    Advances in neural information processing systems33, 6840–6851 (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)

    2020

  42. [50]

    In: Proceedings of the 32nd ACM International Conference on Multimedia, pp

    Majumder, N., Hung, C.-Y., Ghosal, D., Hsu, W.-N., Mihalcea, R., Poria, S.: Tango 2: Aligning diffusion-based text-to-audio generations through direct pref- erence optimization. In: Proceedings of the 32nd ACM International Conference on Multimedia, pp. 564–572 (2024)

    2024

  43. [51]

    arXiv preprint arXiv:2307.06942 (2023)

    Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al.: Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 (2023)

    2023 arXiv

  44. [52]

    Neurocomputing568, 127063 (2024)

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced transformer with rotary position embedding. Neurocomputing568, 127063 (2024)

    2024

  45. [53]

    Journal of Machine Learning Research25(70), 1–53 (2024)

    Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S.,et al.: Scaling instruction-finetuned language 29 models. Journal of Machine Learning Research25(70), 1–53 (2024)

    2024

  46. [54]

    arXiv preprint arXiv:2408.02666 (2024)

    Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi-Yu, J., Pang, R.Y., Fazel-Zarandi, M., Weston, J., Li, X.: Self-taught evaluators. arXiv preprint arXiv:2408.02666 (2024)

    2024 arXiv

  47. [55]

    arXiv preprint arXiv:2210.07839 (2022)

    Gong, Y., Rouditchenko, A., Liu, A.H., Harwath, D., Karlinsky, L., Kuehne, H., Glass, J.: Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839 (2022)

    2022 arXiv

  48. [56]

    arXiv preprint arXiv:2502.05139 (2025)

    Tjandra, A., Wu, Y.-C., Guo, B., Hoffman, J., Ellis, B., Vyas, A., Shi, B., Chen, S., Le, M., Zacharov, N., et al.: Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139 (2025)

    2025 arXiv

  49. [57]

    In: ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp

    Wu, H.-H., Seetharaman, P., Kumar, K., Bello, J.P.: Wav2clip: Learning robust audio representations from clip. In: ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563–4567 (2022). IEEE

    2022

  50. [58]

    arXiv preprint arXiv:2403.05659 (2024)

    Zhang, L., Mo, S., Zhang, Y., Morgado, P.: Audio-synchronized visual animation. arXiv preprint arXiv:2403.05659 (2024)

    2024 arXiv

  51. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for con- trastive language-image learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp...

    2023

  52. [60]

    arXiv preprint arXiv:2110.08791 (2021)

    Iashin, V., Rahtu, E.: Taming visually guided sound generation. arXiv preprint arXiv:2110.08791 (2021)

    2021 arXiv

  53. [61]

    Advances in neural information processing systems30(2017)

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems30(2017)

    2017

  54. [62]

    arXiv preprint arXiv:1812.08466 (2018)

    Kilgour, K., Zuluaga, M., Roblek, D., Sharifi, M.: Fr\’echet audio distance: A met- ric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 (2018)

    2018 arXiv

  55. [63]

    In: 2017 Ieee International Conference on Acoustics, Speech and Signal Processing (icassp), pp

    Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B.,et al.: Cnn architec- tures for large-scale audio classification. In: 2017 Ieee International Conference on Acoustics, Speech and Signal Processin...

    2017

  56. [64]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Xie, Z., Yu, S., He, Q., Li, M.: Sonicvisionlm: Playing sound with vision language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26866–26875 (2024) 30

    2024

  57. [65]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Xing, Y., He, Y., Tian, Z., Wang, X., Chen, Q.: Seeing and hearing: Open- domain visual-audio generation with diffusion latent aligners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7151–7161 (2024)

    2024

  58. [66]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180– 15190 (2023) 31

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.