{"id":"510eb343-578d-4c4e-a017-d3df13e954d2","arxiv_id":"2608.11913","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"HarmoniDPO pairs global and frame-level video features with preference-style optimization to generate audio from silent video, reporting improved synchronization and quality metrics over prior V2A baselines.","lead":"HarmoniDPO generates audio from silent video by combining global video features with frame-level details and then fine-tunes the model with automated quality scores through Direct Preference Optimization and a test-time search. Why read it: it promises more synchronized and more natural-sounding video-to-audio generation, although the training signal is a metric proxy rather than human feedback.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DDS inference search directly maximizes the CLIP/CLAP metrics used as headline evaluation, and its stated update equations are Gaussian perturbations with no specified diffusion or decoding step, so the claimed synchronization and quality gains are not independently established.","rationale":"The reader's weakest_assumption focuses on whether the automated composite reward in Eq. 19 captures human preferences, and on the unexamined periodic update of pi_ref. Both are real concerns. However, the most load-bearing problem for the paper's central empirical claim is the DDS inference procedure: it directly optimizes two of the metrics used for evaluation, and its update equations do not describe a valid diffusion or decoding process. This is not a matter of disagreeing with a consensus or of theoretical taste; it is an internal correctness issue that affects the headline numbers. The base model without alignment and the aligned model without DDS also improve on several baselines, so the core V2A model may have genuine merit. Those configurations should be credited on their own terms. But the paper's strongest claim is tied to the DDS row, and that row cannot be interpreted as evidence until the circularity is removed and the algorithm is specified with an actual decoding step. The user study is too small, too lightly described, and too author-affiliated to independently rescue the subjective-quality claim. For these reasons, the rejection stands, and the concrete test would determine whether a resubmission can restore the empirical case.","tokens_in":19154,"tokens_out":3538,"duration_ms":39226,"concrete_test":"Re-run the DDS-enabled configuration with the search objective F replaced by a reward that is not part of the evaluation suite, for example Audiobox-Aesthetics only or a held-out human-preference proxy, and report CLIP, CLAP, MKL, FID, FAD, Onset ACC, and Onset AP on the same test set. If the CLIP/CLAP gains shrink or disappear while the other metrics do not improve, the headline gains are an artifact of optimizing the evaluation metrics. Additionally, instrument Algorithm 1 to log whether each candidate is passed through the diffusion denoiser and VAE decoder before scoring; if candidates are scored without any decoding step, DDS as written cannot produce the reported audio outputs.","verdict_should_be":"REJECT","load_bearing_attack":"The central empirical claim rests on the best configuration, 'HarmoniDPO (aligned + DDS)', whose numbers are produced by a search procedure that is circular with respect to evaluation. In Section 3.4 and Algorithm 1, DDS initializes a population from N(0,1), then updates candidates as x_s = beta_s * x + sqrt(1 - beta_s^2) * eta and x_l = beta_l * x + sqrt(1 - beta_l^2) * eta (Eqs. 22-23). These are Gaussian perturbation moves, not diffusion denoising steps: no call to the denoising network epsilon_theta, no reverse-process update, and no VAE decoding step is specified before the objective F is evaluated. The paper states that F uses 'the CLIP score to measure the audio-visual correspondence and the CLAP score to assess the audio-text alignment.' CLIP and CLAP are exactly two of the headline metrics in Tables 1 and 4. The row 'HarmoniDPO (aligned + DDS)' reports CLIP 13.65 and CLAP 32.57 after a search that directly maximizes those quantities, so these columns do not provide independent evidence of synchronization or semantic alignment. The simultaneous improvements in MKL, FID, and FAD are not explained by the stated algorithm, since F does not include those metrics. The central claim about outperforming prior work therefore relies on a circular evaluation, and the algorithmic description is internally incomplete as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HarmoniDPO, a video-to-audio (V2A) generation framework with three main components: (1) a dual video representation that combines a global InternVid video embedding with per-frame CLIP image embeddings, fused through RoPE-based self-attention and injected into a frozen Tango-2 latent diffusion model; (2) an \"online-DPO\" alignment stage that generates preference pairs from the current policy, ranks them with a composite automated reward R(y) built from normalized CAV-MAE, CLAP, and Audiobox-Aesthetics scores, and trains with a value-aware DPO loss; and (3) a test-time \"Dual-scale Diffusion Search\" (DDS) that evolves a population of candidates via two Gaussian perturbation step sizes and selects by CLIP/CLAP fitness. Experiments on VGGSound and AVSync15 report consistent improvements over SpecVQGAN, Diff-Foley, V2A-Mapper, FoleyCrafter, and Frieren across MKL, CLIP, FID, FAD, CLAP, and onset metrics, plus a small laboratory user study.","tokens_in":19478,"tokens_out":5325,"duration_ms":52546,"significance":"If the reported results are reproducible, the dual-conditioning design and the iterative self-training with automated rewards are useful contributions: the aligned model alone (without DDS) already beats the strongest baseline Frieren on all five VGGSound metrics in Table 1, and the paper reports a reasonably complete set of standard metrics plus onset-based synchronization measures. The paper does not ship code, but the algorithm descriptions are mostly concrete. However, the significance is reduced by two load-bearing issues: the training signal is not human preference but a weighted sum of automated metrics, so the RLHF-style claim in the abstract and introduction is overstated; and the DDS inference search directly optimizes CLIP/CLAP scores that are also reported as headline metrics, making those columns partially circular. The independent MKL/FID/FAD improvements and the user study provide some non-circular evidence, so the contribution is not void, but the paper requires substantial revision.","major_comments":[{"comment":"The DDS procedure is described as a \"diffusion search,\" but the update equations are Gaussian perturbations, x_s = β_s x + sqrt(1-β_s^2) η and x_l = β_l x + sqrt(1-β_l^2) η, with no call to the denoising network ε_θ, no reverse-process update, and no VAE decoding step before evaluating the fitness F. As written, the candidates P_0 are drawn from N(0,1), and if these are raw latents the CLIP/CLAP objective cannot be computed without a decoding step; if they are waveforms, the equations are not diffusion updates. The reported DDS gains in Table 1 therefore cannot be reproduced from the manuscript, and the name \"Dual-scale Diffusion Search\" is not justified by the stated algorithm.","section":"§3.4, Eqs. (22)-(23), Algorithm 1"},{"comment":"The DDS fitness function F uses \"the CLIP score to measure the audio-visual correspondence and the CLAP score to assess the audio-text alignment,\" and Table 1 reports CLIP and CLAP as headline evaluation metrics. The row \"HarmoniDPO (aligned + DDS)\" reports CLIP 13.65 and CLAP 32.57 after a search that directly maximizes those quantities, so these two columns do not provide independent evidence of synchronization or semantic alignment. The simultaneous improvements in MKL, FID, and FAD are not explained by the stated algorithm, since F does not include those metrics. Please re-evaluate DDS with held-out or non-optimized metrics, or report the DDS results with CLIP/CLAP columns removed and explain how the non-circular gains arise.","section":"§3.4, Table 1"},{"comment":"The abstract and introduction claim that HarmoniDPO learns \"from preference judgments\" and generates \"human-preferred audio,\" but Section 3.3.2 explicitly states that the method \"uses no human-annotated preferences in the training loop.\" The reward R(y) in Eq. (19) is a weighted combination of automated scores from CAV-MAE, CLAP, and Audiobox-Aesthetics, and the preference pairs in Eq. (20) are selected by these automated scores. The RLHF-inspired framing therefore overstates the role of human feedback. The paper should either temper the language throughout or provide evidence that the automated composite reward tracks human perceptual judgments (for example, a correlation study between R(y) and human ratings, or a baseline comparison in the user study).","section":"§3.3.2, Eq. (19), abstract"},{"comment":"The DPO derivation in Eqs. (15)-(18) assumes a fixed reference policy π_ref, because the reward reparameterization in Eq. (16) relies on a constant partition function Z(v). The paper states that \"regarding the reference model π_ref, we update it periodically; specifically, at the end of each training epoch,\" but does not justify this choice or analyze its effect. Periodically replacing π_ref changes the implicit reward and means the loss in Eq. (21) is no longer the DPO objective. Please either provide a formal justification for the periodic update or compare it experimentally against a fixed π_ref.","section":"§3.3.2, Eq. (21)"}],"minor_comments":[{"comment":"The row for 4 candidates is misformatted as \"4 1.8312.95 7.76 1.73 30.54\", making the values hard to read.","section":"Table 4"},{"comment":"The sentence \"We tested\" is an incomplete fragment, and Algorithm 1 uses the abbreviation \"DSS\" in its title while the text and elsewhere use \"DDS\".","section":"§4.2 and Algorithm 1"},{"comment":"There are typos: \"Electornic\" should be \"Electronic,\" and \"onlin-DPO\" should be \"online-DPO.\"","section":"Title page and §3.3.2"},{"comment":"Table 1 lists \"Fieren\" while the reference list and text use \"Frieren\"; please standardize the spelling.","section":"Table 1 and References"},{"comment":"The sentence \"Diffusion models have achieve remarkable cross-model generation ability\" should read \"have achieved.\"","section":"§2.1"},{"comment":"The phrase \"comprehensive human evaluation\" overstates the scale of the user study, which involved eight laboratory evaluators; moreover, the study compares candidate counts for the online-DPO alignment but does not compare the full HarmoniDPO model against the baselines in Table 1, so it provides only partial support for the subjective-quality claims.","section":"§4.2, User study"},{"comment":"The caption begins with a sentence fragment, \"Using Online-DPO method can effectively attenuate ambient noise\"; consider rewriting as a complete sentence.","section":"Fig. 6 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper would benefit from a clearer positioning relative to Tango-2 [50] and Diffusion-DPO [47]; the claim of being the first online-DPO for V2A is plausible, but the automated-reward setup borrows heavily from self-taught evaluators [54] and should be cited more carefully. I also note that no code is released, which makes the missing DDS decoding details more consequential than they would otherwise be. The central architectural contribution seems sound, but the empirical claims need re-evaluation after addressing the circularity and the DDS specification issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This is a real engineering contribution with a load-bearing validity problem in the headline numbers, and the 'human preference' framing does not match what the method actually does. I'd send it to review, but I would reject the current version.\n\nWhat's new and good: the dual-feature video conditioning (InternVid global plus CLIP frame features) is well motivated, and the ablation shows it helps across MKL, CLIP, and FID. The base model without any alignment already beats strong baselines like FoleyCrafter and Frieren on VGGSound. That alone is a meaningful result. The idea of online-DPO with an automated multi-reward signal is also a reasonable way to get preference-style alignment without expensive human labels, and the number-of-candidates study is a decent effort at characterizing it.\n\nThe stress-test note is right about DDS. Algorithm 1 initializes from N(0,1), then updates x_s = beta_s * x + sqrt(1-beta_s^2) * eta and x_l analogously. That's spherical interpolation between a noise vector and fresh Gaussian noise. There is no epsilon_theta call, no reverse diffusion step, no VAE decoding before scoring. Calling it 'diffusion search' is wrong. And the objective F directly uses CLIP and CLAP, which are two of the headline metrics. The 'aligned + DDS' row's CLIP and CLAP gains are therefore partly by construction. That is a serious circularity.\n\nBut the non-DDS numbers still mean something. The 'aligned' row also improves MKL, FID, and FAD relative to the base model, and those metrics are not in the reward. So the core model plus the automated-reward alignment likely has real merit. The problem is the paper overclaims: the abstract says 'preference judgments' and 'human-preferred,' while Section 3.3.2 explicitly says no human-annotated preferences are used. That is an internal contradiction, not just a marketing issue.\n\nOther soft spots: updating pi_ref each epoch is asserted without explaining why DPO's derivation survives a changing reference; the user study is small, author-affiliated, and underdescribed; and there are drafting errors like 'Fieren' instead of 'Frieren' and an incomplete sentence in the user study section. These are minor relative to the DDS problem.\n\nWho is this for? Researchers in video-to-audio generation and alignment. It's an instructive example of circular evaluation and of mismatch between framing and method. I'd bring it to a reading group but I would not cite it as evidence for the DDS method.\n\nRecommendation: yes, send to peer review — the base model and alignment idea deserve referee time. But expect rejection in current form. The authors should reframe the reward as automated, fix or remove DDS, and strengthen the user study.","headline":"Solid dual-feature V2A model with a circular test-time search and an over-claimed 'human preference' story; deserves review, but not acceptance as is.","tokens_in":20018,"tokens_out":2900,"would_cite":false,"duration_ms":29820,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HarmoniDPO claims that preference-optimized diffusion with dual video conditioning produces better-synchronized and more natural-sounding video-to-audio output than existing models.","keywords":["video-to-audio generation","diffusion models","direct preference optimization","audio-visual alignment","reinforcement learning from human feedback","test-time scaling","dual video representation"],"falsifier":"A held-out paired listening study comparing the aligned model with the unaligned base model would settle the claim: if human raters do not significantly prefer the aligned outputs, or if the outputs they prefer score lower on the composite reward, then the online-DPO stage is optimizing the wrong objective. An even more direct check is to compute the composite reward on human-chosen winners and losers and see whether the reward ordering matches the human ordering.","tokens_in":18873,"feed_emoji":"🎬","tokens_out":9325,"duration_ms":89769,"temperature":0.7,"pith_summary":"Video-to-audio generation usually struggles because single visual embeddings lose temporal detail and because reconstruction losses do not track what listeners like. The paper argues that both problems can be addressed in one framework: a diffusion model conditioned on both a global video embedding and per-frame visual features, then fine-tuned with an online, automated preference-optimization loop. The loop scores candidate audios with a composite reward built from audiovisual, audio-text, and intrinsic-quality measures, and it adds an inference-time search that refines candidate latents. If the claims hold, generated audio for a silent video becomes simultaneously better synchronized and more natural-sounding than output from previous approaches.","feed_headline":"Beats prior video-to-audio models with preference-optimized diffusion","feed_subtitle":"Dual video features plus online-DPO push MKL to 1.82 and CLIP to 13.65 on VGGSound.","key_machinery":"The central mechanism is a two-stage pipeline. In the first stage, a pre-trained text-to-audio latent diffusion model is kept frozen and adapted to video by injecting a fused visual condition: a global video embedding and a set of frame-wise image embeddings are projected, concatenated, and processed by a self-attention layer with rotary position embeddings (position encodings that preserve frame order) before being fed into the denoising network through cross-attention. In the second stage, online-DPO repeatedly samples candidate audio clips from the current policy, scores them with the composite reward $R(y) = w_{\\mathrm{av}}R'_{\\mathrm{av}}(y) + w_{\\mathrm{at}}R'_{\\mathrm{at}}(y) + w_{\\mathrm{quality}}R'_{\\mathrm{quality}}(y)$ built from audiovisual, audio-text, and acoustic-quality scorers, selects the best and worst candidates, and updates the model with a value-aware DPO loss that includes the reward margin. At inference, Dual-scale Diffusion Search (DDS) maintains a population of latent candidates and proposes both small and large mixing steps, accepting whichever candidate scores higher. This machinery is what carries the paper's reported gains in synchronization and quality.","core_discovery":"On its own terms, HarmoniDPO's discovery is that preference-based optimization transfers to video-to-audio generation without human preference labels: the model generates candidate sounds itself, ranks them by an automated composite reward, and updates the policy with a Direct Preference Optimization loss that also includes the reward margin. The paper reports that on VGGSound the full configuration reaches MKL 1.82, CLIP 13.65, FID 6.42, FAD 1.59, and CLAP 32.57, beating every listed baseline on most metrics, and on AVSync15 it reports the best onset accuracy (32.53) and onset average precision (69.97) among the compared methods. The claim is that the dual video condition supplies the temporal synchronization while the online-DPO stage and Dual-scale Diffusion Search supply the perceptual quality.","pith_inferences":["Beyond the paper, the automated reward design invites a test for reward hacking: if the audiovisual, audio-text, and quality scorers can be satisfied by audible artifacts, the alignment stage may improve scores without improving perceived audio.","Beyond the paper, DDS is a generic population-based search for latent diffusion; the same dual-scale accept-or-reject scheme could be applied to music generation, sound-effect synthesis, or other latent-diffusion modalities.","Beyond the paper, the paper's own limitation note about single-reference metrics implies that reference-based scores such as MKL and FID may systematically penalize valid alternative sounds, so paired human comparisons are the more decisive evaluation for one-to-many video-to-audio mapping.","Beyond the paper, a practical extension is to make the number of preference candidates adaptive: the reported trade-off between CLAP and FID/FAD suggests an automatic stopping rule based on reward-margin variance could replace the fixed eight-candidate setting."],"forward_implications":["The dual video representation alone, before any preference alignment, already outperforms all listed baseline systems on the reported VGGSound metrics.","Adding the online-DPO stage improves every reported automated metric and raises the human rating in the paper's user study, with eight candidate sounds giving the best balance.","Adding Dual-scale Diffusion Search at inference pushes the reported scores further, showing that test-time computation can substitute for some additional training.","The optional text prompt changes the generated sound in controllable ways, so the same video can be matched with different plausible audio styles.","The alignment gains also appear on AVSync15, the temporally hardest subset, where onset accuracy and onset average precision are highest."],"supporting_citations":[{"why":"It supplies the frozen text-to-audio latent diffusion model that HarmoniDPO adapts for video conditioning and reuses for preference-based training.","marker":"[50]"},{"why":"It provides the adaptation of Direct Preference Optimization to diffusion models that the online-DPO loss calculation follows.","marker":"[47]"},{"why":"It provides the video encoder whose global video embedding forms one half of the dual visual condition.","marker":"[51]"},{"why":"It provides the image encoder used to extract the frame-wise features that form the second half of the dual visual condition.","marker":"[59]"},{"why":"It supplies the audio-visual correspondence score used in the composite preference reward.","marker":"[55]"},{"why":"It supplies the audio-text similarity score used as the text-consistency component of the composite reward.","marker":"[17]"},{"why":"It supplies the intrinsic audio quality score used as the perceptual-quality component of the composite reward.","marker":"[56]"},{"why":"It provides the CLIP-based audio-visual similarity used both as an evaluation metric and as the selection score inside Dual-scale Diffusion Search.","marker":"[57]"},{"why":"It provides the VGGSound training and evaluation data on which the reported comparisons are run.","marker":"[33]"},{"why":"It supplies the AVSync15 test set used to measure onset accuracy and onset average precision for temporal synchronization.","marker":"[58]"}],"fun_headline_variants":["Preference-optimized diffusion syncs video to audio better","Dual video features + online-DPO sharpen video-to-audio sync","HarmoniDPO: online-DPO and dual video beat video-to-audio baselines","Video-to-audio gets preference tuning and better sync","Dual-scale search and online-DPO improve V2A generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The preference stage rests on the assumption that a machine-computed score combining audiovisual alignment, text alignment, and acoustic quality is a faithful stand-in for human preference, because no human-annotated preference labels are used in training.","fun_headline_variants_meta":{"raw":{"variants":["Preference-optimized diffusion syncs video to audio better","Dual video features + online-DPO sharpen video-to-audio sync","HarmoniDPO: online-DPO and dual video beat video-to-audio baselines","Video-to-audio gets preference tuning and better sync","Dual-scale search and online-DPO improve V2A generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001044,"raw_usage":{"total_tokens":4392,"prompt_tokens":951,"completion_tokens":3441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":3348}},"tokens_in":567,"tokens_out":3441,"duration_ms":23271,"temperature":1.0,"reasoning_tokens":3348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:22:50.534838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A held-out paired listening study comparing the aligned model with the unaligned base model would settle the claim: if human raters do not significantly prefer the aligned outputs, or if the outputs they prefer score lower on the composite reward, then the online-DPO stage is optimizing the wrong objective. An even more direct check is to compute the composite reward on human-chosen winners and losers and see whether the reward ordering matches the human ordering.","supporting_citations":[{"cited_title":"In: Proceedings of the 32nd ACM International Conference on Multimedia, pp","cited_arxiv_id":null,"evidence_quote":"It supplies the frozen text-to-audio latent diffusion model that HarmoniDPO adapts for video conditioning and reuses for preference-based training."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"It provides the adaptation of Direct Preference Optimization to diffusion models that the online-DPO loss calculation follows."},{"cited_title":"In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp","cited_arxiv_id":null,"evidence_quote":"It supplies the audio-text similarity score used as the text-consistency component of the composite reward."},{"cited_title":"In: ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp","cited_arxiv_id":null,"evidence_quote":"It provides the CLIP-based audio-visual similarity used both as an evaluation metric and as the selection score inside Dual-scale Diffusion Search."},{"cited_title":"In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp","cited_arxiv_id":null,"evidence_quote":"It provides the VGGSound training and evaluation data on which the reported comparisons are run."},{"cited_title":"Audio-Synchronized Visual Animation","cited_arxiv_id":"2403.05659","evidence_quote":"It supplies the AVSync15 test set used to measure onset accuracy and onset average precision for temporal synchronization."}],"review_version":1}