{"id":"7f789d82-c251-49bb-af67-4ec66b12fa8c","arxiv_id":"2412.18157","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Smooth-Foley uses frame-level visual features and label-guided temporal conditions to generate continuous, synchronized audio for videos with moving or ambiguous sound sources.","lead":"This paper presents Smooth-Foley, a system that generates soundtracks for video by aligning audio events with visual frames using text labels. It targets continuous sounds like sirens or aircraft that previous video-to-audio models miss, and reports better alignment and audio quality on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported superiority of Smooth-Foley is not pinned to its proposed mechanism: the only ablation in Table I shows frame-wise features contribute almost nothing, while the label-guided temporal adapter is never removed or transferred to baselines, leaving the central claim underdetermined.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that verdict. The reader's weakest_assumption focused on the threshold of 0.5 in the CLIP-based temporal condition; that is a real fragility, but the more load-bearing issue is that the component embodying that fragility is never ablated. The only ablation removes the frame-wise adapter, and it shows essentially no difference, so the reported improvement must come from the temporal adapter, the text label, or the fine-tuning procedure. Without an ablation that removes the temporal adapter or adds it to a baseline, the central claim that label-guided semantic conditions improve temporal alignment is not actually tested. The small test set (95 manually selected clips) and the absence of error bars compound this problem, but the missing mechanism isolation is the sharpest threat to the claim. I would therefore keep the reader's CONDITIONAL verdict: the direction is plausible and the paper has genuine engineering content, but the headline claim needs a direct causal test before it can be accepted. My concrete test is designed to provide that test. This is why I marked agreement as partial: the reader and I both see the temporal-condition design as fragile, but I locate the decisive problem in the lack of isolation rather than in the specific threshold value.","tokens_in":7346,"tokens_out":4711,"duration_ms":43877,"concrete_test":"Run the missing ablations on the same 95-clip test set: (1) Smooth-Foley with the temporal adapter removed or fed a constant temporal condition while keeping the frame adapter, and (2) FoleyCrafter augmented with the same CLIP-similarity temporal condition (threshold 0.5) and the same VGGSound-Continuous fine-tuning data. If Smooth-Foley's margins over variant (1) and control (2) do not persist, the central claim should be narrowed from 'semantic guidance improves alignment' to 'adding label-based temporal conditioning improves scores under this evaluation.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that semantic guidance from frame-level video features and label-based temporal conditions yields better continuous and general V2A generation. The only ablation in Table I compares 'Ours' with 'Ours (w/o frame-wise)': on VGG-C, MKL is 1.558 vs. 1.559 and CLIP Score is 55.124 vs. 55.076; on VGG, MKL is 2.515 vs. 2.498 (slightly worse with frame-wise) and CLIP Score is 55.236 vs. 55.233. Thus the frame adapter is not the driver of the reported gains. The remaining load-bearing component is the temporal adapter, which uses binarized CLIP frame-label similarities with a fixed threshold of 0.5 (Section II.C), but no experiment disables this adapter, and no baseline such as FoleyCrafter is augmented with the same label-derived temporal condition. Consequently, Tables I–III cannot distinguish the proposed semantic-guidance mechanism from (a) the extra fine-tuning on VGGSound-Continuous, (b) the oracle text label itself, or (c) a thresholded similarity condition that may be near-constant for continuous-sound clips. The temporal-alignment claim is further supported only by 10 subjective samples and qualitative examples, not by an event-boundary metric. The result may be true, but the current evidence does not isolate it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Smooth-Foley, a video-to-audio (V2A) generation model that adapts a pre-trained text-to-audio diffusion model (Auffusion) using two lightweight adapters: a frame adapter that injects frame-wise CLIP visual embeddings through parallel cross-attention (Section II.B, Eq. 2), and a temporal adapter that derives a temporal condition by binarizing frame-label CLIP similarities with a fixed threshold of 0.5 (Section II.C). The model is initialized from FoleyCrafter and fine-tuned on a filtered VGGSound-Continuous subset (Section II.D). The authors report that Smooth-Foley outperforms Diff-Foley and FoleyCrafter on MKL, CLIP Score, FAD, and subjective ratings on both VGGSound-Continuous and VGGSound, and they claim improved temporal alignment and adherence to physical laws such as the Doppler effect.","tokens_in":7580,"tokens_out":5480,"duration_ms":47517,"significance":"The paper addresses a real limitation of existing V2A systems: maintaining synchronized audio when the sound source is continuous or becomes visually ambiguous. The design is concrete and testable: frame-wise features and label-derived temporal conditions are well-motivated, and the use of a lightweight adapter on a frozen T2A model is practical. The authors report several standard metrics and include one ablation, which is a useful starting point. However, as presented, the experiments do not isolate the proposed semantic-guidance mechanism from additional fine-tuning data, from the oracle text label, or from the hand-set CLIP threshold. The core contribution therefore remains plausible but not established. If follow-up experiments with a temporal-adapter ablation, controlled baselines, and statistical evaluation confirm the reported gains, the method would be a solid incremental contribution to the V2A literature.","major_comments":[{"comment":"The only component-level ablation tests the frame adapter, and it shows that this component contributes essentially nothing to the headline metrics: on VGG-C, MKL is 1.558 with frame-wise features versus 1.559 without, and CLIP Score is 55.124 versus 55.076; on VGG, MKL is 2.515 versus 2.498 (slightly better without frame-wise), and the FAD comparisons are mixed (e.g., PANNs FAD on VGG is 18.31 with frame-wise versus 13.07 without). The proposed temporal adapter, which is the other half of the semantic guidance, is never ablated. Consequently, Tables I and II cannot attribute the reported superiority over FoleyCrafter and Diff-Foley to the proposed mechanism, as opposed to the extra fine-tuning on VGGSound-Continuous or to the oracle text label. I request an ablation that removes the temporal adapter and, ideally, an experiment that adds the same label-derived temporal condition to a baseline.","section":"III.B, Tables I and II"},{"comment":"The central temporal-alignment claim is supported only by 10 subjective ratings per model and by qualitative examples in Figure 4; there is no event-boundary metric such as onset/offset accuracy or temporal IoU. The binarization threshold of 0.5 on CLIP frame-label similarities is asserted without justification or sensitivity analysis, and it is not validated against ground-truth sound event intervals. Because the temporal adapter can only be as good as its input condition, the paper needs either a direct evaluation of the estimated temporal conditions against AudioSet-strong-style annotations or an ablation with a different threshold (including a non-binarized similarity) to show that the threshold choice is not responsible for the results.","section":"II.C, Table III"},{"comment":"The comparison is confounded by training protocol. Smooth-Foley is initialized from FoleyCrafter and fine-tuned on VGGSound-Continuous (Section II.D), while the FoleyCrafter and Diff-Foley baselines appear to be evaluated without this fine-tuning. Any improvement on VGGSound-Continuous could therefore come from exposure to the target-domain data rather than from the frame adapter or label-guided temporal adapter. A controlled comparison would fine-tune the baselines on the same subset, or evaluate Smooth-Foley without fine-tuning, to separate the mechanism from the training data. The same concern applies to the 95-clip manually selected test set, which is not accompanied by error bars or significance tests; the reported differences in Table III (e.g., 8.42 vs. 5.89 for semantic alignment) need confidence intervals or a paired test to be interpretable.","section":"II.D, III.A"},{"comment":"The claim that generated audio 'exhibits higher quality and better adherence to physical laws' (Abstract; also the Doppler-effect discussion in Section III.B.2) is supported only by a qualitative example. The Doppler claim in particular is stated as a frequency rise-peak-decline pattern but no spectrogram, pitch track, or quantitative comparison against ground truth is provided. If physical plausibility is part of the contribution, it should be measured; otherwise the statement should be softened.","section":"Abstract and III.B.2"}],"minor_comments":[{"comment":"The mixing weight lambda in Eq. (2) is never specified; please report its value or state how it was chosen. Also, the denominator uses a generic d without defining the dimension of K.","section":"II.B, Eq. (2)"},{"comment":"Please clarify the binarization operation: are frame similarities above 0.5 set to 1 and below to 0, and is the condition then used as a ControlNet-style conditioning map for the temporal adapter? The text says the temporal adapter is trained on AudioSet-strong with ground-truth timestamps but does not explain how the training condition is represented (e.g., binary mask, per-frame probability, event embedding).","section":"II.C"},{"comment":"The phrase 'Frechect Audio Distance' should be 'Fréchet Audio Distance'; also, the subjective evaluation uses only 10 samples per model, so please state whether the same 10 videos were used across models and report per-sample variance or inter-rater agreement.","section":"III.A.2"},{"comment":"The qualitative examples in Figure 4 would be much more informative with aligned spectrograms or waveform onset markers; as printed, the 'miss/✓' annotations are hard to verify without audio or time-aligned evidence.","section":"Figure 4"},{"comment":"The filtering procedure relies on text-to-audio grounding [22] but no details are given about the grounding model or the filtering threshold; please provide the number of clips after each filtering stage and the overlap between the 95 test clips and the training subset to avoid train/test leakage.","section":"II.D"}],"recommendation":"major_revision","confidential_remarks":"The method is essentially FoleyCrafter plus frame-wise features and a label-derived temporal condition, and the relationship to prior work should be made explicit in the revision. The editor may also want to confirm that the baseline FoleyCrafter checkpoint and the VGGSound-Continuous subset will be released, since the evaluation depends on the manual 95-clip selection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Smooth-Foley is a reasonable engineering contribution to a real problem — video-to-audio models that drop continuous or off-screen sounds. The label-guided temporal condition, derived from CLIP frame-label similarities, is a neat trick that could help. The VGGSound-Continuous filtered subset is a useful resource. The writing is clear, and the authors report their ablation honestly, including the fact that dropping frame-wise features makes almost no difference (Table I: 1.558 vs 1.559 MKL on VGG-C; 2.515 vs 2.498 on VGG, slightly worse with frame-wise). That honesty is to their credit, but it is also the paper's soft spot: the only ablation they run is the one that shows their frame adapter contributes little. The temporal adapter, which is the other load-bearing component, is never removed, never applied to a baseline like FoleyCrafter, never tested with a different threshold. So the reported gains over baselines could come from the label-derived temporal condition, from extra fine-tuning on the continuous subset, or from the oracle text label itself. The current experiments cannot separate those.\n\nThe evaluation scale is small: 95 manually selected test clips, subjective scores from 10 evaluators on 10 samples per model, no error bars or significance tests, no code or artifacts released. The claim that generated audio follows the Doppler effect is supported only by qualitative examples. These are fixable weaknesses, not fatal ones. The MKL/FAD/CLIP improvements over two baselines are consistent, and the problem is well motivated.\n\nBottom line: this is a solid idea from a competent group, but the paper needs a stronger causal story before I'd trust the headline. It deserves a serious peer-review round — a good reviewer can push for the temporal-adapter ablation and a larger, careful evaluation. I'd take it to reading group as a case study in why ablations matter. I would probably cite it if I worked in V2A, mostly for the continuous-sound subset.","headline":"A sensible idea for a real V2A failure mode, but the paper's own ablation leaves the mechanism underdetermined; worth reviewing, not worth taking on faith.","tokens_in":8152,"tokens_out":2133,"would_cite":true,"duration_ms":20288,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Smooth-Foley claims that frame-wise visual conditions plus text-label-guided temporal signals keep video-to-audio generation continuous for moving sound sources, outperforming Diff-Foley and FoleyCrafter on continuous-sound benchmarks.","keywords":["video-to-audio generation","Foley sound synthesis","semantic guidance","temporal alignment","diffusion model adaptation","CLIP similarity","continuous sound","VGGSound"],"falsifier":"Take a VGGSound-Continuous clip in which the label's sound source is off-screen or visually tiny for part of the clip, such as an ambulance siren heard before the ambulance enters the frame. Compute per-frame CLIP similarities between the frames and the label text; if the audible portion has similarities below 0.5 while the inaudible portion has similarities above it, the temporal condition is inverted, and Smooth-Foley should generate sound at the wrong times. If it does not fail on such a clip, the thresholding is not the mechanism driving temporal alignment.","tokens_in":7087,"feed_emoji":"🔊","tokens_out":5579,"duration_ms":50725,"temperature":0.7,"pith_summary":"This paper tackles a specific failure of video-to-audio generation: models go silent or lose synchronization when the sound source moves, becomes small, or leaves the frame. The proposed Smooth-Foley feeds the generator two cleaner conditions—per-frame CLIP image embeddings instead of one clip-level embedding, and a temporal on/off signal obtained by comparing each frame to the text label with CLIP and thresholding at 0.5. Both conditions enter through lightweight adapters into a frozen text-to-audio diffusion model, preserving pre-trained audio quality while sharpening video conditioning. The authors also filter a continuous-sound subset, VGGSound-Continuous, and fine-tune on it. They report that on VGGSound and on this subset, Smooth-Foley beats Diff-Foley and FoleyCrafter on objective metrics and human ratings, and that its outputs follow physical patterns such as Doppler-like frequency shifts.","feed_headline":"Sound stays on when objects move: Smooth-Foley syncs video audio","feed_subtitle":"Frame-by-frame video cues plus text labels fix the silent gaps where prior video-to-audio models lose the sound.","key_machinery":"Two trainable adapters attached to a frozen pre-trained text-to-audio diffusion model (Auffusion). The frame adapter projects per-frame CLIP image embeddings through a linear layer and injects them via parallel cross-attention, keeping only the frame key/value projection matrices trainable. The temporal adapter reuses ControlNet's UNet-encoder design, is trained on AudioSet-strong with ground-truth timestamp conditions, and at inference consumes a binarized temporal condition: cosine similarity between projected frame CLIP embeddings and the label's CLIP text embedding, thresholded at 0.5. Training the two adapters separately keeps the backbone frozen, so the model retains pre-trained audio quality while adding video conditioning.","core_discovery":"The central claim is that temporal alignment failures in video-to-audio generation stem from two correctable defects—low temporal resolution of video features and inaccurate temporal conditions—and that replacing clip-wise video embeddings with frame-wise CLIP embeddings, plus deriving the temporal condition from binarized frame-label CLIP similarities, fixes both. On VGGSound and on the filtered VGGSound-Continuous subset, Smooth-Foley reports lower FAD and mean KL divergence, higher CLIP scores, and better human ratings for semantic alignment, temporal alignment, and audio quality than Diff-Foley and FoleyCrafter. Qualitatively, it maintains sound during continuous motion, detects sound when the object becomes visually ambiguous, and produces frequency rises and falls consistent with the Doppler effect. The paper interprets these results as evidence that semantic guidance from textual labels yields more temporally aligned and physically plausible audio.","pith_inferences":["Editorial inference: because the 0.5 threshold is global, per-category thresholds or a learned detector could further improve off-screen or visually weak sound sources.","Editorial inference: the binarized CLIP similarity could be replaced by soft weights, letting the generator interpolate onset and offset instead of committing to hard on/off states.","Editorial inference: the VGGSound-Continuous filtering recipe (label-based selection plus text-to-audio grounding) could be reused as a public benchmark for continuous-sound video-to-audio evaluation.","Editorial inference: since the frame adapter and temporal adapter are trained separately, joint training could resolve potential disagreements between visual and temporal conditions."],"forward_implications":["For videos with moving or visually ambiguous sound sources, generated audio should remain present and synchronized instead of going silent.","The same frozen text-to-audio backbone can be adapted to video-to-audio generation with only lightweight adapters, avoiding retraining the audio generator from scratch.","Frame-wise visual features raise the effective temporal resolution of semantic video conditions beyond clip-level features, improving onset accuracy.","Using the textual label as a semantic guide for temporal-condition extraction makes the temporal signal more robust when the sounding object is small, blurred, or off-center.","On continuous-sound benchmarks such as VGGSound-Continuous, the model's outputs should exhibit higher audio quality, semantic alignment, and temporal alignment than Diff-Foley and FoleyCrafter."],"supporting_citations":[{"why":"Supplies the frozen CLIP image and text encoders used for both frame-wise embeddings and frame-label similarity.","marker":"[4]"},{"why":"Diff-Foley is a baseline that uses contrastive visual-audio pretraining and must be outperformed.","marker":"[9]"},{"why":"FoleyCrafter provides the initialization and architecture that Smooth-Foley extends with frame-wise features and label-guided temporal conditions.","marker":"[11]"},{"why":"Auffusion is the pre-trained text-to-audio diffusion model used as the frozen backbone.","marker":"[12]"},{"why":"IP-Adapter provides the linear projection and adapter design used to inject frame embeddings.","marker":"[13]"},{"why":"ControlNet supplies the architecture and training recipe for the temporal adapter.","marker":"[18]"},{"why":"AudioSet-strong provides temporally strong labels used to train the temporal adapter.","marker":"[19]"},{"why":"VGGSound is the primary dataset and the source of the filtered VGGSound-Continuous subset.","marker":"[20]"},{"why":"Text-to-audio grounding is used to filter out clips whose audio does not match their labels.","marker":"[22]"}],"fun_headline_variants":["Frame-wise features plus text labels keep video audio continuous","Smooth-Foley syncs audio to moving objects using text-guided frames","Semantic text guidance makes video-to-audio sound stay on track","New model uses frame CLIP and temporal labels to fix audio dropouts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a visual-text similarity score (CLIP) above 0.5 between a frame and the label reliably marks when the sound is actually audible, and that this same threshold works for every clip.","fun_headline_variants_meta":{"raw":{"variants":["Frame-wise features plus text labels keep video audio continuous","Smooth-Foley syncs audio to moving objects using text-guided frames","Semantic text guidance makes video-to-audio sound stay on track","New model uses frame CLIP and temporal labels to fix audio dropouts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1385,"prompt_tokens":921,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":537,"tokens_out":464,"duration_ms":4817,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:58:12.296316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a VGGSound-Continuous clip in which the label's sound source is off-screen or visually tiny for part of the clip, such as an ambulance siren heard before the ambulance enters the frame. Compute per-frame CLIP similarities between the frames and the label text; if the audible portion has similarities below 0.5 while the inaudible portion has similarities above it, the temporal condition is inverted, and Smooth-Foley should generate sound at the wrong times. If it does not fail on such a clip, the thresholding is not the mechanism driving temporal alignment.","supporting_citations":[{"cited_title":"Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Diff-Foley is a baseline that uses contrastive visual-audio pretraining and must be outperformed."},{"cited_title":"The benefit of temporally-strong labels in audio event classification,","cited_arxiv_id":null,"evidence_quote":"AudioSet-strong provides temporally strong labels used to train the temporal adapter."},{"cited_title":"Vggsound: A large- scale audio-visual dataset,","cited_arxiv_id":null,"evidence_quote":"VGGSound is the primary dataset and the source of the filtered VGGSound-Continuous subset."}],"review_version":1}