{"id":"604ec37b-b7cc-4859-b99f-bbbe978e3955","arxiv_id":"2507.02271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning a video encoder with self-distillation on cropped and shifted clips makes video-to-audio generation robust to partially visible Foley targets.","lead":"This paper teaches a video-to-audio model to handle cinematic shots like close-ups and camera moves, where the sound-making object is only partly visible. The method makes a student video encoder imitate a frozen teacher on original clips while being shown cropped and shifted versions, improving generated audio on both simulated and real cinematic videos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains under partial visibility are measured on test sets generated by the same simulated variations used in training, so improvement may reflect augmentation matching rather than real cinematic transfer.","rationale":"I agree with the reader's conditional verdict, but I isolate the evaluation leakage as the single most load-bearing issue. The encoder-bottleneck assumption is partly supported by the within-distribution gains, whereas the transfer to real cinematic language is untested. The paper's synthetic test sets are generated with the same augmentation code (fcu/fcm) used in training; because the test-time distortion distribution equals the training-time distribution, the metrics measure robustness to the augmentation, not to cinematic language. The only out-of-distribution evidence is a small human preference study, which cannot support the quantitative claim. The fix is straightforward: test on out-of-range distortions and on real cinematic clips. If the results hold, the paper is strong; if not, the central claim should be substantially weakened. This does not change the reader's CONDITIONAL verdict.","tokens_in":11886,"tokens_out":5132,"duration_ms":60218,"concrete_test":"Fix the confound by evaluating the k=75% fcu model on (a) VGG-CU variants with crop ratios outside the training range, e.g., U(0.2,0.3) and U(0.7,0.8), and (b) a new set of real cinematic close-up/camera-movement clips with ground-truth audio, computing FAD, KL, and Align Acc. If performance on out-of-range crops or real clips drops to near the Diff-Foley baseline, the reported improvements are due to augmentation distribution matching rather than generalizable partial-visibility learning. Additionally, report multi-seed error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the main evidence for the central claim is confounded by the evaluation protocol. Section 4 states that VGG-CU and VGG-CM are created by applying the same cinematic language variations fcu and fcm to the VGGSound test set used during training. Training applies fcu with crop ratios rh, rw ~ U(0.4, 0.6) (Eq. 1, Section 3.2); the VGG-CU test set is generated with the same fcu distribution. Thus the student is evaluated on crops drawn from the exact augmentation distribution it was trained on. The observed gains on VGG-CU/VGG-CM could reflect learning to be invariant to these specific uniform crops and shifts, not an understanding of partial visibility in real cinematic language. Real close-ups and camera movements involve occlusions, focus changes, framing, motion blur, and scene context that uniform cropping does not capture. The only real-world evidence is the human study on 31 YouTube videos (Table 2), which reports only preference (74.5% prefer Ours) without audio quality or alignment metrics, and with 20 participants it is too small to establish generalization. Consequently, the abstract's claim of 'impressive improvements under partial visibility across all evaluation metrics' is not yet supported for actual cinematic footage; the method may be matching the synthetic distortion distribution rather than learning the intended audio-visual associations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses Video-to-Audio (V2A) generation in cinematic scenarios where Foley targets are only partially visible due to close-ups and camera movements. The authors propose a self-distillation method: cinematic language variations fcu (random spatial cropping) and fcm (crop plus directional shift) create paired videos (xv, xf_v), and a student video encoder is trained to align its features from the partially visible video with the frozen teacher's features from the fully visible video, leaving the downstream diffusion audio generator unchanged. Experiments on VGGSound and two synthetically modified test sets, VGG-CU and VGG-CM, report consistent improvements over the Diff-Foley baseline across all metrics, and a small human study on 31 real YouTube videos reports 74.5% preference for the proposed method.","tokens_in":12155,"tokens_out":5664,"duration_ms":67822,"significance":"If the central claim holds, this is a practical and general contribution: a simple, model-agnostic self-distillation procedure that improves V2A under spatial and temporal partial visibility without retraining the audio generator or collecting cinematic footage. The authors are the first to frame partial visibility as a cinematic-language problem for V2A, and the consistent gains on the synthetic benchmarks plus the positive human preference on real clips are encouraging. The main open question is whether the synthetic augmentations transfer to real cinematic footage; the current evidence for that claim is a small preference study, so the real-world significance is not yet fully established.","major_comments":[{"comment":"The VGG-CU and VGG-CM test sets are produced by applying the same transformations fcu and fcm that are used to augment the training set, and Table 4 selects the augmentation proportion k directly on these test sets. The reported gains may therefore reflect matching the exact augmentation distribution rather than learning to handle genuine partial visibility, and the headline numbers are optimistically biased. Please introduce a held-out validation split for model selection and evaluate on test transformations that differ from the training augmentation, including objective metrics on real cinematic clips.","section":"Section 4 (Implementation Details; Tables 1 and 4)"},{"comment":"The only real-world evidence is a preference study with 20 participants over 31 YouTube videos, reporting only the fraction of clips preferred and no objective audio quality or synchronization metrics. This is too small to establish that the synthetic fcu/fcm distortions transfer to real close-ups and camera movements, which involve occlusions, motion blur, and framing changes not captured by uniform cropping and shifting. Please add objective metrics on the real clips and report inter-participant agreement or a confidence interval.","section":"Section 4.1 (Real Cinematic Scene; Table 2)"},{"comment":"All experiments are single training runs without error bars or significance tests, while the evaluation itself samples 10 generated clips per video. Given the stochasticity of diffusion sampling and training, the reported differences may be within run-to-run or sample-to-sample noise; please provide multiple seeds or bootstrap confidence intervals for at least the main comparison and the k ablation.","section":"Tables 1, 3, and 4"}],"minor_comments":[{"comment":"The phrase 'audio-visual corrections' should presumably be 'audio-visual correlations'.","section":"Section 3.1"},{"comment":"Equation (2) uses c_s′ before the notation is introduced; please define c_s, c_sf, and c_t explicitly in the text preceding the equation.","section":"Equation (2)"},{"comment":"The statement that at k = 100% 'no data from the original dataset is used' is inaccurate: the teacher still receives original videos x_v to produce c_t, so original data are used as supervision. Please rephrase.","section":"Section 4.3 (Table 4 discussion)"},{"comment":"The text says 'even when trained only on VGG-CU training set', but the method is trained on VGGSound with fcu augmentation; this wording is confusing and should be corrected.","section":"Section 4.1"},{"comment":"Table 3 compares fcu-only at k = 50%, fcm-only at k = 50%, and fcu&fcm at 66.7% total, so the proportion of augmented data is not matched; please report results at matched total augmentation proportions or justify the chosen values.","section":"Table 3"},{"comment":"Figure 5 reports per-category Align Acc for the top 25 categories without error bars; please clarify how the improvement ranking is computed and whether the differences are significant.","section":"Figure 5"},{"comment":"No code or checkpoints are released; for a method whose main benefits are practical, releasing code would aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal, and the core idea is worth publishing if the evaluation can be deconfounded. The most serious issue is that the augmentation proportion k is selected directly on the VGG-CU/VGG-CM test sets (Table 4), and those sets are generated with the same transformations used in training; this makes the headline gains hard to interpret. I would like to see a validation-based selection procedure and a stronger real-world evaluation before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The problem this paper tackles is real and has been overlooked: cinematic close-ups and camera moves obscure Foley targets, and current V2A models degrade sharply in those conditions. The proposed fix is simple and well-motivated — simulate those variations, then align a student encoder's features from the degraded clips to a frozen teacher's features from the original clips. That is a clean, general recipe that doesn't touch the audio generator, which matters for adoption.\n\nWhat the paper does well: the gains on VGG-CU and VGG-CM are consistent across all reported metrics, the qualitative examples are convincing, and the per-category breakdown shows improvements on exactly the kinds of sounds (instruments, impacts, vehicles) you'd expect to suffer under partial occlusion. The model also holds its own on the unfiltered VGGSound test set, which is a useful sanity check.\n\nThe soft spots are real, though not fatal. The biggest one is exactly what your stress-test note says: VGG-CU and VGG-CM are generated with the same fcu and fcm functions used in training, with the same crop-ratio distribution. So the model is being evaluated on the exact distortion distribution it was trained to be invariant to. That makes the headline 'impressive improvements under partial visibility' read as partly an augmentation-matching result, not yet evidence about real cinematic footage. The human study on 31 YouTube videos with 20 participants is too small and only asks for preference, so it can't carry the generalization claim by itself.\n\nTwo smaller but legitimate concerns: k = 75% is chosen by looking at test-set performance (Table 4), and there are no error bars or multi-seed runs. Given the reported margins, some of the differences could be noise. Also, the paper assumes the video encoder is the bottleneck and never tests that assumption directly; the results are consistent with it, but the reasoning in Section 3.1 is inference, not proof. And the abstract's 'enhances performance' claim is undercut by the Align Acc drop on VGGSound (92.946 to 91.718) — worth reconciling.\n\nNone of this falsifies the method. The core idea is plausible, the experiments are mostly well designed, and the limitations section is honest. This paper deserves a serious referee: it addresses a genuine gap, the method is simple and reproducible in principle, and the flaws are fixable. I'd send it out, but ask the authors to validate on real cinematic footage (or at least a held-out distortion type), add error bars, use a proper validation split for k, and directly test the encoder-bottleneck assumption. For my own work, I'd cite it as the first to frame cinematic partial visibility as a V2A problem.","headline":"A sensible, clearly motivated self-distillation method for video-to-audio under partial visibility, but the main evaluation is partly circular because the VGG-CU/VGG-CM test sets come from the same augmentation distribution used in training.","tokens_in":12681,"tokens_out":1350,"would_cite":true,"duration_ms":17549,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that video-to-audio models fail on cinematic shots because the video encoder cannot cope with partial visibility, and proposes a self-distillation fix that aligns partially visible features with full-view features.","keywords":["video-to-audio generation","cinematic language","partial visibility","self-distillation","teacher-student framework","latent diffusion","audio-visual alignment","data augmentation"],"falsifier":"A reader could test this by taking real cinematic footage with known Foley ground truth, computing audio metrics with the student encoder versus the teacher encoder while freezing the same generator; if the teacher's full-view features do not yield better audio than the student's partial-view features, the encoder-side diagnosis fails. A second check: if a generator fine-tuned directly on real close-up clips outperforms the distilled encoder while the distilled encoder underperforms on real footage, then the synthetic crop simulation does not transfer.","tokens_in":11665,"feed_emoji":"🎬","tokens_out":3397,"duration_ms":33995,"temperature":0.7,"pith_summary":"This paper argues that state-of-the-art video-to-audio (V2A) models fail on cinematic shots—close-ups and camera movements—because the video encoder produces weak conditions when the sound source is only partially visible, not because the audio generator is broken. The proposed fix is a teacher–student self-distillation: the frozen original encoder serves as teacher, and a trainable student encoder is pushed to map features from artificially cropped or shifted clips onto the teacher's features from the same clip's full view. Because both clips share the same audio, aligning their features teaches the student to infer sound from partial visual clues. The paper reports that this alignment improves every metric on two simulated cinematic test sets, and also improves the original VGGSound benchmark.","feed_headline":"Self-distillation makes video-to-audio work on close-ups","feed_subtitle":"Aligning partial-view features to full-view ones restores Foley sound in cinematic shots and even lifts VGGSound scores.","key_machinery":"The load-bearing mechanism is a self-distillation alignment between two views created by Cinematic Language Variations f. A close-up variation fcu crops each frame to a random rectangle (ratio uniform in [0.4, 0.6]); a camera-movement variation fcm applies the same crop and shifts it left/right or up/down along the central axis. The teacher is the frozen pre-trained CA VP encoder, and the student is the same architecture initialized from it. The loss Lp = cos(ct, cs′) + MSE(ct, cs′) aligns the student's feature from either the varied or the original clip with the teacher's feature from the original clip. This pushes partially visible videos into the original feature space without touching the audio generator.","core_discovery":"On the paper's own terms, the central discovery is that partial visibility in cinematic language is primarily an encoder-side failure: the frozen video encoder extracts degraded audio-visual conditions, so the generative model receives poor priors. Simulating close-ups with random crops and camera movements with random shifts creates paired videos (xv, xfv) with identical audio-visual content but different visibility. Training the student encoder to align features of the partially visible clip with the teacher's full-view features, using a cosine-plus-MSE loss, transfers the pre-trained audio-visual knowledge into the partial-view regime. The trained student then supplies better conditioning for the unchanged latent-diffusion audio generator, restoring synchronized Foley sounds under close-ups and camera movements.","pith_inferences":["If the encoder-conditioning diagnosis holds, other video-conditioned generation tasks that suffer under occlusion or cropping—such as text-to-audio with video context or visual question answering with partial views—could benefit from the same view-alignment self-distillation recipe.","The random-crop and random-shift simulation may not cover real cinematic phenomena like depth-of-field blur, rack focus, or complex dolly moves; a natural extension would train on real close-up footage with synthetic or transferred audio labels.","Because the student is initialized from the teacher and only features are aligned, the approach is architecture-agnostic and likely compatible with newer V2A backbones beyond Diff-Foley.","The k=75% sweet spot (versus 50% and 100%) suggests the pairing signal matters more than augmentation volume, pointing toward a general scheduling principle for self-distillation under partial visibility."],"forward_implications":["Under the close-up test set (VGG-CU), the method improves all reported metrics over the Diff-Foley baseline.","Under the camera-movement test set (VGG-CM), improvements appear even when training used only close-up variations, suggesting partial-visibility learning transfers across spatial and temporal occlusion.","The original VGGSound test set also improves slightly, indicating the alignment does not sacrifice ordinary-video performance.","The method requires no extra cinematic data and no modification to the generative model, so it can be layered onto existing V2A pipelines.","In a human study on 31 real YouTube cinematic videos, 74.51% of preferences favored the proposed method over the baseline."],"supporting_citations":[{"why":"Supplies the Diff-Foley baseline, the frozen CA VP video encoder used as teacher, and the Align Acc evaluation metric.","marker":"[Luo et al., 2024]"},{"why":"Provides the VGGSound dataset on which training and all three test sets are built.","marker":"[Chen et al., 2020a]"},{"why":"Defines cinematic language and the close-up and camera-movement techniques that motivate partial visibility.","marker":"[Mercado, 2019]"},{"why":"Supplies the latent diffusion model framework used for the audio generator.","marker":"[Rombach et al., 2022]"},{"why":"Provides the classifier-free guidance configuration used for generation and comparison.","marker":"[Ho and Salimans, 2022]"},{"why":"Supplies the FAD, FD, KID, KL, and ISc evaluation metrics.","marker":"[Liu et al., 2023]"}],"fun_headline_variants":["Self-distillation teaches video-to-audio to see close-ups","Aligned features restore Foley sound in cinematic close-ups","Spotlighting partial views: self-distillation for V2A","Seeing less, hearing better: V2A via self-distillation","Close-up audio from self-distilled video features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach assumes the audio-quality drop in cinematic shots comes from the video encoder's features and not from the audio generator, and that random crops and shifts faithfully recreate the partial visibility of real close-ups and camera moves.","fun_headline_variants_meta":{"raw":{"variants":["Self-distillation teaches video-to-audio to see close-ups","Aligned features restore Foley sound in cinematic close-ups","Spotlighting partial views: self-distillation for V2A","Seeing less, hearing better: V2A via self-distillation","Close-up audio from self-distilled video features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1147,"prompt_tokens":827,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":237}},"tokens_in":443,"tokens_out":320,"duration_ms":3782,"temperature":1.0,"reasoning_tokens":237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:33:14.356602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test this by taking real cinematic footage with known Foley ground truth, computing audio metrics with the student encoder versus the teacher encoder while freezing the same generator; if the teacher's full-view features do not yield better audio than the student's partial-view features, the encoder-side diagnosis fails. A second check: if a generator fine-tuned directly on real close-up clips outperforms the distilled encoder while the distilled encoder underperforms on real footage, then the synthetic crop simulation does not transfer.","supporting_citations":[{"cited_title":"Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models.Advances in Neural Information Processing Systems, 36,","cited_arxiv_id":null,"evidence_quote":"Supplies the Diff-Foley baseline, the frozen CA VP video encoder used as teacher, and the Align Acc evaluation metric."},{"cited_title":"The filmmaker’s eye: The language of the lens: The power of lenses and the expressive cinematic image","cited_arxiv_id":null,"evidence_quote":"Defines cinematic language and the close-up and camera-movement techniques that motivate partial visibility."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the latent diffusion model framework used for the audio generator."},{"cited_title":"AudioLDM: Text-to-audio generation with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the FAD, FD, KID, KL, and ISc evaluation metrics."}],"review_version":1}