{"id":"3af46085-e52b-450f-93ec-f1cd45780753","arxiv_id":"2505.13062","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A chain-of-thought fine-tuned vision-language model can infer audio descriptions from silent videos, and using those descriptions as prompts improves video-to-audio generation.","lead":"Researchers tested whether vision-language AI models can guess what sounds would accompany a silent video, then fine-tuned a model to produce those audio descriptions. The method improves downstream text-assisted video-to-audio generation that normally requires manually written sound captions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoT-AudioCaps derives sound-event labels from audio captions, not visual evidence; without measuring visual inferability, SVAD gains may reflect memorized correlations rather than modal-mismatch reasoning.","rationale":"The reader's weakest assumption and my concern converge: neither the task ceiling nor the training targets are grounded in measured visual sufficiency. I agree with the CONDITIONAL verdict; no change is needed. I considered other issues, such as missing code/data, absent error bars, and the confounded CoT-SFT ablation, but the visual-sufficiency premise is more load-bearing because the paper's distinctive 'reasoning' claim fails if the training signal is not visually grounded. The proposed test directly isolates this assumption: annotate test sound events for visual inferability and compare model performance on the two groups. If non-inferable performance is near zero, then SVAD as formulated is not a reasoning task for a large fraction of AudioCaps, and the headline claim should be weakened to caption-copying under distribution shift. If performance is comparable across both groups, the assumption is not limiting. Thus the reader's CONDITIONAL verdict remains appropriate.","tokens_in":9478,"tokens_out":5038,"duration_ms":52811,"concrete_test":"Take the 866 AudioCaps test videos and have at least two annotators (or a strong LLM with visual input, validated against humans) label each ground-truth sound event as 'visually inferable' or 'not visually inferable' from the silent frames alone. Split VL2-CoT-SFT's generations by this label and compute event-level recall plus CLAP on each subset. If performance collapses on non-inferable events while improving on inferable ones, the CoT-SFT gain is correlation-based rather than reasoning, and the visual-sufficiency assumption is violated. If performance is comparable across both subsets, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CoT-SFT's central claim is that it improves VLMs' modal-mismatch reasoning for SVAD. For that claim to hold, the intermediate sound-event targets in Algorithm 1 must be visually derivable: given video objects, the model should be able to reason out the sound events. But line 3 of Algorithm 1 generates <V_object, S_event> by having GPT-4 parse both C_video and the ground-truth C_audio, so the sound events are labels taken from audio, not visual evidence. For any sound that has no visible correlate (off-screen noise, room tone, ambiguous source), the training pair is not a reasoning example; the model can only memorize object-sound co-occurrences. The paper never quantifies how many AudioCaps sound events are visually inferable, so the CLAP/CIDEr gains in Table 2 do not establish 'reasoning'; they are equally consistent with caption-pattern memorization. Table 3 inherits this ambiguity because it feeds the same generated captions into VT2A models. This unmeasured visual-sufficiency premise is the load-bearing assumption behind the abstract's 'modal-mismatch reasoning' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the task of Reasoning Audio Descriptions from Silent Videos (SVAD), in which a vision-language model must generate an audio caption from a silent video without access to the audio. To improve VLM performance on this task, the authors construct a CoT-AudioCaps dataset: for each AudioCaps video, they use VideoLLaMA2 to produce a video caption, then use GPT-4 to parse the video caption and the ground-truth audio caption into structured video objects and sound events, which are assembled into three subtask datasets (visual objects, sound events, audio captions). They fine-tune VideoLLaMA2 with LoRA under two strategies (two-stage and single-stage) and a chain-of-thought supervised fine-tuning (CoT-SFT) variant. They evaluate on AudioCaps using CLAP and text-text captioning metrics, and they feed the generated captions as text prompts into two VT2A models (STA-V2A and FoleyCraft) to assess downstream video-to-audio generation. The reported results show that CoT-SFT improves CLAP from 0.404 to 0.424 in the single-stage setting and generally improves VT2A metrics relative to captions from the base VLM.","tokens_in":9675,"tokens_out":4551,"duration_ms":43479,"significance":"If the reported gains reflect genuine modal-mismatch reasoning, the paper offers a practical solution to the missing-text-prompt problem in VT2A inference and a new benchmark task for evaluating VLMs' cross-modal reasoning abilities. The construction of a CoT dataset from an established corpus and the systematic comparison of two-stage versus single-stage SFT strategies are useful contributions. The experimental comparisons are clearly described and cover multiple pre-trained VLMs, two SFT strategies, and two downstream VT2A systems. However, the central interpretive claim is not yet fully supported because the supervision for the reasoning steps (video objects and sound events) is derived from the ground-truth audio captions rather than from visual evidence, and no statistical significance analysis accompanies the claimed improvements.","major_comments":[{"comment":"The CoT-AudioCaps construction derives S_event from ground-truth audio captions via GPT-4 (Algorithm 1, line 3), not from visual evidence. For any sound event without a visual correlate in the video (e.g., off-screen noise, room tone, or an ambiguous source), the training pair does not provide a reasoning example; the model can only memorize object-sound co-occurrences. The paper never quantifies what fraction of AudioCaps sound events is visually inferable, so the improvements in Table 2 (e.g., CLAP from 0.404 to 0.424) are equally consistent with caption-pattern memorization as with the claimed 'modal-mismatch reasoning'. To support the central claim, please provide either (a) a human-judgment study of how many AudioCaps sound events are inferable from silent video, (b) an oracle upper bound where the model is given the correct video objects, or (c) a held-out split separating visually inferable from non-inferable sound events and showing that CoT-SFT's advantage is concentrated in the inferable subset.","section":"Section 2.2, Algorithm 1, Table 2"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere. The abstract and Section 3.3 use the word 'significantly', but with single runs the differences between VL2-SFT and VL2-CoT-SFT (CLAP 0.404 vs 0.424, CIDEr 0.550 vs 0.578) may be within run-to-run noise, particularly for CLAP and the VT2A metrics, which are known to be noisy. Please report results over multiple seeds or provide bootstrap or paired significance tests, especially for the headline CLAP numbers and the FD/FAD improvements in Table 3.","section":"Section 3.1.2 and Tables 2 and 3"},{"comment":"The AVCap-V and DALI baselines are cited from their original papers rather than re-run under the same evaluation protocol. Because the SVAD task definition, video preprocessing, and metric implementations may differ, the claimed 'significant improvements' over these baselines may reflect protocol differences rather than model capability. Please either re-run these baselines with the same video encoder and evaluation harness or explicitly justify comparability (e.g., same test split, same metric code, same caption decoding settings).","section":"Section 3.1.1 and Table 2"},{"comment":"Table 3 evaluates only the best SVAD model (VL2-CoT-SFT) as a text-prompt source; it does not include VL2-SFT or the two-stage CoT-SFT model. As a result, the table does not demonstrate that VT2A performance tracks SVAD caption quality, which is the paper's second central claim. Please include at least one intermediate model (e.g., VL2-SFT) so the reader can see that better SVAD captions monotonically improve VT2A metrics. In addition, for FoleyCraft the 'w/o AD' row has AV-Align 0.233 versus GT 0.232, so the practical benefit for that model appears marginal and deserves explicit discussion.","section":"Section 3.4 and Table 3"}],"minor_comments":[{"comment":"The row label 'VL2-CoT-SFT23.43' in Table 3 is missing a space before the numeric value; it should read 'VL2-CoT-SFT 23.43'.","section":"Table 3"},{"comment":"The phrase 'Fréchet distance distance' contains a duplicated word, and 'ROUGEl' should be 'ROUGE-L'.","section":"Section 3.1.2"},{"comment":"The text uses both 'Single-Stage' and 'One-Stage' for the same strategy; please standardize the terminology throughout.","section":"Sections 2.1 and 3.3"},{"comment":"The metric 'A V-Align' should be written as 'AV-Align' for consistency with the standard name of the metric.","section":"Section 3.1.2"},{"comment":"The paper states that AudioCaps contains 43,941 training instances but does not specify the size of the constructed CoT-AudioCaps dataset or how many instances were used for SFT; please clarify the dataset statistics.","section":"Section 3.1.1"},{"comment":"The training procedure for CoT-SFT does not state whether the intermediate targets V_object and S_event are used as teacher forcing during training or whether model predictions are fed back during training; please clarify the training protocol.","section":"Section 2.3"},{"comment":"The implementation details report the LoRA hyperparameters and GPU type but not the number of training steps, batch size, or validation-based model selection procedure; adding these details would improve reproducibility.","section":"Section 3.1.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its experimental execution, and the proposed task and CoT-SFT strategy are likely to be of interest to the multimedia and audio-generation communities. The main risk is interpretive: because the sound-event supervision is extracted from ground-truth audio captions, the claimed 'modal-mismatch reasoning' may largely reflect memorization of audio-derived labels. I encourage the editor to require the visual-inferability analysis and significance testing described in the major comments before considering acceptance. The manuscript is not ready for acceptance in its current form, but the concerns are addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper does two things: it frames SVAD (audio description from silent video) as a task for VT2A caption acquisition, and it shows a CoT-SFT recipe on a self-built CoT-AudioCaps dataset improves a VLM's generated captions, which then help foley models. The second contribution is real. The CLAP/CIDEr gains over zero-shot and plain SFT are consistent, and the VT2A results with STA-V2A/FoleyCraft show the generated captions close much of the gap to GT prompts. That is a useful engineering result for the video foley community.\n\nWhat's genuinely new: the CoT-AudioCaps dataset (not released) and the empirical finding that decomposing the task into objects -> sound events -> captions improves over direct SFT. The paper also makes an honest comparison against DALI and AVCap, even if only by citation.\n\nNow the soft spots, in order of size.\n\nFirst, the biggest: the paper claims 'modal-mismatch reasoning,' but Algorithm 1 builds the intermediate sound-event labels by asking GPT-4 to parse the ground-truth audio caption. Those labels are not derived from visual evidence. The model is trained to output them from video, so for any sound without a visible correlate (off-screen, room tone, etc.) it can only memorize training-set co-occurrences. The paper never measures how many AudioCaps sounds are visually inferable, so the ceiling on 'reasoning' is unknown. The practical captioning gains could still be real — I think they are — but calling them reasoning is overreach.\n\nSecond, the evaluation lacks error bars and significance tests. The headline CLAP gain from CoT-SFT over SFT is 0.02, which may or may not be stable. Also AVCap-V and DALI are not re-run under the same setup, so some comparisons are indirect.\n\nThird, the CoT-SFT confound. It trains with extra intermediate labels and likely more data/viewing; SFT trains only with final captions. The improvement might come from the added structure or just from seeing more supervised tokens. A simple ablation (same number of steps, or SFT with the subtask labels but no chain) would isolate this.\n\nThe paper is clearly written, the baselines are sensible, and the practical motivation is solid. The citation pattern looks fine. It deserves a serious referee: the SVAD task and the dataset are worth engaging with, but the authors should be pushed to release data/code, add variance estimates, and analyze visual inferability before the reasoning claim can stand.\n\nMy recommendation: send to peer review with major revisions. I wouldn't cite it myself until the dataset is public.","headline":"A practical SVAD pipeline that likely helps VT2A inference, but the 'reasoning' claim needs a lot more evidence than the paper provides.","tokens_in":10255,"tokens_out":2898,"would_cite":false,"duration_ms":31811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a chain-of-thought fine-tuning method that lets a vision-language model write audio descriptions from silent videos, replacing manual annotation in video-to-audio generation.","keywords":["audio description","silent video","vision-language model","modal-mismatch reasoning","chain-of-thought","supervised fine-tuning","video-to-audio","CoT-AudioCaps"],"falsifier":"Take the AudioCaps evaluation clips, have human annotators mark each ground-truth sound as visually inferable or not, then run the CoT-SFT model on the two subsets. If the model's CLAP score on the not-inferable subset is no better than the base VLM's, while the inferable subset shows the reported gain, the claimed reasoning ability reduces to visual association rather than genuine inference.","tokens_in":9265,"feed_emoji":"🔊","tokens_out":7246,"duration_ms":67393,"temperature":0.7,"pith_summary":"The paper introduces a task called SVAD: given only a silent video, a model must infer and describe the sounds that plausibly occurred. It claims that off-the-shelf vision-language models do this poorly, because they describe what they see rather than reason about implied audio. The authors build a dataset of reasoning chains that lead from visual objects to sound events to audio captions, and fine-tune a VLM on those chains. They report that this chain-of-thought fine-tuning raises the semantic match between predicted and real audio, and that the resulting captions work as text prompts for video-to-audio generators, reducing the need for human annotation. If right, the method turns an ordinary VLM into a usable source of audio descriptions for sound synthesis.","feed_headline":"Chain-of-thought tuning lets VLMs infer sounds from silent videos","feed_subtitle":"VLM trained on reasoning chains writes audio captions that nearly match human annotations, removing manual prompts.","key_machinery":"The load-bearing mechanism is CoT-based supervised fine-tuning (CoT-SFT) on a constructed dataset, CoT-AudioCaps. A reasoning chain is a three-step template: visible objects, then plausible sound events, then an audio caption. During training the VLM is shown the silent video and asked each step in sequence; during inference the outputs of earlier steps are fed into the later steps. This decomposes a hard cross-modal inference into smaller steps and is what the paper credits for the measured gains.","core_discovery":"The central claim is that modal-mismatch reasoning can be taught to a vision-language model by supervised fine-tuning on explicit reasoning chains. On the SVAD task, pre-trained VLMs tend to recite visual details and miss implicit sound events, whereas the same VLM fine-tuned on the constructed CoT-AudioCaps data decomposes the problem into three steps: identify objects visible in the video, infer the sound events those objects would produce, and then write an audio caption from the inferred events. This single-stage CoT-SFT raises the CLAP text-audio similarity from 0.252 for the base model to 0.424, and the resulting captions improve downstream video-to-audio generation, narrowing the gap to ground-truth captions on both tested systems.","pith_inferences":["The paper does not measure how many AudioCaps sounds are visually inferable, so the practical ceiling of SVAD is unknown; performance may plateau at the fraction of captions whose sound sources actually appear on screen.","The same three-step decomposition could transfer to other modality-mismatch tasks, such as inferring tactile or olfactory descriptions from video, if paired human annotations can be obtained.","A direct test of the weakest assumption would be to split the evaluation set into sounds with visible sources and sounds without them; CLAP near chance on the latter would indicate the model is associating rather than reasoning."],"forward_implications":["Video-to-audio systems can be run from silent footage alone, since the fine-tuned VLM supplies the text prompt that currently requires a human annotator.","SVAD becomes a testbed for whether VLMs can reason about a modality they never observe, rather than only align modalities they have seen.","The single-stage CoT-SFT outperforms the two-stage pipeline, suggesting that letting the VLM see the raw video during reasoning beats routing through an intermediate video caption.","The reported gains are metric-based: CLAP rises, longer-phrase text metrics improve, and downstream video-to-audio metrics approach those obtained with ground-truth prompts."],"supporting_citations":[{"why":"Supplies the video-audio caption pairs from which CoT-AudioCaps is built and provides the evaluation set for SVAD.","marker":"[27]"},{"why":"Provides the vision-language backbone whose visual understanding and fine-tuning carry the SVAD predictions.","marker":"[22]"},{"why":"Gives the low-rank adaptation method used to fine-tune the VLM without retraining the full model.","marker":"[25]"},{"why":"Introduces chain-of-thought prompting, the reasoning pattern that the CoT-SFT strategy formalizes into training steps.","marker":"[26]"},{"why":"Defines the CLAP metric that measures text-audio similarity and acts as the key SVAD evaluation signal.","marker":"[29]"},{"why":"One of the two downstream video-to-audio models whose inference is improved when it receives CoT-SFT captions as prompts.","marker":"[10]"},{"why":"The second downstream video-to-audio model used to show that generated audio descriptions can substitute for manual prompts.","marker":"[12]"},{"why":"Supplies the video-only captioning baseline whose reported results are compared against in the SVAD evaluation.","marker":"[19]"}],"fun_headline_variants":["CoT fine-tuning makes VLMs hear silent videos accurately","Reasoning chains let VLMs deduce audio from silent clips","VLMs infer sounds from silent videos via explicit reasoning","Chain-of-thought SFT boosts VLMs' audio inference from video","Teaching VLMs modal-mismatch reasoning for silent video audio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes that the visual content of an AudioCaps video carries enough information to infer the ground-truth audio events; when a sound has no visible source, the model cannot reason it out and can only memorize training captions.","fun_headline_variants_meta":{"raw":{"variants":["CoT fine-tuning makes VLMs hear silent videos accurately","Reasoning chains let VLMs deduce audio from silent clips","VLMs infer sounds from silent videos via explicit reasoning","Chain-of-thought SFT boosts VLMs' audio inference from video","Teaching VLMs modal-mismatch reasoning for silent video audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2608,"prompt_tokens":854,"completion_tokens":1754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":1670}},"tokens_in":470,"tokens_out":1754,"duration_ms":13853,"temperature":1.0,"reasoning_tokens":1670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:20:45.375382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the AudioCaps evaluation clips, have human annotators mark each ground-truth sound as visually inferable or not, then run the CoT-SFT model on the two subsets. If the model's CLAP score on the not-inferable subset is no better than the base VLM's, while the inferable subset shows the reported gain, the claimed reasoning ability reduces to visual association rather than genuine inference.","supporting_citations":[{"cited_title":"An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching,","cited_arxiv_id":null,"evidence_quote":"Supplies the video-audio caption pairs from which CoT-AudioCaps is built and provides the evaluation set for SVAD."},{"cited_title":"Automatic video captioning using tree hierarchical deep convolutional neu- ral network and asrnn-bi-directional lstm,","cited_arxiv_id":null,"evidence_quote":"Provides the vision-language backbone whose visual understanding and fine-tuning carry the SVAD predictions."},{"cited_title":"LoRA: Low-Rank Adaptation of Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Gives the low-rank adaptation method used to fine-tune the VLM without retraining the full model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces chain-of-thought prompting, the reasoning pattern that the CoT-SFT strategy formalizes into training steps."}],"review_version":1}