{"id":"50042302-573a-45d1-8ca7-36529f5ad2d8","arxiv_id":"2411.15447","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A sound-source-aware image-to-audio generator that detects objects, disambiguates their audio semantics in a learned cross-modal manifold, and mixes them to synthesize audio.","lead":"Researchers built an audio generator that first finds the individual objects in an image that could make sound, like a dog or a guitar, then mixes their learned sound descriptions to create the final audio. The approach improves relevance and gives intuitive control by letting users add text or sound snippets as extra conditions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on the VGG-SS evaluation protocol, and the paper does not show that SS2A wins without test-set hyperparameter tuning.","rationale":"The reader's verdict is CONDITIONAL and identifies missing baselines, missing CIs, and no code as weaknesses. I agree with those, but the single most load-bearing concern is more specific: the paper's central quantitative claims for multi-source generation come from a test set on which the model's hyperparameters were explicitly tuned. The ablation tables (Tab. 4 α, Tab. 8 Cycle Mix, Tab. 9 CMSS variant, Tab. 10 Remixer layers) each report the chosen value as best on VGG-SS, and the paper states VGG-SS has no official split, with the authors creating their own random split. That makes the headline numbers on VGG-SS optimistically biased. This is not internal inconsistency—the method and ablations are logically coherent—but it is a correctness risk in the empirical claim. A practical fix is to evaluate the final frozen model on a disjoint split or an external source-annotated benchmark. Until that is done, the strength of the central claim should stay conditional. I would not move to REJECT because the method has independent plausibility and the reported gains on VGG-SS are large, but I would require the corrected evaluation before accepting the SOTA claim.","tokens_in":22198,"tokens_out":1225,"duration_ms":11919,"concrete_test":"Re-evaluate the fixed SS2A model (with the reported α=0.35, variant B, 1-layer Remixer, and Cycle Mix 64/64) on a freshly held-out split of VGG-SS, or on the VGG-SS version from prior work that does not overlap with the authors' random split. If SS2A's multi-source SSMS/V-FAD margins over V2A-Mapper shrink to within the variability of a 38-pair test set (e.g., bootstrap confidence intervals overlap), then the 'superior in all metrics' claim is not supported. The check is computational and requires no new training if the authors provide the trained modules; otherwise retraining with their reported recipe on a 4,000/1,000 split and reporting the resulting metrics would settle whether the selected hyperparameters generalize.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper claims state-of-the-art image-to-audio generation, but the strongest evidence for the claim is the VGG-SS tables (Tab. 2, Tab. 4, Tab. 8, Tab. 9, Tab. 10), where the authors tune module architectures, CCMR strength α, Cycle Mix iterations, and Remixer attention depth. VGG-SS is also used as the only source-annotated test set (455 single-source, 38 multi-source pairs), and the paper states it has 'no official train-test split.' Training on a random 4.5K/500 split and then selecting the best α (0.35 vs 0, 0.65, 1), the best CMSS architecture (variant B), and the best Remixer depth (1 vs 2, 4) on that same 500-pair test set means the reported VGG-SS numbers are the result of test-set selection. This is the load-bearing evidence for the 'superior in all metrics' multi-source claim. The concern is not that the method is fraudulent or that these components are useless; it is that the central claim—explicit sound source modeling gives measured gains—risks being overstated, because the gains are largest on the dataset used for hyperparameter selection, and no held-out validation set or corrected evaluation is reported. The reader's weakest_assumption (YOLOv8 detection quality) is plausible, but the paper's own ablations (Tab. 3, e.g., V-FAD 39.6→2.8) show that detection-based source modeling is what drives the effect, so the failure mode would be a uniform degradation, not a selective advantage on VGG-SS. The evaluation protocol is the more decisive and testable vulnerability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SS2A, a sound-source-aware image-to-audio generator. SS2A first perceives sound sources in an image with a visual detector (YOLOv8x), embeds the detected crops with CLIP and optionally takes text/audio prompts via CLIP/CLAP, projects these into a learned Cross-Modal Sound Source (CMSS) manifold using a contrastive objective with a new Cross-Modal Contrastive Mask Regularization (CCMR), and then mixes the source embeddings with an attention-based Sound Source Remixer before feeding the resulting CLAP embedding into a pretrained AudioLDM generator. The paper also introduces a new single-source dataset VGGS3, a new Sound Source Matching Score (SSMS) metric, and a temporal aggregation extension for video-to-audio. The experimental section reports objective and subjective results on VGGSound, VGG-SS, MUSIC, and ImageHear, together with ablations of the CMSS manifold, CCMR, Cycle Mix, and Remixer architecture.","tokens_in":22605,"tokens_out":7838,"duration_ms":70312,"significance":"If the central claim holds, the paper makes a useful contribution by showing that explicitly modeling local sound sources, rather than conditioning only on a global scene, improves the relevance and multi-source fidelity of generated audio. The VGGS3 dataset and SSMS metric are potentially reusable resources, and the multimodal composition experiments demonstrate an interesting new control modality. The evidence is extensive in terms of datasets, ablations, out-of-distribution tests, and human MOS. However, the paper's headline state-of-the-art claim is currently overstated: on general VGGSound and single-source VGG-SS, a baseline (V2A-Mapper) has better FAD scores, and the key multi-source VGG-SS results are obtained after selecting several hyperparameters on the same small test set. The work is therefore a promising empirical direction whose central claim needs a corrected evaluation protocol before it can be accepted at face value.","major_comments":[{"comment":"The VGG-SS test set is used both for hyperparameter selection and for reporting final results. The paper states that VGG-SS has no official train-test split and randomly samples 4.5K pairs for training and 500 pairs for testing. The ablations then select CCMR alpha (Tab. 4), Cycle Mix sample size and iterations (Tab. 8), CMSS architecture variant (Tab. 9), and Remixer attention depth (Tab. 10) by evaluating on the same 500-pair test set that is reported in Tab. 2. Since the multi-source portion of VGG-SS contains only 38 pairs, repeated selection on this set likely inflates the reported multi-source gains. Please add a proper held-out validation split (or nested cross-validation) for all hyperparameter choices, report final results on a truly unseen test set, and provide confidence intervals or significance tests for the multi-source comparisons.","section":"Datasets and Tables 4, 8, 9, 10"},{"comment":"The claim that SS2A 'achieves state-of-the-art performance in extensive image-to-audio tasks' is broader than the data support. In Tab. 1, V2A-Mapper has better V-FAD (0.946 vs. 1.150) and C-FAD (5.516 vs. 6.716) on the general VGGSound test set. In Tab. 2, for single-source VGG-SS, V2A-Mapper also has better V-FAD (1.666 vs. 2.815) and C-FAD (13.583 vs. 15.150). The multi-source VGG-SS and MUSIC results are favorable to SS2A, and the relevance metrics and MOS favor SS2A, but the 'state-of-the-art in all metrics' language should be restricted to the multi-source setting, and the single-source fidelity trade-off should be discussed explicitly.","section":"Abstract, Conclusion, and Tables 1-2"},{"comment":"The SSMS metric is introduced as a contribution, but its key parameter N (the number of predicted labels) is fixed to 10 without sensitivity analysis, and its validity is argued mainly through the observation that it places Oracle below SS2A on CS. Please report SSMS for several values of N, compare SSMS against the human relevance MOS on the same set of samples, and analyze whether the BEATs label vocabulary covers the VGG-SS and MUSIC source categories. Without this, the metric's role as a clear measure of localized audio relevance is not fully established.","section":"Matching Score"}],"minor_comments":[{"comment":"The mask definition uses alpha twice, once as a coefficient and once as an exponent; the behavior is therefore hard to interpret. Consider renaming one of the two occurrences and adding a short explanation of how the mask scales with the CLIP and CLAP similarities.","section":"Equation (7)"},{"comment":"The header 'CMSS CLIP' is ambiguous; the caption should state explicitly which rows correspond to removing the CMSS embeddings and which to removing the CLIP embeddings.","section":"Table 3"},{"comment":"The column header 'MS' is used for the metric that is called 'SSMS' in the main text and other tables; please use a consistent abbreviation throughout.","section":"Tables 8-10"},{"comment":"The text says 1024 is the positional embedding's frequency resolution, while 1024 appears as the base in the denominator of the positional encoding; please clarify the notation so that the reader understands the difference between the base and the resolution.","section":"Temporal Aggregation, Eq. (11)-(12)"},{"comment":"The demo website URL uses 'SSV2A' while the method is named 'SS2A'; please align the naming to avoid confusion.","section":"Abstract and Demo URL"},{"comment":"The paper says the curated dataset VGGS3 will be released, but no release link or availability statement is given; please include one in the final version.","section":"Ethical Statement"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the evaluation protocol: the VGG-SS test set is used both for hyperparameter selection and for final reporting, which is fixable with a proper validation split. The central idea is sound and the ablations support the benefit of source-level conditioning, but the state-of-the-art language should be moderated. I recommend a major revision rather than rejection because the empirical claims are likely salvageable with corrected evaluation and more careful wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: this is a real contribution to image-to-audio and video-to-audio generation, and the core idea—explicitly detecting and disambiguating local sound sources instead of conditioning on a global scene—holds up better than I expected. The VGGS3 dataset (106K single-source pairs), the SSMS metric, and the CMSS/CCMR machinery are all genuinely new, and the paper is dense with ablations, out-of-distribution tests, and a human MOS study. The authors also acknowledge real limitations (naive temporal module, weaker audio conditioning) in the supplementary, which I read as honest engagement.\n\nThe soft spot is the evaluation protocol for the headline claim. VGG-SS has no official train/test split, so the authors randomly carve out 455 single-source and 38 multi-source pairs as a test set. Then they select CCMR alpha, Cycle Mix size/iterations, CMSS architecture variant, and Remixer depth by running ablations on that same test set (Tables 4, 8, 9, 10) and report the best configuration's VGG-SS numbers as state of the art. That is test-set tuning, and it means the 'superior in all metrics' multi-source claim is partly an artifact of the selection procedure. The gains are largest on the dataset used for selection. This is not fatal—the method clearly does something right, and the detection-driven ablation (Table 3) shows the effect is real—but it means the published margins are optimistic. Also missing: confidence intervals, code/data release, and a direct comparison with the object-aware baseline (Li et al. 2024) that is only cited in passing.\n\nThe YOLOv8 detection assumption the reader worried about is less of a problem in practice. The paper's own ablations show detection-based source modeling is what drives the effect, so a missed source would degrade all configurations roughly uniformly, not selectively favor SS2A on VGG-SS.\n\nWho this is for: anyone working on audio-visual synthesis, especially V2A/I2A conditioning. The dataset and metric are worth having regardless of the SOTA claim. I'd send it to peer review, but the authors need to fix the evaluation protocol before it's publishable—held-out validation, error bars, and at least one honest comparison without test-set tuning.\n\nRecommendation: accept with major revision, on the condition they address the tuning issue.","headline":"A sound-source-aware audio generation system with real artifacts (VGGS3, SSMS, CMSS) and a plausible core idea, but the SOTA claim is inflated by tuning hyperparameters on the same VGG-SS test set used for the headline numbers.","tokens_in":23136,"tokens_out":2520,"would_cite":true,"duration_ms":22632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Explicitly modeling local sound sources—detected visually and disambiguated in a cross-modal manifold—improves image-to-audio generation, the paper claims.","keywords":["image-to-audio generation","sound source modeling","cross-modal contrastive learning","vision-to-audio synthesis","multimodal composition","sound source matching score","audio generation","VGGS3 dataset"],"falsifier":"A direct test: build a set of images in which the real sound maker is not visually detectable (a bird behind a wall, an offscreen engine) while detected objects are silent. If SS2A without a text prompt performs as well as SS2A with the prompt, or as well as a global-scene baseline, the visual source-perception step is not doing the load-bearing work; the measurable outcome is SSMS and CLIP-Score on that set.","tokens_in":22004,"feed_emoji":"🔊","tokens_out":8416,"duration_ms":67138,"temperature":0.7,"pith_summary":"The paper sets out to show that image-to-audio generation improves when the model is told about the individual sound sources in a scene rather than only the whole scene. To that end it proposes SS2A, a generator that detects objects in an image, projects their CLIP or CLAP embeddings through a contrastively learned Cross-Modal Sound Source (CMSS) manifold that separates one source from another, and then attention-mixes the source embeddings into a single CLAP audio embedding that a pretrained AudioLDM turns into a waveform. The authors claim that this source-aware conditioning surpasses existing global-scene methods on fidelity, relevance, and human opinion scores, and that it works best for scenes with several sound sources at once. A reader would care because the method points toward audio synthesis that can be controlled at the level of individual objects and combined across vision, text, and audio prompts.","feed_headline":"Object-level sound sources, not whole scenes, boost audio synthesis","feed_subtitle":"Detecting and separating each sound source yields more relevant, higher-fidelity audio than whole-scene conditioning.","key_machinery":"The load-bearing mechanism is the Cross-Modal Sound Source (CMSS) manifold: a joint semantic space, learned contrastively from single-source image-audio pairs, into which CLIP and CLAP embeddings of detected sound sources are projected so that matching visual and audio sources attract and non-matching sources repel. An auxiliary reconstructor maps CMSS embeddings back toward CLAP audio space so the manifold keeps audio character, and a Cross-Modal Contrastive Mask Regularization (CCMR) weakens the contrastive pressure on pairs that are already similar in both CLIP and CLAP space. A separate attention-based Sound Source Remixer takes the disambiguated CMSS embeddings, concatenated with their original CLIP embeddings, and mixes them into one CLAP embedding that is fed to the frozen pretrained AudioLDM generator.","core_discovery":"On the paper's own terms, the central discovery is that a sound source-aware conditioning path—visual detection of objects, cross-modal translation of text or audio prompts into the same semantic space, and contrastive disambiguation of each source before mixing—lets a frozen audio diffusion model produce audio that is more relevant and more immersive than conditioning on the global image alone. The paper builds this path as SS2A and evaluates it on both single-source and multi-source image-to-audio benchmarks. It reports that SS2A achieves the best relevance and competitive fidelity in general image-to-audio tests, is superior to all compared baselines on every metric in multi-source generation, and reaches competitive video-to-audio results after attaching a temporal aggregation module. It also introduces VGGS3, a dataset of single-sound-source image-audio pairs filtered from VGGSound, and a Sound Source Matching Score (SSMS) that measures localized relevance as an F1 score over predicted sound labels.","pith_inferences":["The detector threshold is a hidden bottleneck: if a real sound source has no visual bounding box, SS2A cannot condition on it, so the method's ceiling is tied to detection quality rather than to the manifold or remixer.","Because SSMS relies on an audio classifier's predicted labels, it inherits that classifier's vocabulary and blind spots; extending SSMS to another label set or open vocabulary would test whether the metric itself is stable.","The same architecture suggests a natural interactive editing loop: replacing or deleting one source token in the remixer's input should change only that part of the generated scene, which is testable as a controllability benchmark.","The reported lower sensitivity to audio prompts indicates that supplying CLIP-style semantics for audio conditions, for instance through a Wav2CLIP-style translator, is a concrete next step suggested by the paper's own limitation section."],"forward_implications":["If SS2A is right, conditioning on detected local objects should improve audio fidelity and relevance on arbitrary images, with the largest gains in scenes that contain multiple sounding objects.","A contrastive manifold trained only on single-source pairs transfers to multi-source mixing, so object-level audio semantics can be learned from simpler data and composed at generation time.","Because all conditions are projected into one manifold, users can mix vision, text, and audio prompts to edit or steer a generated sound scene.","The same source-level representation, combined with a positional temporal aggregator, yields competitive video-to-audio synthesis, suggesting source modeling is not image-specific.","The SSMS metric gives a source-level relevance score that can be measured automatically and that the authors show can disagree with image-domain relevance scores like CLIP-Score."],"supporting_citations":[{"why":"Supplies the CLIP image and text encoders whose embeddings carry visual and textual source semantics throughout the pipeline.","marker":"Radford et al. 2021"},{"why":"Supplies the CLAP audio encoder whose embedding space is both the audio-source condition space and the target representation that AudioLDM consumes.","marker":"Elizalde et al. 2023"},{"why":"Provides the frozen AudioLDM generator and its CLAP audio embedding as the final synthesis backbone that turns the mixed representation into a waveform.","marker":"Liu et al. 2023"},{"why":"VGGSound is the large image-audio corpus from which the single-source dataset VGGS3 is filtered and on which the Remixer is trained and tested.","marker":"Chen et al. 2020a"},{"why":"VGG-SS supplies the manually annotated single- and multi-source visual-audio pairs used to train the teacher manifold and to evaluate source-annotated generation.","marker":"Chen et al. 2021"},{"why":"The YOLOv8x detector trained on OpenImagesV7 produces the object crops that define the visual sound sources, so the whole perception stage depends on it.","marker":"Jocher, Chaurasia, and Qiu 2023"},{"why":"The DALL·E-2 prior translates text prompts and translated text-audio pairs into CLIP image space, letting text and audio sources enter the same visual semantic space.","marker":"Ramesh et al. 2022"},{"why":"BEATs is the audio classifier that predicts sound source labels for both ground-truth and generated audio, making the SSMS relevance metric computable.","marker":"Chen et al. 2023"},{"why":"V2A-Mapper is both the main image-to-audio baseline and the source of the temporal aggregation idea adapted for video-to-audio.","marker":"Wang et al. 2024a"},{"why":"Mean-Teacher training is used to supervise the CMSS modules with noisy curated and translated pairs, grounding the contrastive manifold learning.","marker":"Tarvainen and Valpola 2017"}],"fun_headline_variants":["Audio generation improves when sound sources are modeled locally","Object-level sound source modeling beats whole-scene audio synthesis","SS2A: Detect each sound source to generate richer audio","Perceiving local sound sources yields higher-fidelity audio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the visual detector reliably finds the objects that actually make sound: any source that is missed, too small, visually ambiguous, or silent-but-cropped will be absent from, or pollute, the conditioning, and the rest of the pipeline cannot correct for that.","fun_headline_variants_meta":{"raw":{"variants":["Audio generation improves when sound sources are modeled locally","Object-level sound source modeling beats whole-scene audio synthesis","SS2A: Detect each sound source to generate richer audio","Perceiving local sound sources yields higher-fidelity audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":3082,"prompt_tokens":996,"completion_tokens":2086,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":2020}},"tokens_in":612,"tokens_out":2086,"duration_ms":14118,"temperature":1.0,"reasoning_tokens":2020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:17:39.778309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: build a set of images in which the real sound maker is not visually detectable (a bird behind a wall, an offscreen engine) while detected objects are silent. If SS2A without a text prompt performs as well as SS2A with the prompt, or as well as a global-scene baseline, the visual source-perception step is not doing the load-bearing work; the measurable outcome is SSMS and CLIP-Score on that set.","supporting_citations":[{"cited_title":"CLAP learning audio concepts from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLAP audio encoder whose embedding space is both the audio-source condition space and the target representation that AudioLDM consumes."},{"cited_title":"Ultralytics YOLOv8 , 2023","cited_arxiv_id":null,"evidence_quote":"The YOLOv8x detector trained on OpenImagesV7 produces the object crops that define the visual sound sources, so the whole perception stage depends on it."},{"cited_title":"Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results","cited_arxiv_id":null,"evidence_quote":"Mean-Teacher training is used to supervise the CMSS modules with noisy curated and translated pairs, grounding the contrastive manifold learning."}],"review_version":1}