{"id":"0949f24f-3393-4775-b5c9-9a8f7f29cad4","arxiv_id":"2504.18283","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-class audio-visual separator with contrastive alignment lets a pretrained BigGAN generator produce combined and class-separated images from mixed audio, scored with a new detection-based CRS metric.","lead":"This paper introduces a model that takes a mixed audio clip containing two sound classes and generates images: one combined scene plus a separate image per class. It also defines new evaluation metrics for this task and reports gains over a single-class audio-to-image baseline on 20 VGGSound classes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4's convex combination of two separated embeddings is assumed, without demonstration, to lie in the pretrained generator's conditioning manifold; if this linear-compositionality premise fails, the mixed-image generation claim loses its mechanism.","rationale":"Read in good faith: the paper proposes a genuinely new task and an internally consistent training scheme. The A2A-only ablation outperforming A2A and A2A+A2V at least shows the pipeline is not vacuous, and the separation-task CRS values (0.47 for foreground, 0.57 for background) provide some evidence that disjoint halves carry class information. The main unexamined link is the one between the separator's two halves and the generator's ability to compose them. Eq. 4 is the entire mechanism for the combined-image claim, yet no evidence is given that the generator's conditioning space is closed under convex combinations. This is exactly the kind of premise that can silently fail with pretrained GANs: class embeddings are not guaranteed to be linearly interpolable in a semantically meaningful way. The proposed oracle test would settle it. The weak, high-variance quantitative results in Table 1 make the need for such a test more acute. I do not see a reason to change the reader's CONDITIONAL verdict; the concern reinforces it. I agree with the reader's identification of the weakest assumption; the evaluation concerns (single baseline, hand-tuned prompts, no released artifacts) are secondary but real, and would also need to be addressed before the quantitative claims are accepted.","tokens_in":13646,"tokens_out":10156,"duration_ms":104179,"concrete_test":"Take the Sound2Scene audio encoder, compute ground-truth embeddings z_A1_GT and z_A2_GT for unmixed samples of two evaluation classes (e.g., baltimore oriole calling and stream burbling). Generate images with the same pretrained generator G using λ z_A1_GT + (1−λ) z_A2_GT for λ ∈ {0, 0.25, 0.5, 0.75, 1}, and evaluate with the paper's own CRS and R@2* protocol. If the oracle combinations fail to produce both classes at rates comparable to AV-GAS, the linear-compositionality premise behind Eq. 4 is falsified; if they succeed, the premise is supported and the main mechanism stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 and Eq. 4 define mixed-image generation as G(z^N, λ z_AV1 + (1−λ) z_AV2). This presumes that the BigGAN/ICGAN generator pre-trained by Sound2Scene can render a convex combination of two class-specific audio-visual embeddings as an image containing both classes. The separator's halves are trained, via InfoNCE (Eqs. 1–3), to match single-class audio embeddings (A2A) or image embeddings (A2V); nothing in the training objective constrains the sum λ z_AV1 + (1−λ) z_AV2 to be a valid conditioning input, and Sound2Scene's generator was trained on single-class embeddings, not on mixtures or averages. This premise is load-bearing for the first task: without it, reported mixed-image successes are not explained by the proposed audio-visual separation mechanism. The empirical support is also thin: Table 1 reports CRS 0.11±0.12 vs 0.04±0.05 and R@2* 0.06±0.06 vs 0.02, so the claimed 7%/4% gains are within the reported variability. No oracle experiment is run in which G is fed λ z_A1_GT + (1−λ) z_A2_GT from Sound2Scene's own audio encoder, which would directly test whether the generator's latent space is linearly compositional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AV-GAS, an audio-to-image system that takes a mixed audio input containing two sound sources and generates either a single composite image containing both classes or two class-separated images. The method trains an 'audio-visual separator' (a ResNet-18 in the Sound2Scene pipeline) whose 4096-dimensional output is split into two 2048-dimensional embeddings. The first and second halves are aligned, via InfoNCE losses, to the audio and image embeddings of the two ground-truth classes (A2A and A2V alignments). At inference, each half is fed into a pretrained BigGAN/ICGAN generator to produce separated images, and a convex combination of the two halves is fed to the same generator to produce a mixed image (Eq. 4). The paper also introduces two tasks—mixed-audio generation and audio-visual separation—and a new metric, Class Representation Score (CRS), plus a modified R@K. Experiments are conducted on VGGSound, using 20 hand-crafted class combinations of one foreground and one background sound, and the method is compared only against Sound2Scene.","tokens_in":14013,"tokens_out":5174,"duration_ms":54845,"significance":"If the central claims held, the paper would be a useful first step toward a genuinely harder audio-visual generation setting: generating scenes from soundscapes rather than single-class audio. The proposal of a distinct audio-visual separation task and a detector-based class-representation metric are constructive contributions, and the reuse of Sound2Scene's pretrained encoders and generator makes the ablation clean. The paper is also honest about several limitations, including failure cases and the restriction to background/foreground distinctions. However, the evidence is not yet sufficient: the main mechanism relies on an unvalidated linear-compositionality assumption for the generator's latent space; the quantitative gains are small relative to their reported variance; the new metrics depend on manually tuned prompts; and the comparison rests on a single baseline. The core idea is defensible and the missing experiments are well defined, so the paper is repairable within its scope.","major_comments":[{"comment":"The mixed-image generation claim depends on the assumption that the pretrained generator's conditioning space is linearly compositional: the model feeds λz_AV1 + (1−λ)z_AV2 directly to G. Nothing in the InfoNCE training (Eqs. 1–3) constrains this convex combination to be a valid conditioning input for a generator trained only on single-class embeddings. This is load-bearing for the first task. I recommend adding an oracle experiment: feed λz_A1_GT + (1−λ)z_A2_GT (or its visual counterpart) using Sound2Scene's own encoders into G and report CRS and R@2*. If the oracle also produces images containing both classes, the linear-compositionality premise is supported; if not, the reported mixed-image successes are not explained by the proposed separation mechanism. The paper should also report how λ was chosen, since λ is a free parameter that directly controls the trade-off.","section":"§3.2 and Eq. (4)"},{"comment":"The claimed quantitative gains are small relative to the reported variability. For mixed-image generation, CRS is 0.11±0.12 for the proposed A2A method versus 0.04±0.05 for Sound2Scene, and R@2* is 0.06±0.06 versus 0.02±0.00. The '7% higher CRS and 4% higher R@2*' figures are absolute percentage-point differences that lie within one standard deviation of the measurements. No significance test, confidence interval, or per-combination breakdown is provided, so the statement in §4.2.2 that the method 'significantly outperforms' the baseline is not supported by the presented evidence.","section":"§4.2.2, Table 1"},{"comment":"The state-of-the-art comparison is limited to Sound2Scene. The paper justifies this by noting that other single-class audio-to-image methods are either not designed for audio-visual generation or are not publicly available, but the abstract and introduction nevertheless claim state-of-the-art performance and 'first' success on mixed audio. To support that claim, at least one additional baseline that can generate multiple objects or a recent open-source audio-to-image model should be evaluated, or the claim should be explicitly scoped to the Sound2Scene architecture family.","section":"§4.2.1 and Tables 1–3"},{"comment":"CRS and the modified R@K depend on prompts that are manually tuned to the detector and to CLIP. The supplementary rules allow replacing prompts when detection fails (e.g., 'mountain' instead of 'volcano', 'aquarium' instead of 'underwater bubbling'), and the same prompt list is used for both CRS and R@2*/R@1. Because the proposed metrics are a core contribution, they need to be robust to evaluation choices. I ask for a sensitivity analysis: report CRS and R@K with a fixed, pre-registered prompt set, with no substitution rules, and with an alternative detector, to show that the reported ranking is not an artifact of prompt selection.","section":"Supplementary B, Eq. (5)"},{"comment":"The assignment of the first and second halves of the split embedding to foreground and background classes is fixed by the training tuple order ('We aligned the first half of the embedding vector to foreground classes and the second half to background classes'), and no permutation inference is performed at test time. The conclusion itself acknowledges that the method is 'limited to background/foreground distinctions' and lists automated assignment as future work. This means the claim that the model can 'generate separate images for each class present in a mixed audio input' is overstated for arbitrary soundscapes: for two foreground classes or an unseen ordering of the same classes, the model has no mechanism to know which half corresponds to which class. In addition, the statement in §4.1 that 'class labels were not used for training' is in tension with the use of foreground/background ordering to construct training tuples. Please reconcile this and either restrict the claims to the 20 pre-defined (background, foreground) combinations or add a permutation/assignment module.","section":"§3.3.1 and Conclusion"}],"minor_comments":[{"comment":"The column header 'A2A† A2A†' appears to be a typo; one of the two columns should presumably be 'A2V†'.","section":"Table 1"},{"comment":"The control parameter λ is used in Figure 3 but is not defined until Eq. (4); please define it where it first appears or add a forward reference.","section":"Figure 3 and Eq. (4)"},{"comment":"Figure 1 labels 'Task 1: Audio-Visual Generation' and 'Task 2: Audio-Visual Separation', but the caption text and Figure 3 use different numbering conventions that may confuse readers; please standardize the task labels.","section":"Figure 1 caption"},{"comment":"The phrase 'two ground truth separated images that cannot be mixed like audio' is vague; clarify that the two images are kept separate because there is no natural 'mixed image' ground truth for the combined scene.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of the AV-GAS paper. The genuinely new thing is the task: generating a mixed-scene image and per-class separated images from mixed audio. That framing is useful, and the paper is the first I've seen that explicitly tries it. The method is a thin extension of Sound2Scene: a ResNet-18 outputs a 4096-d vector, split in half, each half aligned with single-class audio (and optionally image) embeddings via InfoNCE, then fed to the pre-trained BigGAN generator. The core ablation is internally consistent: A2A alignment beats A2V and A2A+A2V for this task, which is a useful datapoint.\n\nThe paper earns credit for being honest about limitations: only two sources, foreground/background assignment, volume imbalances, GAN failures on people. The qualitative figures show the intended behavior.\n\nThe soft spots are real. Evaluation uses one baseline, Sound2Scene, not designed for mixed audio, so beating it is a low bar. CRS and R@2* are introduced in the same paper, and the prompts are visibly hand-tuned to YOLO-World (e.g., 'mountain' for volcano). Without human evaluation or an independent detector, the metric could be measuring detector artifacts. Absolute scores are low: CRS 0.11 vs 0.04, R@2* 0.06 vs 0.02, with overlapping standard deviations. The stress-test concern about Eq. 4 is valid: the convex combination of two separated embeddings is fed to a generator trained on single-class embeddings, and nothing in the training objective ensures the mixture stays in the generator's input manifold. An oracle test using ground-truth audio embeddings would have settled that cheaply; its absence is a genuine gap.\n\nAll that said, the paper deserves a serious referee. The task is legitimate, the method is simple and reproducible, and a good referee can push for stronger baselines, the oracle compositionality test, and release of code and prompts. I'd rather see this in the literature after revision than not at all. Would I cite it? Only if I worked on audio-visual generation and needed to reference the mixed-audio formulation; the method itself is not yet compelling enough to build on.","headline":"New task framing for mixed-audio image generation, but the method is a thin extension and the evaluation is too weak to support the state-of-the-art claim.","tokens_in":14501,"tokens_out":3417,"would_cite":false,"duration_ms":29983,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mixed audio containing two sound classes can be split into two class-aligned embeddings, letting a pretrained image generator draw either a combined scene or a separate image for each class.","keywords":["audio-visual generation","mixed audio","soundscape","audio-visual separation","contrastive learning","Class Representation Score","VGGSound","image generation from audio"],"falsifier":"Take a held-out pair of classes outside the 20 training combinations, such as 'dog barking' plus 'playing harp', feed the mixed audio at $\\lambda = 0.5$, and score the resulting image with an open-vocabulary detector for both prompts. If both classes are not detected at a rate significantly above the single-class baseline, the claimed linear composition does not generalise beyond the trained pairs.","tokens_in":13419,"feed_emoji":"🎨","tokens_out":7674,"duration_ms":63470,"temperature":0.7,"pith_summary":"This paper introduces AV-GAS, a model that generates images from soundscapes—audio containing several sound classes at once—rather than only from single-class audio. It claims to be the first to produce both a single image containing all classes and one separate image per class from the same mixed audio. The paper also proposes new evaluation metrics, Class Representation Score (CRS) and a modified R@K, and reports that its method outperforms the state of the art by 7% in CRS and 4% in R@2* on the VGGSound dataset. The central idea is that a contrastively trained separator can split a mixed-audio embedding into two halves aligned with the two source classes, and that a weighted sum of those halves yields a combined image.","feed_headline":"Mixed audio now generates a combined scene or per-class images","feed_subtitle":"AV-GAS splits a mixed sound into class embeddings, then feeds each half or a weighted sum to an image generator.","key_machinery":"The load-bearing object is the audio-visual separator $f_{AV}$, a ResNet-18 that maps mixed audio to a single 4096-dimensional embedding $z^{AV}_{mix}$, with the first 2048 dimensions trained to match one class and the second 2048 to match the other. Training uses InfoNCE contrastive losses that pull each half toward the corresponding unmixed audio embedding (A2A alignment) and optionally toward the corresponding image embedding (A2V alignment), without class labels. At inference, the pretrained image generator $G$ produces separated images from each half and a combined image from $\\lambda z^{AV}_1 + (1-\\lambda) z^{AV}_2$, where $\\lambda$ controls the blend. The whole argument rests on this linear split-and-recombine operation being valid in the pretrained latent space.","core_discovery":"The paper's central claim is that mixed-audio generation reduces to an embedding problem rather than a generator problem. Given a pretrained image generator, a separator network can map a mixed audio signal to a 4096-dimensional vector whose first and second halves are contrastively aligned to the two source classes; feeding either half to the generator produces a class-separated image, and feeding a weighted sum produces a single image containing both classes. The authors report that audio-to-audio alignment alone gives the best results, with CRS 7 points higher and R@2* 4 points higher than the state-of-the-art baseline on mixed-audio generation, and that separated foreground and background images reach CRS values of 47% and 57% respectively. They also define the new audio-visual separation task of generating one image per class from a mixed input, which prior single-class audio-to-image models cannot do.","pith_inferences":["If the latent space is truly linearly compositional, the same split-and-recombine trick could be transferred to other pretrained generators, including diffusion-based text-to-image models, provided their conditioning latents are approximately linear; the paper does not make this extension.","A direct next test is whether the separator can be trained on more than two sources by splitting the embedding into more segments; the paper lists this as future work, and the architecture does not obviously forbid it.","Because the separator is trained without class labels and the half-ordering is fixed by training tuple order, test-time inference has no mechanism to know which half is foreground and which is background; downstream uses would need a permutation-invariant step.","The CRS metric could be reused to evaluate multi-object text-to-image generation, where the same 'are all requested objects present?' question arises in a neighbouring task."],"forward_implications":["From a mixed audio input, the model can generate a single image containing all classes present, a capability single-class audio-to-image models lack.","The same mixed audio also yields one separate image per class, defining the new audio-visual separation task.","The proposed Class Representation Score (CRS) measures directly whether all audio classes appear in a generated image and can be applied to any multi-class generation model.","Reported results on VGGSound are 7% higher CRS and 4% higher R@2* than the state of the art for mixed-audio image generation.","The model also produces plausible images for unrealistic mixtures (e.g., an airplane over underwater bubbling), suggesting the separated embeddings carry class semantics rather than memorised training scenes."],"supporting_citations":[{"why":"Supplies the pretrained audio and image encoders and the pretrained image generator that the separator aligns to and that produces the final images.","marker":"[20]"},{"why":"VGGSound is the large-scale audio-visual dataset from which all training tuples and evaluation samples are drawn.","marker":"[6]"},{"why":"An open-vocabulary object detector used to compute the proposed Class Representation Score.","marker":"[10]"},{"why":"A contrastive vision-language model used to rank candidate class labels against generated images for the R@K metrics.","marker":"[28]"},{"why":"The architecture of the pretrained image generator used at inference.","marker":"[2]"},{"why":"Provides the instance-conditioning method that lets the generator accept the separated embeddings.","marker":"[3]"},{"why":"The convolutional backbone architecture used by the audio-visual separator, chosen to match the baseline's encoder.","marker":"[16]"}],"fun_headline_variants":["AV-GAS splits sound into class-specific images","Mixed audio? Now you get per-class images","Soundscape to scene: one clip, two image types","New audio-visual separation: per-class images from a single clip","7% higher CRS, 4% higher R@2*: mixed audio wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the latent space of the pretrained audio and image encoders and the image generator is linearly compositional: splitting a mixed-audio embedding into two halves, and later recombining them with a scalar weight, preserves each class's identity in the generated image.","fun_headline_variants_meta":{"raw":{"variants":["AV-GAS splits sound into class-specific images","Mixed audio? Now you get per-class images","Soundscape to scene: one clip, two image types","New audio-visual separation: per-class images from a single clip","7% higher CRS, 4% higher R@2*: mixed audio wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001238,"raw_usage":{"total_tokens":5075,"prompt_tokens":931,"completion_tokens":4144,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":4058}},"tokens_in":547,"tokens_out":4144,"duration_ms":27682,"temperature":1.0,"reasoning_tokens":4058,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:20:08.209084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out pair of classes outside the 20 training combinations, such as 'dog barking' plus 'playing harp', feed the mixed audio at $\\lambda = 0.5$, and score the resulting image with an open-vocabulary detector for both prompts. If both classes are not detected at a rate significantly above the single-class baseline, the claimed linear composition does not generalise beyond the trained pairs.","supporting_citations":[{"cited_title":"Sound to visual scene generation by audio-to-visual latent alignment","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained audio and image encoders and the pretrained image generator that the separator aligns to and that produces the final images."},{"cited_title":"Vggsound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"VGGSound is the large-scale audio-visual dataset from which all training tuples and evaluation samples are drawn."},{"cited_title":"Yolo-world: Real-time open- vocabulary object detection","cited_arxiv_id":null,"evidence_quote":"An open-vocabulary object detector used to compute the proposed Class Representation Score."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"A contrastive vision-language model used to rank candidate class labels against generated images for the R@K metrics."},{"cited_title":"Instance- conditioned gan","cited_arxiv_id":null,"evidence_quote":"Provides the instance-conditioning method that lets the generator accept the separated embeddings."},{"cited_title":"Zhang, Shaoqing Ren, and Jian Sun","cited_arxiv_id":null,"evidence_quote":"The convolutional backbone architecture used by the audio-visual separator, chosen to match the baseline's encoder."}],"review_version":1}