{"id":"ba863762-ab0d-47dc-95c0-b58ad686c83c","arxiv_id":"2507.18750","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CatchPhrase improves audio-to-image generation by enriching weak class labels with LLM- and audio-caption-based prompts, filtering and retrieving the best prompt per clip, and training a mapping adapter with contrastive learning.","lead":"This paper presents CatchPhrase, a system that generates images from audio by enriching sound labels with detailed text descriptions, then selecting the best description for each sound clip to train a lightweight adapter. It reports clearer images for confusing cases like homographs and auditory illusions, beating several prior methods on alignment benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AIS evaluation may be circular: CLAP is used both as the prompt filter/retriever and as the alignment metric, so reported gains could reflect prompt-side similarity rather than image fidelity.","rationale":"The reader identified CLAP oracle reliance as the weakest assumption; however, the dominant concern is more specific than that: AIS is not merely an unvalidated metric but is likely the metric that the method explicitly optimizes through CLAP-based selection. This is a circular-evaluation risk, not only a risk of noisy prompts. The paper does provide some independent support: qualitative figures, user studies, and ablations that improve generation quality. The user study (Table 14) is a genuine external signal, but it is limited (15 pairs, 30 participants, selecting from only two models) and evaluates generated images, not the adapter's fidelity to audio. The retrieval experiment in the supplementary material actually confirms that EXPrompts are CLAP-similar to audio, which strengthens the circularity concern rather than alleviating it. The InfoNCE contribution is marginal (differences of 0.0005-0.0007 on AIS), and the component ablation shows that the main gains come from the prompt selection pipeline itself. The remaining independent argument is the qualitative improvement shown in Figures 5-8, which is plausible but small in scale. Therefore the paper should be CONDITIONAL: the central claim that CatchPhrase improves audio-to-image alignment by mitigating semantic misalignment is not yet externally validated. The stated conditional concerns (error bars, held-out hyperparameter selection, marginal InfoNCE contribution) are secondary; the circular AIS evaluation is the load-bearing issue.","tokens_in":16631,"tokens_out":1567,"duration_ms":14925,"concrete_test":"Recompute the headline AIS comparison after removing the EXPrompt selection from the evaluation loop: hold the trained adapter fixed but generate images from (a) the retrieved EXPrompts and (b) the raw class-template prompts, then compute AIS with an independent metric (e.g., ImageBind-audio-image similarity or human ratings) instead of the CLAP-style metric. If the fixed adapter's AIS gain disappears or reverses when evaluated with a non-CLAP metric, the claimed improvement is not a model-level semantic alignment improvement.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing concern is that the headline AIS gains are not shown to measure genuine image-semantic alignment. In Section 3.3, CLAP is used as the oracle for both filtering (Eq. 4) and retrieval (Eq. 5), and in Section 4.1 the primary metric AIS is also computed with CLAP-style audio-text/image similarity. The training objective directly optimizes CLAP-space similarity: filtered/retrieved prompts are selected as the CLAP-nearest text for each audio, and the mapping network is trained with MSE and InfoNCE losses to move mapped audio features toward text-encoder features of those selected prompts. Thus a model that merely outputs images whose CLIP text embeddings match the selected prompt can inflate AIS without improving true semantic alignment. This circularity is compounded by using CLAP for both mining-related selection and for the retrieval-based ablation in the supplementary material (Audio-to-Text retrieval), which confirms that the selected prompts are CLAP-near but does not independently validate semantic quality. The paper does not report whether AIS is computed with the same CLAP checkpoint used for filtering, nor does it provide human or external verification of alignment. The InfoNCE contribution is also tiny (0.0005-0.0007), suggesting that the measured gains are accounted for by prompt selection rather than by a learned adapter or contrastive objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CatchPhrase proposes a framework for audio-to-image generation that mitigates semantic misalignment caused by homographs and auditory illusions. The method first mines enriched textual prompts (EXPrompts) from weak class labels using LLMs and audio captioning models, then applies a CLAP-based filtering and retrieval procedure to select one semantically aligned prompt per audio instance, and finally trains a lightweight mapping network that adapts a frozen text-to-image diffusion model to audio features, using MSE, reconstruction, adversarial, and InfoNCE losses. Experiments on UrbanSound8K, ESC50, and VGGSound report higher Audio-Image Similarity (AIS) than Sound2Scene, GlueGen, and SonicDiffusion, together with improved FID/KID, plus ablations showing the contribution of each component. The paper claims that the enriched prompts and the prompt selector reduce both class-level and instance-level misalignment.","tokens_in":16913,"tokens_out":2511,"duration_ms":28564,"significance":"If the reported gains are robust, CatchPhrase would be a practically useful contribution to audio-to-image generation: it addresses a real failure mode (weak labels in audio classification datasets causing homograph and auditory-illusion errors), it is built from modular and reproducible components, and it includes ablations, a retrieval-based sanity check, and a small user study. The method's reliance on CLAP both as the prompt-selection oracle and as the basis for the evaluation metric is, however, a serious validation concern, and the absence of uncertainty quantification makes the small reported differences difficult to interpret. The central idea is defensible, but the paper currently does not establish that the AIS gains reflect genuine image-semantic alignment rather than prompt-side similarity.","major_comments":[{"comment":"The evaluation may be partly circular: the same cross-modal similarity paradigm (CLAP-style audio-text embeddings) is used both to filter and retrieve training prompts in Eq. (4)-(5) and to compute the headline AIS metric in Section 4.1. The paper does not state whether the AIS checkpoint is the same CLAP model used in the selector, nor does it provide an independent (human or external model) validation of semantic alignment. Since the training objective directly moves mapped audio features toward text features of CLAP-selected prompts, a model that merely matches those prompt embeddings could inflate AIS without improving true image semantics. Please report the exact AIS computation, disclose the checkpoint, and add an evaluation that does not rely on the same embedding space used for prompt selection, e.g., human judgment or a separately trained audio-image alignment model.","section":"§3.3, §4.1, Eq. (4)-(5)"},{"comment":"All quantitative claims are made without error bars, multiple seeds, or significance tests. The headline differences are often small (e.g., VGGSound AIS 0.2017 vs 0.1950 for GlueGen), and the InfoNCE ablation in Table 6 reports differences of 0.0005-0.0007 on AIS. With a single fixed seed, it is impossible to determine whether any of these differences are reliable. Please run at least 3-5 seeds and report mean and standard deviation, or otherwise justify that the differences are above the noise floor.","section":"§4.2, Table 1, §4.1 'random seed is fixed at 42'"},{"comment":"The filtering hyperparameters NAS and top-K are swept over the values shown in Supplementary Table 9 and the best values are selected using AIS, which is the same metric used for the main comparisons. Since no held-out validation split or nested selection procedure is described, the reported numbers are likely optimistically biased. Please specify the validation protocol used for hyperparameter selection and report test performance under that protocol.","section":"Supplementary Table 9 and Section 4.3 'Component Sequence Ablation Studies'"},{"comment":"Table 6 shows that InfoNCE contributes only 0.0005 (US8K), 0.0001 (ESC50), and 0.0007 (VGGSound) to AIS. Given the lack of uncertainty quantification, the text's claim that the InfoNCE loss 'contributes to mitigating auditory illusions' (Section 4.3) is not supported by the presented evidence. If this loss is retained as a core contribution, please provide a more sensitive evaluation (e.g., instance-level human judgments on auditory-illusion examples) or soften the claim.","section":"§3.4, Table 6"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'adpater' (Figure 3 and text), 'auido' (Table 7 caption), 'confusion cased by homographs' (Section 1), and inconsistent spacing around equations and in Table 6. A careful proofread is needed.","section":"Throughout"},{"comment":"The notation for the filtering score is confusing: the variable r_p seems to be both a single prompt embedding and an element of a set, and the indices i, j are not fully defined beyond 'I is the set of audio samples with the same class label as the prompt'. Please clarify the indexing and the exact aggregation over the NAS audio samples.","section":"§3.3, Eq. (4)"},{"comment":"The loss weights are described only as 'determined empirically'. Please report the range considered and the selection criterion, or at least state that the weights were fixed before evaluating the main comparisons.","section":"§3.4, Eq. (7)"},{"comment":"The paper says AIS follows [51], but it is not described precisely enough to be reproducible: which audio encoder, which image encoder, and what aggregation over generated images are used? Please provide the exact AIS computation or a reference to an implementation.","section":"§4.1 'Evaluation Metrics'"},{"comment":"The audio-to-text retrieval results (R@1) are reported as raw percentages, but the comparison is only between CatchPhrase prompts and a single template set; the retrieval experiment does not control for the number of candidate prompts or the effect of prompt diversity. Please state the candidate pool size and how ties are handled.","section":"Supplementary, Table 13"},{"comment":"The user study is small (30 participants, 15 pairs) and the statistical significance of the 64.22% vs 35.78% split is not reported. Please add a confidence interval or a significance test, and specify whether the participants were shown the audio, the image, and the text in both conditions consistently.","section":"Supplementary, Section G"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible systems contribution to ACM Multimedia, and the prompt-mining/selection pipeline is clearly described. The main risk is evaluative circularity: CLAP is used both to select prompts and to measure alignment, and no independent validation is provided. If the authors can add a non-CLAP evaluation (e.g., human A/B testing with more participants, or an external audio-image alignment model), report uncertainty quantification, and fix the hyperparameter-selection protocol, I would be willing to reconsider. The current version, however, does not yet establish the central claim beyond the circular metric."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here’s my take. The paper is not a breakthrough, but it is a genuinely useful engineering contribution. The idea is straightforward: replace template prompts with LLM/ACM-generated enriched prompts, filter those prompts with CLAP against same-class and other-class audio, retrieve a per-instance prompt, then train the GlueGen-style adapter with an extra InfoNCE loss. That combination is new, and it targets a real failure mode—homographs and what they call auditory illusions—that prior audio-to-image work mostly ignored. The paper does the obvious ablations and they mostly support the design: every added component helps, merge-before-filtering wins, negative-class penalization helps, and the gains are consistent across US8K, ESC50, and VGGSound. Credit where due: this is a clean, reproducible recipe, and the qualitative examples are persuasive.\n\nThe soft spots are real but not fatal. First, no error bars or significance tests anywhere. With AIS differences like .1910 vs .1905, that is a problem for the InfoNCE claim. Second, the filtering hyperparameters (top-K, NAS) are swept in Table 9 and the best values are read off the same AIS metric used for final reporting. That is selection on the test metric. Third, and most important, the circularity concern has teeth. CLAP is the oracle for filtering and retrieval, and the AIS metric—inherited from AudioToken—is itself a cross-modal similarity that plausibly uses the same family of embeddings. The paper never says whether the AIS encoder is the same CLAP checkpoint used for selection. If it is, the reported gains could largely reflect prompt-side similarity rather than genuine image-semantic alignment. The supplementary audio-to-text retrieval only reinforces that concern by confirming selected prompts are CLAP-near. The user study breaks out of the loop, and it favors CatchPhrase, but 30 participants on 15 pairs is a small sample.\n\nThe citation pattern looks honest; the related work is standard and the baselines are the right ones. I would not desk-reject this. It deserves peer review, but a careful referee should push hard on metric provenance, ask for held-out validation for hyperparameters, error bars, and a clearer demonstration—human or external—that the AIS gains reflect true audio-image alignment rather than CLAP-space optimization.","headline":"A sensible prompt-enrichment pipeline with consistent gains on three datasets, but the headline AIS numbers may be inflated by CLAP being both the selector and the metric.","tokens_in":17419,"tokens_out":2766,"would_cite":true,"duration_ms":30243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing fixed class templates with per-audio enriched prompts fixes homograph and auditory-illusion errors in audio-to-image generation, the paper reports.","keywords":["audio-to-image generation","semantic misalignment","prompt enrichment","homograph disambiguation","auditory illusion","encoder adaptation","cross-modal retrieval","diffusion models"],"falsifier":"On a held-out set of audio clips from classes with known homographs and ambiguous instances, ask human raters which candidate prompt best matches each clip; if a substantial fraction of the time raters prefer a prompt that the CLAP-based selector ranked below the top few, then the selection mechanism is not picking the semantically right anchors and the reported alignment gains rest on a faulty oracle.","tokens_in":16444,"feed_emoji":"🎨","tokens_out":7124,"duration_ms":68318,"temperature":0.7,"pith_summary":"CatchPhrase is a training recipe for audio-to-image generation that attacks the semantic poverty of the labels rather than the encoder alignment itself. Its claim is that fixed template prompts such as \"a photo of X\" systematically mislead the generator on homographs (\"crane\" the bird versus \"crane\" the machine) and on sounds whose class label underdescribes the instance (\"engine idling,\" \"drilling\"). The framework mines candidate prompts from an LLM using visual, auditory, and semantic queries plus zero-shot audio captions, then selects the best prompt per audio clip with a similarity-based filter-and-retrieve step. Trained with those selected prompts as anchors, a lightweight adapter improves audio-image similarity over the template-based baseline on US8K, ESC-50, and VGGSound while also lowering FID and KID.","feed_headline":"Enriched prompts push audio-to-image alignment past template baselines","feed_subtitle":"Per-audio prompt selection lifts AIS scores on US8K, ESC-50, and VGGSound over the GlueGen baseline.","key_machinery":"The load-bearing object is the EXPrompt, an enriched cross-modal prompt that fuses text knowledge about a class (from an LLM asked to describe how the class looks, sounds, and what it means) with audio-derived captions produced by a zero-shot audio captioning model. The EXPrompt Selector then applies two score functions in the shared audio-text embedding space: a filtering score (Eq. 4) that rewards prompts similar to same-class audio and penalizes similarity to audio from other classes, keeping top-K prompts per class; and a retrieval score (Eq. 5) that selects the single most similar prompt for each audio clip. The selected prompts supervise a compact mapping network that adapts a frozen audio encoder's features into the text-encoder space of a pretrained text-to-image model, trained with InfoNCE contrastive loss combined with MSE, reconstruction, and adversarial losses.","core_discovery":"The central claim is that the limiting factor in audio-to-image generation is the text prompt derived from a weak class label, not the cross-modal encoder itself. CatchPhrase shows that enriching labels with LLM-generated visual, auditory, and semantic descriptions together with zero-shot audio captions, then filtering those candidates against same-class versus other-class audio and retrieving the top match for each audio instance, yields prompts that resolve both class-level homograph confusion and instance-level auditory illusion. With these prompts as training targets, a lightweight mapping network inserted between a frozen audio encoder and a frozen text encoder produces images with higher Audio-Image Similarity (a score measuring how well the generated image matches the input audio) than GlueGen on US8K (0.191 vs 0.144), ESC-50 (0.242 vs 0.192), and VGGSound (0.202 vs 0.195), and lower FID (65.62 vs 76.66) and KID (0.0119 vs 0.0188) on VGGSound.","pith_inferences":["Because the same CLAP space is used both to select prompts and to compute the reported AIS metric, the gains may partly reflect optimizing the selection toward the evaluation metric; a held-out human preference test or a different embedding model for selection would separate method improvement from metric alignment.","The EXPrompt plus selector recipe transfers in principle to any label-poor generation task where one modality should disambiguate another, such as generating images from animal vocalizations or industrial machinery sounds, since the pipeline only requires weak labels plus an audio-text embedder.","A testable variant would swap the CLAP embedder for a different audio-text model at selection time and measure AIS; if the improvement shrinks, the contribution is tied to the embedding space rather than to prompt enrichment per se."],"forward_implications":["Template-based adapters leave systematic class- and instance-level misalignment; replacing templates with selected enriched prompts raises AIS on all three reported datasets.","Homograph errors such as \"siren\" and auditory-illusion errors such as \"engine idling\" are reduced, and the supplementary retrieval results show EXPrompts are far more likely to be the top-1 text match for an audio clip than template prompts.","Auditory-only or onomatopoeia-heavy prompts degrade generation, so the semantic query from the LLM is doing essential disambiguation work, not decoration.","Only a lightweight mapping network is trained, so the approach plugs into any frozen text-to-image model without retraining the generator."],"supporting_citations":[{"why":"Supplies the template-based adapter and fixed-prompt baseline that CatchPhrase replaces and compares against.","marker":"[35]"},{"why":"Supplies the audio-text embedding space used in the EXPrompt Selector's filtering and retrieval scores.","marker":"[11]"},{"why":"Supplies the zero-shot audio captioning model that generates the audio-derived EXPrompts.","marker":"[17]"},{"why":"Motivates the LLM-description approach for class-level semantic enrichment from weak labels.","marker":"[34]"},{"why":"Supplies the frozen audio encoder whose features the mapping network adapts into text space.","marker":"[13]"},{"why":"Supplies the frozen text encoder and the text-conditioning space used to align audio features with the generator.","marker":"[36]"},{"why":"Supplies the InfoNCE contrastive objective added to the adapter training loss.","marker":"[48]"},{"why":"Provides the pretrained text-to-image latent diffusion model used to generate images at inference.","marker":"[41]"}],"fun_headline_variants":["CatchPhrase: LLM-enriched prompts fix audio-image mismatch","Per-audio prompt selection lifts audio-to-image generation","LLM prompts resolve homograph illusions in audio-to-image","Enriched prompts from LLMs align audio to images better than templates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole filter-and-retrieve pipeline trusts that the audio-text similarity scores used to select prompts reflect true semantic agreement; if those scores are unreliable for these sounds, the chosen prompts will be noisy and the trained adapter will inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["CatchPhrase: LLM-enriched prompts fix audio-image mismatch","Per-audio prompt selection lifts audio-to-image generation","LLM prompts resolve homograph illusions in audio-to-image","Enriched prompts from LLMs align audio to images better than templates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000791,"raw_usage":{"total_tokens":3474,"prompt_tokens":924,"completion_tokens":2550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2479}},"tokens_in":540,"tokens_out":2550,"duration_ms":18932,"temperature":1.0,"reasoning_tokens":2479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:08:50.217579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of audio clips from classes with known homographs and ambiguous instances, ask human raters which candidate prompt best matches each clip; if a substantial fraction of the time raters prefer a prompt that the CLAP-based selector ranked below the top few, then the selection mechanism is not picking the semantically right anchors and the reported alignment gains rest on a faulty oracle.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the template-based adapter and fixed-prompt baseline that CatchPhrase replaces and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot audio captioning model that generates the audio-derived EXPrompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the LLM-description approach for class-level semantic enrichment from weak labels."}],"review_version":2}