{"id":"66da19ba-e2bd-4fba-81d0-4be21047f91a","arxiv_id":"2506.06537","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A training-free pipeline that converts audio into a text query via classification, captioning, or textual inversion and feeds it to a referring image segmentation model achieves state-of-the-art zero-shot audiovisual segmentation on three datasets.","lead":"This paper builds a zero-shot audiovisual segmentation system by chaining pretrained audio, vision, and text models: audio becomes a text query, and a referring image segmentation model draws the mask. It reports large gains over unsupervised sound-localization baselines, with the best variant using an image caption to filter candidate sound classes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The best variant inherits an untested caption-recall bottleneck: if BLIP omits the sounding object's noun, CLAP cannot select it and the segmentation query is wrong.","rationale":"The reader identified the same load-bearing premise: VCap+ACls can only segment a sound source if the image caption mentions that source's noun. My stress-test agrees. The paper's central claim is that a training-free composition of pretrained audio, vision, and text models achieves state-of-the-art zero-shot audiovisual segmentation. For that claim to hold, the textual query passed to the referring-image-segmentation model must denote the true source. The cross-modal verification pipeline adds audio evidence only after the candidate set has been fixed by BLIP and SpaCy, so caption omission is an unrecoverable error. The paper's own warning about aggressive early rejection and error propagation underscores that this is a known failure mode, but the manuscript gives no measurement of its frequency. The proposed oracle-candidate test would settle whether the reported results are robust to this limitation or whether they depend on the favorable, label-constructed nature of the evaluation datasets. Because the reader's conditional verdict already captures this uncertainty, no verdict change is needed; the concern strengthens the argument for requiring code, error bars, and an oracle analysis before unconditional endorsement. I avoid treating the absence of code or the duplicated VPO-SS row in Table 3 as the central issue; those are reproducibility concerns, whereas the caption-recall premise is the step at which the core claim can silently fail.","tokens_in":9132,"tokens_out":5024,"duration_ms":57942,"concrete_test":"Re-run VCap+ACls on IS3, AVSBench-S4, and VPO-SS with an oracle candidate set formed by appending the ground-truth source label (or canonical VGGSound label) to the SpaCy-extracted nouns from BLIP, and compare cIoU/mIoU against Table 2. Also compute BLIP noun recall: the fraction of test samples whose true source noun appears among the extracted nouns. If oracle metrics rise materially (e.g., by more than 5 points) or recall falls below about 90%, then caption omission is the binding constraint and the reported state-of-the-art claim should be scoped to images whose sounding objects are captioned; if metrics are nearly unchanged, the caption-recall premise is not the bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's Cross-modal Verification (VCap+ACls) builds the candidate set for audio classification from noun phrases extracted from a single BLIP caption. The audio branch can only choose among those nouns, so the reported state-of-the-art numbers in Tables 1 and 2 presuppose that, on IS3, AVSBench-S4, and VPO-SS, BLIP captions contain the true sound-source noun and that SpaCy extracts it. This is an empirical claim about caption coverage, not a consequence of the method. The manuscript provides no per-dataset caption recall, no oracle analysis, and no error bars, so the large margins over prior methods cannot separate 'the caption reliably contains the source' from 'the audio-visual verification step itself adds value.' The paper even acknowledges that aggressive early rejection can cause severe error propagation, yet it never measures how often the true noun is omitted from the candidate set. Because this step sits before the referring-image-segmentation model, a miss here makes the downstream query wrong and the segmentation cannot recover. The premise is plausible on VPO-SS and IS3, which are constructed from label-conditioned prompts, but it is untested on less controlled scenes and is the load-bearing assumption for the central zero-shot claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free zero-shot audiovisual segmentation (AVS) framework that combines pretrained audio, vision, and language models. The audio signal is converted into a text query via one of four strategies (audio classification, audio captioning, text inversion, or cross-modal verification), and the resulting text is fed to a referring image segmentation (RIS) model to produce the mask. On three single-source benchmark datasets (IS3, AVSBench-S4, VPO-SS), the cross-modal verification variant (VCap+ACls) is reported to outperform prior unsupervised methods by roughly 2x on most metrics and 3x in cIoU on VPO-SS. The paper also includes ablations of the RIS backbone and a qualitative comparison.","tokens_in":9364,"tokens_out":5097,"duration_ms":47604,"significance":"If the results hold, the paper would demonstrate that competitive zero-shot AVS can be achieved by composing off-the-shelf pretrained models without any AVS-specific training. The systematic comparison of four bridging strategies, the evaluation on standard benchmarks, and the public code release are concrete strengths. The central claim, however, rests on an untested caption-recall assumption in the best-performing variant and on the comparability of the baseline numbers, so the significance at this stage is conditional: the experimental evidence needs reinforcement before the claimed performance can be fully credited.","major_comments":[{"comment":"The VCap+ACls variant constructs its candidate class set from noun phrases extracted from a single BLIP caption and then classifies the audio only among those nouns. Consequently, the reported state-of-the-art numbers presuppose that the true sound-source noun is present in the BLIP caption and is successfully parsed by SpaCy. The paper provides no per-dataset caption-recall statistics, no oracle analysis (e.g., adding the true label to the candidate set), and no ablation separating caption quality from the cross-modal verification step. Because a missed noun makes the downstream RIS query wrong and cannot be recovered, this is a load-bearing empirical assumption rather than a consequence of the method. Please report how often the true source noun appears in the candidate set for each dataset, and include an oracle experiment that injects the ground-truth label to quantify the ceiling of the approach.","section":"Section 3.2, Tables 1 and 2"},{"comment":"The manuscript does not state whether the baseline numbers (LVS, EZ-VSL, SSL-TIE, etc.) were produced by the authors using the same evaluation code, the same test splits, and the same adaptive top-k pixel selection, or are taken from the original papers. Section 4.3 changes the evaluation protocol from the top-50% selection used in earlier work to the adaptive selection from [2], so inconsistent protocols could inflate the reported margin. Please specify the exact protocol for each baseline and, if numbers are reused, verify that they are comparable. Also, since the pipeline includes stochastic components (BLIP caption generation and CLAP classification), please report results over multiple runs or seeds to support the large-margin claims.","section":"Table 2 and Section 4.3"},{"comment":"All three evaluation datasets (IS3, AVSBench-S4, VPO-SS) are single-source subsets. The paper claims zero-shot audiovisual segmentation in general, but the proposed cross-modal filtering is not evaluated on multi-source scenes such as AVSBench-MS or VPO-MS, where the candidate set may contain several visual objects and the audio may be a mixture. Please either include multi-source results or explicitly restrict the claim to single-source AVS.","section":"Section 4.2 and title/abstract"},{"comment":"The algorithmic details needed for reproduction are missing. The cross-modal filtering is described only in prose ('rejecting scores for objects that are not visually present') without specifying the thresholds or decision rule. For Text Inversion, the optimization in Eq. (1) is underspecified: no number of steps, learning rate, initialization, or details on how the estimated token embeddings are injected into ASDA. Please provide precise pseudocode and hyperparameters for both procedures.","section":"Section 3.2, Eq. (1)"}],"minor_comments":[{"comment":"The VPOSS row is numerically identical to the IS3 row, which appears to be a copy-paste error; the corresponding VPO-SS results in Table 1 are different.","section":"Table 3"},{"comment":"The manuscript has inconsistent spacing in 'A VS' and several typographical errors, including 'circumventsthis' (Section 4.4), 'avaliable' (footnote), and 'na¨ıve' (Section 3.2).","section":"Throughout"},{"comment":"The model name 'WavCap' is used inconsistently with the reference to WavCaps [14]; please adopt a single consistent name.","section":"Section 4.1"},{"comment":"The claim that VCap+ACls outperforms Classification by 'over 5.8% relative improvements' on IS3 is inconsistent with Table 1, where the cIoU difference (65.4 vs. 58.3) is roughly 12% relative; please clarify or correct the stated figure.","section":"Section 4.4"},{"comment":"The notation for the text token embeddings ed in Eq. (1) is not defined clearly, and the decoding details for BLIP caption generation (e.g., beam size) are not given; please add short definitions and implementation notes.","section":"References and notation"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the paper makes a useful empirical contribution, but the evaluation lacks caption-recall analysis, error bars, and a clear baseline-comparison protocol. The identical numbers between the IS3 and VPOSS rows in Table 3 should be corrected. The authors should be asked to supply the reproducibility details and the multi-source caveat before the claimed 2x/3x margins are taken at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper shows that you can get strong zero-shot audiovisual segmentation by chaining an image captioner, a noun extractor, an audio classifier, and a referring segmenter—no AVS training at all. On IS3, AVSBench-S4, and VPO-SS the best variant beats the previous unsupervised methods by a wide margin. That is real and useful. Second, the best variant works only if the image caption contains the noun for the sounding object, because CLAP is asked to classify the audio among nouns extracted from that single caption. The paper never measures how often BLIP actually surfaces that noun. That is the load-bearing assumption and it is untested.\n\nWhat is new: the cross-modal verification idea—using a caption to constrain the audio classifier's candidate set—is genuinely neat and I haven't seen it in AVS. The paper also deserves credit for systematically comparing five bridging strategies and for reporting the failure mode of textual inversion honestly. The RIS ablation is a nice sanity check.\n\nSoft spots, in order of severity. (1) No caption-recall statistics, no oracle experiment, no error bars. The 2x/3x margins cannot be separated into 'the caption usually contains the right noun' versus 'the verification step adds value.' The paper itself warns that early rejection causes error propagation, so this is not a nitpick. (2) The baseline numbers may come from different evaluation protocols. The metrics section says prior work used top-50% pixels while the paper uses adaptive top-k selection, but Table 2 does not state whether the prior scores were recomputed under the same protocol. That needs to be explicit. (3) Table 3 appears to have a copy-paste error: the IS3 and VPO-SS rows for ASDA are identical. That doesn't undermine the conclusions but must be fixed. (4) Evaluation on single-source subsets only—minor, given the benchmarks, but the multi-source claim is not supported.\n\nWho is this for: anyone working on zero-shot AVS or model composition for multimodal grounding. It gives a strong baseline and a clear design space.\n\nRecommendation: the paper deserves a serious referee. The core idea is sound and the experiments are suggestive, but the caption-recall bottleneck and the evaluation protocol need to be addressed before the SOTA claim is credible. I would send it to review, expecting major revision.","headline":"A clean, training-free AVS pipeline that gets 2x over self-supervised baselines, but the top variant's reliance on BLIP caption coverage is unmeasured and the baseline protocol is under-specified.","tokens_in":9825,"tokens_out":3514,"would_cite":true,"duration_ms":35086,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that zero-shot audiovisual segmentation can be achieved by converting audio into a text query and feeding it to a referring image segmentation model, with a cross-modal verification variant that outperforms prior…","keywords":["audiovisual segmentation","zero-shot learning","referring image segmentation","cross-modal verification","pretrained models","audio captioning","sound source localization"],"falsifier":"Evaluate VCap+ACls on a set of audiovisual samples where BLIP's caption omits the object that produces the audio (verifiable by checking ground-truth labels against the caption's nouns); the method should collapse specifically on those samples, and adding the missing noun manually should restore much of the performance. A second check is to compare against a variant that replaces CLAP with an open-vocabulary classifier over all AudioSet classes rather than only the caption nouns.","tokens_in":8957,"feed_emoji":"🎧","tokens_out":4610,"duration_ms":44196,"temperature":0.7,"pith_summary":"The paper tries to establish that audiovisual segmentation—finding the pixels corresponding to whatever is making noise in a video—can be done without any task-specific training, by stitching together off-the-shelf pretrained audio, vision, and language models. Its pipeline turns the audio into a short text description and hands that text to a referring image segmentation model, which draws the mask. The best variant, cross-modal verification, first asks an image captioning model what objects are in the frame, then uses an audio-text model to pick which of those objects is actually making the sound. On three benchmarks this training-free method reports roughly double the segmentation scores of prior unsupervised methods, and about triple the cIoU on VPO-SS. If this holds, costly pixel-level audiovisual annotations are not needed to bootstrap sound-source segmentation.","feed_headline":"Sound-to-text pipeline triples zero-shot segmentation scores","feed_subtitle":"Converting audio to a text query for a referring-image segmenter beats trained baselines without task training.","key_machinery":"The central mechanism is the text modality used as a bridge: audio and vision are each mapped into natural language by separate pretrained models, and a referring image segmentation (RIS) model then turns a phrase into a mask. In the best variant, cross-modal verification constrains audio classification to noun phrases that were visible in the image, so the sound source is chosen only from objects actually present. The load-bearing query chain is BLIP captioning, then SpaCy noun parsing, then CLAP audio classification, with ASDA performing the final segmentation.","core_discovery":"On its own terms, the central claim is that a training-free composition of pretrained models achieves state-of-the-art zero-shot audiovisual segmentation. Specifically, the variant called VCap+ACls—where BLIP generates a caption of the image, SpaCy extracts noun phrases, CLAP classifies the audio among those candidate classes, and ASDA segments the region described by the winning query such as 'a photo of guitar'—outperforms both audio-only variants and prior self-supervised methods across IS3, AVSBench-S4, and VPO-SS, with roughly 2x improvement on most metrics and 3x improvement in cIoU on VPO-SS. The paper also claims that the choice of referring segmentation model matters: ASDA beats ETRIS and CRIS in every setting.","pith_inferences":["If caption recall is the bottleneck, replacing BLIP with a stronger open-vocabulary detector or ensembling several captions could push the method further, a direction the paper does not explore.","The dependence on visible objects suggests a hidden systematic failure on occluded or visually subtle sound sources, even though overall benchmark numbers are high.","Because the method is training-free, it can absorb improvements in any component model, so its edge over trained baselines may grow as component models improve.","The text bridge could be inverted: use audio captions to disambiguate referring expressions in embodied or surveillance settings where multiple objects make similar sounds."],"forward_implications":["Audiovisual segmentation can be built without task-specific training or pixel annotations, by composing pretrained audio, vision, and language models.","The zero-shot pipeline inherits the segmentation quality of the referring image segmentation model, so improving RIS directly improves audiovisual segmentation.","Cross-modal verification is the key design choice: using visual context to restrict candidate sound classes gives consistent gains over audio-only query construction.","The same text-bridge recipe can be adapted to other multimodal grounding tasks that currently rely on supervised audiovisual data."],"supporting_citations":[{"why":"CLAP supplies the shared audio-text embedding space used both for open-vocabulary audio classification in cross-modal verification and for the text-inversion baseline.","marker":"[13]"},{"why":"BLIP generates the image caption whose noun phrases become the candidate sound-source classes in the best-performing VCap+ACls variant.","marker":"[17]"},{"why":"ASDA is the referring image segmentation model that turns the final text query into the pixel-level mask.","marker":"[20]"},{"why":"BEATs provides the audio classifier for the audio-only classification baseline.","marker":"[21]"},{"why":"WavCaps provides the audio captioning model used in the captioning baseline and in the reverse ACap+VCls verification variant.","marker":"[14]"},{"why":"SpaCy performs the noun phrase parsing that turns the BLIP caption into the candidate class list for CLAP.","marker":"[24]"},{"why":"CLIP's text encoder is the shared text representation used by the RIS model and by CLAP, and CLIP also plays a role in the reverse verification process.","marker":"[12]"}],"fun_headline_variants":["Zero-shot AVS without training, just pretrained parts","Audio-to-text query triples zero-shot AVS scores","Pretrained models bridge audio and vision for zero-shot AVS","Training-free model composition achieves state-of-the-art AVS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline is only as strong as the image caption's recall: if BLIP does not name the true sound source in its caption, that class is never among CLAP's candidates, so no correct query can be formed and the segmentation fails.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot AVS without training, just pretrained parts","Audio-to-text query triples zero-shot AVS scores","Pretrained models bridge audio and vision for zero-shot AVS","Training-free model composition achieves state-of-the-art AVS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00124,"raw_usage":{"total_tokens":5035,"prompt_tokens":834,"completion_tokens":4201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":4133}},"tokens_in":450,"tokens_out":4201,"duration_ms":32790,"temperature":1.0,"reasoning_tokens":4133,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:53:56.199007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate VCap+ACls on a set of audiovisual samples where BLIP's caption omits the object that produces the audio (verifiable by checking ground-truth labels against the caption's nouns); the method should collapse specifically on those samples, and adding the missing noun manually should restore much of the performance. A second check is to compare against a variant that replaces CLAP with an open-vocabulary classifier over all AudioSet classes rather than only the caption nouns.","supporting_citations":[{"cited_title":"Marginnce: Robust sound localization with a negative margin,","cited_arxiv_id":null,"evidence_quote":"CLAP supplies the shared audio-text embedding space used both for open-vocabulary audio classification in cross-modal verification and for the text-inversion baseline."},{"cited_title":"Bavs: Bootstrapping audio-visual segmentation by integrating foundation knowledge,","cited_arxiv_id":null,"evidence_quote":"BLIP generates the image caption whose noun phrases become the candidate sound-source classes in the best-performing VCap+ACls variant."},{"cited_title":"WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language mul- timodal research,","cited_arxiv_id":null,"evidence_quote":"ASDA is the referring image segmentation model that turns the final text query into the pixel-level mask."},{"cited_title":"AudioCaps: Generating Captions for Audios in The Wild,","cited_arxiv_id":null,"evidence_quote":"BEATs provides the audio classifier for the audio-only classification baseline."},{"cited_title":"Audio–visual segmen- tation,","cited_arxiv_id":null,"evidence_quote":"WavCaps provides the audio captioning model used in the captioning baseline and in the reverse ACap+VCls verification variant."},{"cited_title":"Learning audio-visual source local- ization via false negative aware contrastive learning,","cited_arxiv_id":null,"evidence_quote":"CLIP's text encoder is the shared text representation used by the RIS model and by CLAP, and CLIP also plays a role in the reverse verification process."}],"review_version":1}