Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models

Paul Hongsuck Seo; Seung-Jae Lee

REVIEW 4 major objections 5 minor 37 references

Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that zero-shot audiovisual segmentation can be achieved by converting audio into a text query and feeding it to a referring image segmentation model, with a cross-modal verification variant that outperforms prior…

desk verdict A clean, training-free AVS pipeline that gets 2x over self-supervised baselines, but the top variant's reliance on BLIP caption coverage is unmeasured and the baseline protocol is under-specified. read the letter →

arxiv 2506.06537 v1 pith:SX5HNSSX submitted 2025-06-06 cs.CV cs.SDeess.AS

Seung-jae Lee , Paul Hongsuck Seo This is my paper

classification cs.CVcs.SDeess.AS

keywords audiovisualsegmentationzero-shotlearningreferringimagecross-modalverificationpretrainedmodelsaudiocaptioningsoundsourcelocalization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that audiovisual segmentation—finding the pixels corresponding to whatever is making noise in a video—can be done without any task-specific training, by stitching together off-the-shelf pretrained audio, vision, and language models. Its pipeline turns the audio into a short text description and hands that text to a referring image segmentation model, which draws the mask. The best variant, cross-modal verification, first asks an image captioning model what objects are in the frame, then uses an audio-text model to pick which of those objects is actually making the sound. On three benchmarks this training-free method reports roughly double the segmentation scores of prior unsupervised methods, and about triple the cIoU on VPO-SS. If this holds, costly pixel-level audiovisual annotations are not needed to bootstrap sound-source segmentation.

What carries the argument

The central mechanism is the text modality used as a bridge: audio and vision are each mapped into natural language by separate pretrained models, and a referring image segmentation (RIS) model then turns a phrase into a mask. In the best variant, cross-modal verification constrains audio classification to noun phrases that were visible in the image, so the sound source is chosen only from objects actually present. The load-bearing query chain is BLIP captioning, then SpaCy noun parsing, then CLAP audio classification, with ASDA performing the final segmentation.

What would settle it

Evaluate VCap+ACls on a set of audiovisual samples where BLIP's caption omits the object that produces the audio (verifiable by checking ground-truth labels against the caption's nouns); the method should collapse specifically on those samples, and adding the missing noun manually should restore much of the performance. A second check is to compare against a variant that replaces CLAP with an open-vocabulary classifier over all AudioSet classes rather than only the caption nouns.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that a training-free composition of pretrained models achieves state-of-the-art zero-shot audiovisual segmentation. Specifically, the variant called VCap+ACls—where BLIP generates a caption of the image, SpaCy extracts noun phrases, CLAP classifies the audio among those candidate classes, and ASDA segments the region described by the winning query such as 'a photo of guitar'—outperforms both audio-only variants and prior self-supervised methods across IS3, AVSBench-S4, and VPO-SS, with roughly 2x improvement on most metrics and 3x improvement in cIoU on VPO-SS. The paper also claims that the choice of referring segmentation model matters: ASDA beats ETRIS and CRIS in every setting.

Load-bearing premise

The pipeline is only as strong as the image caption's recall: if BLIP does not name the true sound source in its caption, that class is never among CLAP's candidates, so no correct query can be formed and the segmentation fails.

Editorial extensions

If this is right

Audiovisual segmentation can be built without task-specific training or pixel annotations, by composing pretrained audio, vision, and language models.
The zero-shot pipeline inherits the segmentation quality of the referring image segmentation model, so improving RIS directly improves audiovisual segmentation.
Cross-modal verification is the key design choice: using visual context to restrict candidate sound classes gives consistent gains over audio-only query construction.
The same text-bridge recipe can be adapted to other multimodal grounding tasks that currently rely on supervised audiovisual data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If caption recall is the bottleneck, replacing BLIP with a stronger open-vocabulary detector or ensembling several captions could push the method further, a direction the paper does not explore.
The dependence on visible objects suggests a hidden systematic failure on occluded or visually subtle sound sources, even though overall benchmark numbers are high.
Because the method is training-free, it can absorb improvements in any component model, so its edge over trained baselines may grow as component models improve.
The text bridge could be inverted: use audio captions to disambiguate referring expressions in embodied or surveillance settings where multiple objects make similar sounds.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the paper shows that you can get strong zero-shot audiovisual segmentation by chaining an image captioner, a noun extractor, an audio classifier, and a referring segmenter—no AVS training at all. On IS3, AVSBench-S4, and VPO-SS the best variant beats the previous unsupervised methods by a wide margin. That is real and useful. Second, the best variant works only if the image caption contains the noun for the sounding object, because CLAP is asked to classify the audio among nouns extracted from that single caption. The paper never measures how often BLIP actually surfaces that noun. That is the load-bearing assumption and it is untested.

What is new: the cross-modal verification idea—using a caption to constrain the audio classifier's candidate set—is genuinely neat and I haven't seen it in AVS. The paper also deserves credit for systematically comparing five bridging strategies and for reporting the failure mode of textual inversion honestly. The RIS ablation is a nice sanity check.

Soft spots, in order of severity. (1) No caption-recall statistics, no oracle experiment, no error bars. The 2x/3x margins cannot be separated into 'the caption usually contains the right noun' versus 'the verification step adds value.' The paper itself warns that early rejection causes error propagation, so this is not a nitpick. (2) The baseline numbers may come from different evaluation protocols. The metrics section says prior work used top-50% pixels while the paper uses adaptive top-k selection, but Table 2 does not state whether the prior scores were recomputed under the same protocol. That needs to be explicit. (3) Table 3 appears to have a copy-paste error: the IS3 and VPO-SS rows for ASDA are identical. That doesn't undermine the conclusions but must be fixed. (4) Evaluation on single-source subsets only—minor, given the benchmarks, but the multi-source claim is not supported.

Who is this for: anyone working on zero-shot AVS or model composition for multimodal grounding. It gives a strong baseline and a clear design space.

Recommendation: the paper deserves a serious referee. The core idea is sound and the experiments are suggestive, but the caption-recall bottleneck and the evaluation protocol need to be addressed before the SOTA claim is credible. I would send it to review, expecting major revision.

Referee Report

4 major / 5 minor

Summary. The paper proposes a training-free zero-shot audiovisual segmentation (AVS) framework that combines pretrained audio, vision, and language models. The audio signal is converted into a text query via one of four strategies (audio classification, audio captioning, text inversion, or cross-modal verification), and the resulting text is fed to a referring image segmentation (RIS) model to produce the mask. On three single-source benchmark datasets (IS3, AVSBench-S4, VPO-SS), the cross-modal verification variant (VCap+ACls) is reported to outperform prior unsupervised methods by roughly 2x on most metrics and 3x in cIoU on VPO-SS. The paper also includes ablations of the RIS backbone and a qualitative comparison.

Significance. If the results hold, the paper would demonstrate that competitive zero-shot AVS can be achieved by composing off-the-shelf pretrained models without any AVS-specific training. The systematic comparison of four bridging strategies, the evaluation on standard benchmarks, and the public code release are concrete strengths. The central claim, however, rests on an untested caption-recall assumption in the best-performing variant and on the comparability of the baseline numbers, so the significance at this stage is conditional: the experimental evidence needs reinforcement before the claimed performance can be fully credited.

major comments (4)

[Section 3.2, Tables 1 and 2] The VCap+ACls variant constructs its candidate class set from noun phrases extracted from a single BLIP caption and then classifies the audio only among those nouns. Consequently, the reported state-of-the-art numbers presuppose that the true sound-source noun is present in the BLIP caption and is successfully parsed by SpaCy. The paper provides no per-dataset caption-recall statistics, no oracle analysis (e.g., adding the true label to the candidate set), and no ablation separating caption quality from the cross-modal verification step. Because a missed noun makes the downstream RIS query wrong and cannot be recovered, this is a load-bearing empirical assumption rather than a consequence of the method. Please report how often the true source noun appears in the candidate set for each dataset, and include an oracle experiment that injects the ground-truth label to quantify the ceiling of the approach.
[Table 2 and Section 4.3] The manuscript does not state whether the baseline numbers (LVS, EZ-VSL, SSL-TIE, etc.) were produced by the authors using the same evaluation code, the same test splits, and the same adaptive top-k pixel selection, or are taken from the original papers. Section 4.3 changes the evaluation protocol from the top-50% selection used in earlier work to the adaptive selection from [2], so inconsistent protocols could inflate the reported margin. Please specify the exact protocol for each baseline and, if numbers are reused, verify that they are comparable. Also, since the pipeline includes stochastic components (BLIP caption generation and CLAP classification), please report results over multiple runs or seeds to support the large-margin claims.
[Section 4.2 and title/abstract] All three evaluation datasets (IS3, AVSBench-S4, VPO-SS) are single-source subsets. The paper claims zero-shot audiovisual segmentation in general, but the proposed cross-modal filtering is not evaluated on multi-source scenes such as AVSBench-MS or VPO-MS, where the candidate set may contain several visual objects and the audio may be a mixture. Please either include multi-source results or explicitly restrict the claim to single-source AVS.
[Section 3.2, Eq. (1)] The algorithmic details needed for reproduction are missing. The cross-modal filtering is described only in prose ('rejecting scores for objects that are not visually present') without specifying the thresholds or decision rule. For Text Inversion, the optimization in Eq. (1) is underspecified: no number of steps, learning rate, initialization, or details on how the estimated token embeddings are injected into ASDA. Please provide precise pseudocode and hyperparameters for both procedures.

minor comments (5)

[Table 3] The VPOSS row is numerically identical to the IS3 row, which appears to be a copy-paste error; the corresponding VPO-SS results in Table 1 are different.
[Throughout] The manuscript has inconsistent spacing in 'A VS' and several typographical errors, including 'circumventsthis' (Section 4.4), 'avaliable' (footnote), and 'na¨ıve' (Section 3.2).
[Section 4.1] The model name 'WavCap' is used inconsistently with the reference to WavCaps [14]; please adopt a single consistent name.
[Section 4.4] The claim that VCap+ACls outperforms Classification by 'over 5.8% relative improvements' on IS3 is inconsistent with Table 1, where the cIoU difference (65.4 vs. 58.3) is roughly 12% relative; please clarify or correct the stated figure.
[References and notation] The notation for the text token embeddings ed in Eq. (1) is not defined clearly, and the decoding details for BLIP caption generation (e.g., beam size) are not given; please add short definitions and implementation notes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the zero-shot pipeline is composed of external pretrained models and evaluated against external baselines; the caption-recall bottleneck is an empirical robustness concern, not a derivation that reduces to its inputs.

full rationale

The paper's derivation chain is a training-free composition of external pretrained models (BEATs, WavCap, CLAP, BLIP, ASDA) with no AVS-specific training and no parameter fitting to the target benchmarks. The segmentation query is produced from the audio and image inputs through explicit pipelines (audio classification, captioning, textual inversion, or cross-modal verification), and the output mask comes from a fixed referring image segmentation model. No equation in the paper defines the prediction in terms of the ground-truth mask or the evaluation labels; formula (1) is a standard textual-inversion optimization, not a fitted-input-renamed-as-prediction. The acknowledged risk that aggressive early rejection in cross-modal verification causes error propagation is a genuine limitation, but it is an empirical assumption about BLIP caption recall and CLAP candidate coverage, not a circular step. The comparisons in Tables 1 and 2 are against external unsupervised baselines, and the method uses only test-set inputs without training. The only potential self-citation (reference [19]) is used as general motivation for using visual context and is not load-bearing for the central claim. The paper is self-contained as an empirical evaluation, so no circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to AVS data; the framework has no learned parameters and relies instead on pretrained models with hand-chosen prompts. The load-bearing assumptions are empirical generalizations about those pretrained models and about the comparability of the zero-shot evaluation. No new entities are introduced.

assumptions (4)

domain assumption The text modality is a shared semantic space: audio-text alignment from CLAP and vision-text alignment from BLIP/CLIP transfer to the text encoder of the RIS model (ASDA).
Section 3.2 states 'text modality as a shared intermediary'; the whole pipeline converts audio into text that ASDA can ground. No evidence in the paper directly verifies this transfer for arbitrary test audio.
domain assumption BLIP image captions list the objects in the scene that could be sound sources.
Cross-modal verification filters audio classes to noun phrases from BLIP caption (Section 3.2); low noun recall removes the true source.
domain assumption Pretrained audio classifiers and captioners (BEATs, WavCap, CLAP) recognize or describe the sound-source classes present in the test datasets.
Section 4.1; if an audio event lies outside their vocabularies, the text query is wrong.
domain assumption Evaluation protocols applied to baselines are comparable to those applied to the proposed method.
Section 4.3 defines metrics with GT-size adaptive pixel selection; Table 2 numbers are taken as fair cross-method comparisons, but the paper does not state whether baselines were re-run under the same protocol.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models." pith.science (2026). https://pith.science/paper/SX5HNSSX

@misc{pith2026250606537,
  author       = {Pith},
  title        = {Pith review of: Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SX5HNSSX}},
  note         = {Machine review of arXiv:2506.06537}
}

read the original abstract

Audiovisual segmentation (AVS) aims to identify visual regions corresponding to sound sources, playing a vital role in video understanding, surveillance, and human-computer interaction. Traditional AVS methods depend on large-scale pixel-level annotations, which are costly and time-consuming to obtain. To address this, we propose a novel zero-shot AVS framework that eliminates task-specific training by leveraging multiple pretrained models. Our approach integrates audio, vision, and text representations to bridge modality gaps, enabling precise sound source segmentation without AVS-specific annotations. We systematically explore different strategies for connecting pretrained models and evaluate their efficacy across multiple datasets. Experimental results demonstrate that our framework achieves state-of-the-art zero-shot AVS performance, highlighting the effectiveness of multimodal model integration for finegrained audiovisual segmentation.

Figures

Figures reproduced from arXiv: 2506.06537 by the authors.

**Figure 2.** Qualitative comparisons of segmentation results across datasets. Our method produces fine-grained masks for sounding objects, outperforming prior methods in visual accuracy and boundary alignment [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 32 canonical work pages

[2]

Many approaches leverage cross-modal attention [1, 2, 3] combined with contrastive learning [4, 5] to align audio and visual features

Related Works Sound Source Localization (SSL) is a fundamental task in au- diovisual learning that aims to localize sound-emitting objects within visual scenes. Many approaches leverage cross-modal attention [1, 2, 3] combined with contrastive learning [4, 5] to align audio and visual features. To improve robustness, various strategies such as noise contr...

work page
[1]

This capability is essential for applica- tions in video understanding, surveillance, human-computer in- teraction, and autonomous systems

Introduction Audiovisual segmentation (A VS) is a fundamental task in mul- timodal learning, that aims to localize visual regions that cor- respond to sound sources through the alignment and fusion of audio and visual cues. This capability is essential for applica- tions in video understanding, surveillance, human-computer in- teraction, and autonomous sy...

work page
[3]

Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models

Proposed Method 3.1. Audiovisual Segmentation Audiovisual Segmentation (A VS) aims to identify and segment regions in a visual frame that correspond to sound-emitting ob- jects, by jointly analyzing audio and visual modalities. Given a visual frame with synchronized audio, the goal is to gener- ate a binary segmentation mask highlighting the spatial regio...

work page Pith review arXiv 2025
[4]

Models Below, we elaborate on the final models built for each approach

Experiments 4.1. Models Below, we elaborate on the final models built for each approach. For all models, we use the same RIS model, ASDA [20]. Audio Classification For this approach, we use BEATs [21], a classifier pretrained on AudioSet [22]. Audio Captioning We use WavCap [14] pretrained on the Audiocaps [15], Clotho [23], and WavCaps dataset [14]. Text...

work page
[5]

By systematically integrating audio, vision, and text representations, we bridge modality gaps and achieve fine-grained segmentation without requiring A VS-specific annotations

Conclusion Our study introduces a novel zero-shot audiovisual segmenta- tion framework that obviates the need for task-specific training by leveraging multiple pretrained models. By systematically integrating audio, vision, and text representations, we bridge modality gaps and achieve fine-grained segmentation without requiring A VS-specific annotations. ...

work page
[6]

Acknowledgements This research was supported by IITP grants (IITP-2025- RS-2020-II201819, IITP-2025-RS-2024-00436857, IITP- 2025-RS-2024-00398115, IITP-2025-RS-2025-02263754, IITP-2025-RS-2025-02304828) and the KOCCA grant (RS- 2024-00345025) funded by the Korea government (MSIT, MOE and MSCT)

work page 2025
[7]

Learning to localize sound sources in visual scenes: Analysis and applications,

A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. S. Kweon, “Learning to localize sound sources in visual scenes: Analysis and applications,” IEEE transactions on pattern analysis and ma- chine intelligence, vol. 43, no. 5, pp. 1605–1619, 2019

work page 2019
[8]

Aligning sight and sound: Advanced sound source localization through audio-visual alignment,

A. Senocak, H. Ryu, J. Kim, T.-H. Oh, H. Pfister, and J. S. Chung, “Aligning sight and sound: Advanced sound source localization through audio-visual alignment,” arXiv preprint arXiv:2407.13676, 2024

arXiv 2024

Show all 37 references

[9]

Exploiting transformation in- variance and equivariance for self-supervised sound localisation,

J. Liu, C. Ju, W. Xie, and Y . Zhang, “Exploiting transformation in- variance and equivariance for self-supervised sound localisation,” in Proceedings of the 30th ACM International Conference on Mul- timedia, 2022, pp. 3742–3753

2022
[10]

Localizing visual sounds the easy way,

S. Mo and P. Morgado, “Localizing visual sounds the easy way,” in European Conference on Computer Vision . Springer, 2022, pp. 218–234

2022
[11]

Localizing visual sounds the hard way,

H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, and A. Zis- serman, “Localizing visual sounds the hard way,” in Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 16 867–16 876

2021
[12]

Learning audio-visual source local- ization via false negative aware contrastive learning,

W. Sun, J. Zhang, J. Wang, Z. Liu, Y . Zhong, T. Feng, Y . Guo, Y . Zhang, and N. Barnes, “Learning audio-visual source local- ization via false negative aware contrastive learning,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2023,...

2023
[13]

Marginnce: Robust sound localization with a negative margin,

S. Park, A. Senocak, and J. S. Chung, “Marginnce: Robust sound localization with a negative margin,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2023, pp. 1–5

2023
[14]

Audio–visual segmen- tation,

J. Zhou, J. Wang, J. Zhang, W. Sun, J. Zhang, S. Birchfield, D. Guo, L. Kong, M. Wang, and Y . Zhong, “Audio–visual segmen- tation,” in European Conference on Computer Vision. Springer, 2022, pp. 386–403

2022
[15]

Improving audio-visual segmentation with bidirectional generation,

D. Hao, Y . Mao, B. He, X. Han, Y . Dai, and Y . Zhong, “Improving audio-visual segmentation with bidirectional generation,” in Pro- ceedings of the AAAI conference on artificial intelligence, vol. 38, no. 3, 2024, pp. 2067–2075

2024
[16]

Selm: Selective mechanism based audio-visual segmentation,

J. Li, S. Yu, Y . Wang, L. Wang, and H. Lu, “Selm: Selective mechanism based audio-visual segmentation,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 3926–3935

2024
[17]

Bavs: Bootstrapping audio-visual segmentation by integrating foundation knowledge,

C. Liu, P. Li, H. Zhang, L. Li, Z. Huang, D. Wang, and X. Yu, “Bavs: Bootstrapping audio-visual segmentation by integrating foundation knowledge,” IEEE Transactions on Multimedia, 2024

2024
[18]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763

2021
[19]

Natural language supervision for general-purpose audio representations,

B. Elizalde, S. Deshmukh, and H. Wang, “Natural language supervision for general-purpose audio representations,” 2023. [Online]. Available: https://arxiv.org/abs/2309.05767

2023 arXiv
[20]

WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language mul- timodal research,

X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y . Zou, and W. Wang, “WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language mul- timodal research,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1–15, 2024

2024
[21]

AudioCaps: Generating Captions for Audios in The Wild,

C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating Captions for Audios in The Wild,” in NAACL-HLT, 2019

2019
[22]

Microsoft coco: Common objects in context,

T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer,...

2014
[23]

Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation,

J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation,” in ICML, 2022

2022
[24]

An image is worth one word: Personalizing text-to-image generation using textual inversion,

R. Gal, Y . Alaluf, Y . Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or, “An image is worth one word: Personalizing text-to-image generation using textual inversion,” arXiv preprint arXiv:2208.01618, 2022

2022 arXiv
[25]

Learning to visually localize sound sources from mixtures without prior source knowl- edge,

D. Kim, S. J. Um, S. Lee, and J. U. Kim, “Learning to visually localize sound sources from mixtures without prior source knowl- edge,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 467–26 476

2024
[26]

Adaptive selection based referring im- age segmentation,

P. Yue, J. Lin, S. Zhang, J. Hu, Y . Lu, H. Niu, H. Ding, Y . Zhang, G. JIANG, L. Cao et al., “Adaptive selection based referring im- age segmentation,” in ACM Multimedia 2024

2024
[27]

Beats: Audio pre-training with acoustic tokenizers,

S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,” 2022

2022
[28]

Audio set: An ontology and human-labeled dataset for audio events,

J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE inter- national conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017,...

2017
[29]

Clotho: An audio cap- tioning dataset,

K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” in ICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 736–740

2020
[30]

spaCy 2: Natural language under- standing with Bloom embeddings, convolutional neural networks and incremental parsing,

M. Honnibal and I. Montani, “spaCy 2: Natural language under- standing with Bloom embeddings, convolutional neural networks and incremental parsing,” 2017, to appear

2017
[31]

High-resolution image synthesis with latent diffusion models,

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” 2021

2021
[32]

Vggsound: A large-scale audio-visual dataset,

H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020

2020
[33]

Segment anything,

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll´ar, and R. Gir- shick, “Segment anything,” 2023

2023
[34]

Unraveling instance associations: A closer look for audio-visual segmentation,

Y . Chen, Y . Liu, H. Wang, F. Liu, C. Wang, H. Frazer, and G. Carneiro, “Unraveling instance associations: A closer look for audio-visual segmentation,” 2024

2024
[35]

A closer look at weakly-supervised audio-visual source localization,

S. Mo and P. Morgado, “A closer look at weakly-supervised audio-visual source localization,” Advances in Neural Informa- tion Processing Systems, vol. 35, pp. 37 524–37 536, 2022

2022
[36]

Bridg- ing vision and language encoders: Parameter-efficient tuning for referring image segmentation,

Z. Xu, Z. Chen, Y . Zhang, Y . Song, X. Wan, and G. Li, “Bridg- ing vision and language encoders: Parameter-efficient tuning for referring image segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 17 503– 17 512

2023
[37]

Cris: Clip-driven referring image segmentation,

Z. Wang, Y . Lu, Q. Li, X. Tao, Y . Guo, M. Gong, and T. Liu, “Cris: Clip-driven referring image segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2022

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2]

Many approaches leverage cross-modal attention [1, 2, 3] combined with contrastive learning [4, 5] to align audio and visual features

Related Works Sound Source Localization (SSL) is a fundamental task in au- diovisual learning that aims to localize sound-emitting objects within visual scenes. Many approaches leverage cross-modal attention [1, 2, 3] combined with contrastive learning [4, 5] to align audio and visual features. To improve robustness, various strategies such as noise contr...

work page

[2] [1]

This capability is essential for applica- tions in video understanding, surveillance, human-computer in- teraction, and autonomous systems

Introduction Audiovisual segmentation (A VS) is a fundamental task in mul- timodal learning, that aims to localize visual regions that cor- respond to sound sources through the alignment and fusion of audio and visual cues. This capability is essential for applica- tions in video understanding, surveillance, human-computer in- teraction, and autonomous sy...

work page

[3] [3]

Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models

Proposed Method 3.1. Audiovisual Segmentation Audiovisual Segmentation (A VS) aims to identify and segment regions in a visual frame that correspond to sound-emitting ob- jects, by jointly analyzing audio and visual modalities. Given a visual frame with synchronized audio, the goal is to gener- ate a binary segmentation mask highlighting the spatial regio...

work page Pith review arXiv 2025

[4] [4]

Models Below, we elaborate on the final models built for each approach

Experiments 4.1. Models Below, we elaborate on the final models built for each approach. For all models, we use the same RIS model, ASDA [20]. Audio Classification For this approach, we use BEATs [21], a classifier pretrained on AudioSet [22]. Audio Captioning We use WavCap [14] pretrained on the Audiocaps [15], Clotho [23], and WavCaps dataset [14]. Text...

work page

[5] [5]

By systematically integrating audio, vision, and text representations, we bridge modality gaps and achieve fine-grained segmentation without requiring A VS-specific annotations

Conclusion Our study introduces a novel zero-shot audiovisual segmenta- tion framework that obviates the need for task-specific training by leveraging multiple pretrained models. By systematically integrating audio, vision, and text representations, we bridge modality gaps and achieve fine-grained segmentation without requiring A VS-specific annotations. ...

work page

[6] [6]

Acknowledgements This research was supported by IITP grants (IITP-2025- RS-2020-II201819, IITP-2025-RS-2024-00436857, IITP- 2025-RS-2024-00398115, IITP-2025-RS-2025-02263754, IITP-2025-RS-2025-02304828) and the KOCCA grant (RS- 2024-00345025) funded by the Korea government (MSIT, MOE and MSCT)

work page 2025

[7] [7]

Learning to localize sound sources in visual scenes: Analysis and applications,

A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. S. Kweon, “Learning to localize sound sources in visual scenes: Analysis and applications,” IEEE transactions on pattern analysis and ma- chine intelligence, vol. 43, no. 5, pp. 1605–1619, 2019

work page 2019

[8] [8]

Aligning sight and sound: Advanced sound source localization through audio-visual alignment,

A. Senocak, H. Ryu, J. Kim, T.-H. Oh, H. Pfister, and J. S. Chung, “Aligning sight and sound: Advanced sound source localization through audio-visual alignment,” arXiv preprint arXiv:2407.13676, 2024

arXiv 2024