Pith. sign in

REVIEW 4 major objections 6 minor 52 references

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that sound-prompted segmentation can be done without any training, by co-factorizing frozen audio and image features with semantic word anchors, and that this training-free approach surpasses existing unsupervised…

desk verdict Strong training-free audio-visual segmentation results, but the algorithm description contradicts the claimed constraints and metric comparability needs tightening. read the letter →

arxiv 2412.01488 v3 pith:DEV5SXLH submitted 2024-12-02 eess.AS cs.LGeess.IV

Hugo Malard , Michel Olvera , Stephane Lathuiliere , Slim Essid This is my paper · ORCID
classification eess.AScs.LGeess.IV
keywords sound-promptedsegmentationaudio-visualnon-negativematrixfactorizationtraining-freezero-shotCLIPCLAPopen-vocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether sound-prompted segmentation — locating the image regions that produce a heard sound — can be solved without any training, using only frozen pre-trained audio and image encoders. It answers yes: by co-factorizing the audio and image feature matrices with non-negative matrix factorization and steering the factorization with a small bank of semantic word anchors, the method surfaces a shared "sounding" factor, which is then fed to an open-vocabulary segmenter. Across AVSBench, ADE Sound Prompted, and AVSS, the resulting system, TACO, reports the best numbers among training-free and unsupervised approaches, including a mask-IoU of 64.04 on AVSBench S4. The practical point is that large frozen models already encode the cross-modal correspondence needed for localization, so task-specific fine-tuning is unnecessary.

What carries the argument

The load-bearing mechanism is Semantically constrained soft co-NMF (Sem co-NMF), a training-free factorization that jointly decomposes frozen audio and image token features while enforcing semantic agreement. Each modality is factorized as $X \approx U V$ with non-negative factors and sigmoid-bounded activations; the semantic bridge is a word bank whose text embeddings are passed through the CLIP and CLAP text encoders to create paired anchors $(b_I^j, b_A^j)$. For each factor $k$, a soft-masked pooled component $C^k = \mathrm{avg}(X \odot U^k)$ is compared with these anchors to form a semantic descriptor $D^k$, and the objective penalizes the cross-entropy between the closest audio and image descriptor pair. The factor $k^*$ minimizing that penalty is the sounding concept; its activations give the localization and its factor vector prompts FC-CLIP for refined masks.

What would settle it

Use a test set of audio-image pairs in which the sounding object is deliberately excluded from the word bank (e.g., a rare instrument not among AudioSet tags). If TACO still localizes the object correctly, the semantic anchors are not doing the claimed work; if segmentation collapses, the word bank coverage is load-bearing. A second check: replace the word embeddings with random vectors of the same dimension; if performance stays high, the semantic alignment is unnecessary, and if it drops sharply, the anchors are genuinely aligning the two spaces.

Watch

Extended reading notes

Core claim

The central claim is that frozen CLIP and CLAP features contain enough aligned information for audio-visual source localization without any parameter updates. The paper's Sem co-NMF decomposes the audio token matrix $X_A$ and the image token matrix $X_I$ into non-negative factor and activation matrices, with activations in $[0,1]$ via a sigmoid reparameterization. To bridge the fact that CLIP and CLAP live in different embedding spaces, the method projects each factor's soft-masked component onto a shared semantic space by computing cosine similarities against word embeddings from a word bank, and then penalizes the cross-entropy between the closest audio and image descriptors. The index $k^*$ with minimal cross-entropy identifies the sounding factor; its image activation row $U^{k^*}_I$ is a coarse segmentation, and the corresponding image factor $V^{k^*}_I$, which lives in CLIP space, is used to prompt the FC-CLIP open-vocabulary segmenter. When these components are combined, the paper reports state-of-the-art unsupervised performance on S4, MS3, ADE Sound Prompted, and the semantic variants.

Load-bearing premise

The results rest on the assumption that the same word, embedded by CLIP's text encoder and by CLAP's text encoder, lands in comparable semantic neighborhoods, so that cosine similarities to a shared word bank describe audio and visual factors on the same scale; if the two spaces drift apart or the sounding concept is absent from the word bank, the cross-entropy penalty cannot identify the correct factor.

Editorial extensions

If this is right

  • If frozen CLIP and CLAP features are sufficient, then sound-prompted segmentation no longer requires task-specific training data or fine-tuning, so the same models can be repurposed for new scenes and audio types at inference time.
  • The interpretable factors let a user read what concept was matched (e.g., "baby babbling"), and the same decomposition can be reused for other downstream tasks such as semantic segmentation.
  • The method's robustness on multi-source audio, where several objects sound together, suggests a single sounding factor can encode multiple co-active sources, potentially simplifying multi-source localization.
  • Because the decomposition is done at inference, any improvement in the frozen backbones (better CLIP or CLAP) immediately improves segmentation without re-training the framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct experimental prediction: if the semantic anchors are replaced by random vectors that are not tied to words, performance should collapse to near-chance; demonstrating that would confirm the anchors (not just the factorization) are carrying the alignment.
  • The word bank is effectively a hyperparameter; using dataset-specific class names improves results, which implies that a fixed general word bank may limit generalization to very rare or unseen sound concepts, making adaptive anchor selection a natural testable extension.
  • The cross-entropy penalty between semantic descriptors assumes the closest audio-image factor pair is the correct one; on clips with multiple simultaneous sounds this could be ambiguous, so an explicit multi-factor matching or a top-k penalty could improve MS3-style tasks.
  • The framework should transfer to other paired modalities (e.g., text-image or video-text) as long as a shared word-bank space exists, opening a route to training-free object localization from natural language descriptions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. TACO proposes a training-free pipeline for sound-prompted segmentation. It uses frozen CLIP and CLAP encoders to obtain image and audio token matrices, factorizes each matrix into a small number of factors and activations via a soft co-NMF objective, and aligns one audio factor with one image factor using cosine similarities to a shared word bank. The selected image activation U^{k*}_I is used directly as a mask, and the corresponding image factor V^{k*}_I prompts FC-CLIP for a refined open-vocabulary segmentation. The paper reports state-of-the-art results on AVSBench S4/MS3 and ADE Sound Prompted, introduces a semantic variant of the ADE Sound Prompted dataset, and supports its design with ablations over the penalty, semantic components, word bank size, K, and beta_p.

Significance. If the method is implemented as described, this is a significant contribution: it would show that frozen CLIP/CLAP features contain enough audio-visual correspondence for sound-prompted localization without any training, and that NMF-style factor decomposition yields interpretable prompts that improve an open-vocabulary segmenter. The paper is transparent in important ways: it reports standard deviations over three runs, ablates the main design choices, and gives a detailed pseudo-code. The significance is currently tempered by the mismatch between the stated non-negativity/bounded constraints and the unconstrained updates in Algorithm 1, and by the partly self-referential selection of the sounding factor.

major comments (4)
  1. [Section 3.2 / Eq. (6) / Algorithm 1 (Appendix C)] The optimization actually described by the manuscript is inconsistent with the claims. Section 3.2 says UA and UI are re-parameterized through the sigmoid so that they lie in [0,1], and Eq. (6) imposes non-negativity on VA, VI, UA, and UI. However, Algorithm 1 initializes UA, UI, VA, and VI from Gaussians and updates them directly by gradient descent, with no sigmoid, clipping, or projection, and Section 4 repeats that the matrices are initialized from Gaussians. Since Appendix C states that the pseudo-code details the exact computations, either the pseudo-code is not the implemented method, or the evaluated method does not satisfy the constraints on which the interpretability and soft-mask claims rest. This is load-bearing: U^{k*}_I is used as a segmentation mask, and negative entries in U affect the Hadamard products in Eq. (4), the cosine descriptors in Eq. (5), and the cross-entropy penalty in Eq. (6). Please state the exact optimization actually used, correct the pseudo-code, and re-run or confirm the experiments under the stated constraints.
  2. [Section 4 (Preliminary experiments) and Appendix A] The clamping experiment validates clamping only the CLIP text embedding before it is fed to FC-CLIP (Table 4). The inputs that actually need to be non-negative for the NMF objective are the CLIP image token features and CLAP audio features, and the semantic components C^k_I and C^k_A are averages of those features. The paper does not measure the effect of clipping those features to zero, nor does it explain how negative entries in the semantic components are handled. The claim that clipping negative values preserves representation power for 'our tasks' is therefore not supported by the provided experiment. Please validate clamping on the actual NMF inputs or specify a different mechanism for satisfying the non-negativity condition.
  3. [Eq. (5) and Eq. (6)] The penalty CE(D_I^k, D_A^k) is not well defined as written. The descriptors D are vectors of cosine similarities, which can take negative values and are not normalized to be probability distributions. Cross-entropy requires two probability distributions (or a distribution and logits, with the direction specified), so an additional normalization such as softmax must be specified. Without this, the objective in Eq. (6) is ambiguous, and different choices of normalization will change the gradient and the final k*. Please give the exact definition of the cross-entropy used, including whether the descriptors are passed through softmax and which descriptor is treated as the target.
  4. [Section 4.1 / Eq. (6) / Interpretation paragraph] The selection of the sounding factor is partly by construction. The optimization objective in Eq. (6) already contains min_k CE(D_I^k, D_A^k), and k* is then defined as the argmin of the same quantity. This guarantees that the selected factor is aligned in descriptor space, so the matching step alone does not provide independent evidence that descriptor alignment identifies the true sound source. The external mask-IoU evaluations support the full pipeline, but the interpretability claim that k* 'corresponds to the dominant semantic component shared between the audio and image modalities' needs additional validation. Please report, for example, how often k* agrees with a ground-truth source label, or compare the final segmentation when k* is chosen versus a random factor or the second-best factor.
minor comments (6)
  1. [Section 3.2] Since UI is defined as an HW x K matrix, the spatial activation of factor k is the k-th column, not the k-th row; the sentence 'the kth row of UI represents the segmentation of the kth factor' should be corrected, as should the similar sentence after Eq. (6).
  2. [References] References [43] and [44] have the same authors and title; one appears to be a duplicate, and the entry used in Table 1 should be aligned with the intended method.
  3. [Section 4.1] The sentence 'only outperformed by MarginNCE in terms of F-score' is imprecise: in Table 1 MarginNCE also reports a higher mask-IoU on S4 than TACO without the segmenter (33.27 vs. 29.68).
  4. [Section 4 (Datasets and Metrics)] The F-score metric used in Tables 1-3 is not defined; please state how it is computed.
  5. [Section 4 (Implementation Details)] The explanation for setting beta_temp = 0 for the multi-source task is confusing: AVS-MS3 is a multi-source video benchmark, so it is not clear why temporal consistency is disabled precisely for the multi-source setting; please clarify.
  6. [Algorithm 1 (Appendix C)] Line 14 calls s(V_It, X_I), but the pseudo-code input list defines only V_I and X_I; align the notation for the multi-frame case.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sounding-factor selection is the method's operational decision rule, and the benchmark evaluations are external to the optimized objective.

full rationale

TACO's central derivation is not circular. The sounding factor is selected as k* = argmin_k CE(D_I^k, D_A^k) after minimizing Eq. (6), which includes the same cross-entropy penalty. This is an operational decision rule, not a prediction smuggled from the inputs: the penalty is the objective being optimized, and nothing in that objective encodes the ground-truth segmentation masks. The reported mask-IoU/mIoU numbers therefore test whether the minimizer of the semantic-alignment penalty actually localizes the sound source, which is an external and falsifiable claim; Appendix I even documents failure cases where the selected factor does not correspond to the true source. Using the same word bank both as semantic anchors and as the label set for semantic segmentation constrains the possible class outputs, but it does not make the spatial mask equivalent to the input: the algorithm still must discover which image factor's masked descriptor matches the audio descriptor. The paper's self-citations ([28], [32]) are peripheral and not load-bearing. One substantive concern is an implementation inconsistency: Section 3.2 claims a sigmoid re-parameterization enforcing U_A, U_I in [0,1], while Algorithm 1 initializes U_A, U_I, V_A, V_I from Gaussians and applies unconstrained gradient updates, with no sigmoid or projection; this is a correctness/reproducibility issue, not circularity, and it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

No new physical or mathematical entities are postulated. The semantic factors and anchors are latent constructions derived from existing pre-trained encoders. The main free parameters are inference-time hyperparameters and the word bank, all selected by hand or on the validation set. The axioms are domain assumptions about cross-modal alignment, feature clipping, and FC-CLIP compatibility, which are load-bearing for the method's success.

free parameters (5)
  • K (number of NMF factors) = 8
    Set on AVSBench validation; ablation shows stability from K=6 to K=12 (Appendix B, Table 5).
  • beta_p (penalty weight) = 125
    Selected on AVSBench validation; ablation (Appendix B, Table 6) shows stability between 100 and 150.
  • beta_temp (temporal consistency weight) = 1 for single-source, 0 for multi-source
    Chosen in Section 4 Implementation details; not extensively ablated.
  • Word bank choice = 527 AudioSet tags for non-semantic; dataset class names for semantic
    Appendix B Table 7 shows the specific word bank changes results by up to about 3.5 points, so the choice is consequential.
  • Optimization schedule = 1800 steps, learning rate 0.25, Gaussian initialization with unspecified variance
    From Section 4; Appendix D shows performance plateaus near 1000 iterations, but no sensitivity analysis for learning rate is provided.
assumptions (6)
  • domain assumption Identical words in the CLIP and CLAP text encoders define comparable semantic directions, so cosine similarities to a shared word bank form a common audio-visual semantic space.
    Invoked in Section 3.2 to define anchors bj_I and bj_A and the cross-entropy penalty; if the text embeddings are not semantically aligned across modalities, the descriptors cannot match factors.
  • domain assumption Clipping negative CLIP/CLAP features to zero preserves the information needed for spatial decomposition.
    Section 3 states clipping is used; Appendix A only validates clamping on text prompts, not on the image features being decomposed.
  • domain assumption A soft-masked average of features (Ck_I = avg(XI * U k_I)) is a valid semantic descriptor of a factor.
    Section 3.2, Equations (4) and (5); the penalty and k* selection rely on this averaging being semantically meaningful.
  • domain assumption The factor with minimum descriptor cross-entropy corresponds to the sounding object or objects.
    Section 3.2, Interpretation; the whole method selects k* = argmin CE without external verification of this correspondence.
  • domain assumption FC-CLIP can be prompted with an arbitrary CLIP image-space vector VI^{k*} to yield a segmentation mask.
    Section 3.3; the performance gain depends on this compatibility, which is asserted but not analyzed.
  • domain assumption The word bank contains lexical entries for the sounding concepts appearing in the datasets.
    Section 4 word bank choices; if the concept is absent, descriptors cannot identify it, and Appendix B shows degradation with smaller word banks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization." pith.science (2026). https://pith.science/paper/DEV5SXLH

@misc{pith2026241201488,
  author       = {Pith},
  title        = {Pith review of: TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DEV5SXLH}},
  note         = {Machine review of arXiv:2412.01488}
}
read the original abstract

Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.

Figures

Figures reproduced from arXiv: 2412.01488 by the authors.

Figure 1
Figure 1. Our method takes a representation of an image and its associated audio as input, decompos [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Complete pipeline: both the audio and the image are encoded using their respec￾tive encoder and their representations are used to perform the co-NMF. FC-CLIP is prompted using the image factors (VI ) and the segmen￾tation corresponding to the sounding image factor (V k⋆ I ) is kept as the final segmentation. CLIP CLAP Audio Image Audio-visual semantic space CE Penalty CLIP Space CLAP Space [PITH_FULL_IMAGE:figures/… view at source ↗
Figure 5
Figure 5. Multiple source segmentation exam￾ples. As the sources change during the video, TACO’s segmentation changes as well [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figures from the paper (6 more)
Figure 6
Figure 6. Figure 6: Illustration of the decomposition [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Proportion of additional computing time induce by the decomposition. The decomposition [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Semantic segmentation examples from AVSS. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Sound-prompted segmentation examples from ADE SP. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Sound prompted-segmentation examples from MS3. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Examples of typical failure cases on MS3. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 45 canonical work pages

  1. [1]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 2021

    work page 2021

  2. [2]

    Localizing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021

    work page 2021

  3. [3]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020

    work page 2020

  4. [4]

    Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset, 2023

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset, 2023

    work page 2023

  5. [5]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    work page 2023

  6. [6]

    Meerkat: Audio-visual large language model for grounding in space and time

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In European Conference on Computer Vision. Springer, 2025

    work page 2025

  7. [7]

    Multilayer nonnegative matrix factorisation

    Andrzej Cichocki, Rafal Zdunek, et al. Multilayer nonnegative matrix factorisation. ELEC- TRONICS LETTERS-IEE, 42(16), 2006

    work page 2006

  8. [8]

    Deep feature factorization for concept discovery

    Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorization for concept discovery. In Proceedings of the European Conference on Computer Vision (ECCV), 2018

    work page 2018

Show all 52 references
  1. [9]

    Neural network matrix factorization

    Gintare Karolina Dziugaite and Daniel M Roy. Neural network matrix factorization. arXiv preprint arXiv:1511.06443, 2015

    2015 arXiv

  2. [10]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023

    2023

  3. [11]

    Avsegformer: Audio-visual segmentation with transformer

    Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024

    2024

  4. [12]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (...

    2017

  5. [13]

    The why and how of nonnegative matrix factorization

    Nicolas Gillis. The why and how of nonnegative matrix factorization. Regularization, optimiza- tion, kernels, and support vector machines, 12(257), 2014

    2014

  6. [14]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  7. [15]

    Contrastive audio-visual masked autoencoder

    Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022

    2022 arXiv

  8. [16]

    Single channel speech music separation using nonnegative matrix factorization and spectral masks

    Emad M Grais and Hakan Erdogan. Single channel speech music separation using nonnegative matrix factorization and spectral masks. In 2011 17th International Conference on Digital Signal Processing (DSP). IEEE, 2011. 10

    2011

  9. [17]

    Non-negative matrix factorization for face recognition

    David Guillamet and Jordi Vitria. Non-negative matrix factorization for face recognition. In Catalonian Conference on Artificial Intelligence. Springer, 2002

    2002

  10. [18]

    chirp" from the

    Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the" chirp" from the" chat": Self-supervised visual grounding of sound and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  11. [19]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29, 2021

    2021

  12. [20]

    Transfer learning from audio-visual grounding to speech recognition

    Wei-Ning Hsu, David Harwath, and James Glass. Transfer learning from audio-visual grounding to speech recognition. arXiv preprint arXiv:1907.04355, 2019

    1907 arXiv

  13. [21]

    A critical assessment of visual sound source localization models including negative audio

    Xavier Juanola, Gloria Haro, and Magdalena Fuentes. A critical assessment of visual sound source localization models including negative audio. arXiv preprint arXiv:2410.01020, 2024

    2024 arXiv

  14. [22]

    Algorithms for non-negative matrix factorization

    Daniel Lee and H Sebastian Seung. Algorithms for non-negative matrix factorization. Advances in neural information processing systems, 13, 2000

    2000

  15. [23]

    Unsupervised sound localization via iterative contrastive learning

    Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. Unsupervised sound localization via iterative contrastive learning. Computer Vision and Image Understanding, 2023

    2023

  16. [24]

    Audio-visual segmentation by exploring cross-modal mutual semantics

    Chen Liu, Peike Patrick Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, and Xin Yu. Audio-visual segmentation by exploring cross-modal mutual semantics. In Proceedings of the 31st ACM International Conference on Multimedia, 2023

    2023

  17. [25]

    Audio-aware query-enhanced transformer for audio-visual segmentation

    Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, and Ya Zhang. Audio-aware query-enhanced transformer for audio-visual segmentation. arXiv preprint arXiv:2307.13236, 2023

    2023 arXiv

  18. [26]

    Exploiting transformation invariance and equivariance for self-supervised sound localisation

    Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, 2022

    2022

  19. [27]

    Image segmentation using text and image prompts

    Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022

    2022

  20. [28]

    An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching

    Hugo Malard, Michel Olvera, Stéphane Lathuilière, and Slim Essid. An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  21. [29]

    A closer look at weakly-supervised audio-visual source localization

    Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. Advances in Neural Information Processing Systems, 2022

    2022

  22. [30]

    A concept-based explainability framework for large multimodal models

    Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. A concept-based explainability framework for large multimodal models. arXiv preprint arXiv:2406.08074, 2024

    2024 arXiv

  23. [31]

    Listen to interpret: Post-hoc interpretability for audio networks with nmf

    Jayneel Parekh, Sanjeel Parekh, Pavlo Mozharovskyi, Florence d’Alché Buc, and Gaël Richard. Listen to interpret: Post-hoc interpretability for audio networks with nmf. Advances in Neural Information Processing Systems, 35, 2022

    2022

  24. [32]

    Guiding audio source separation by video object information

    Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc QK Duong, Patrick Pérez, and Gaël Richard. Guiding audio source separation by video object information. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2017

    2017

  25. [33]

    Marginnce: Robust sound localization with a negative margin

    Sooyoung Park, Arda Senocak, and Joon Son Chung. Marginnce: Robust sound localization with a negative margin. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023. 11

    2023

  26. [34]

    Can clip help sound source localization? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024

    Sooyoung Park, Arda Senocak, and Joon Son Chung. Can clip help sound source localization? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024

    2024

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021

    2021

  28. [36]

    Laion- 5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Inform...

    2022

  29. [37]

    Soft nonnegative matrix co-factorization

    Nicolas Seichepine, Slim Essid, Cédric Févotte, and Olivier Cappé. Soft nonnegative matrix co-factorization. IEEE Transactions on Signal Processing, 62(22), 2014

    2014

  30. [38]

    Learning to localize sound source in visual scenes

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018

    2018

  31. [39]

    Learning to localize sound sources in visual scenes: Analysis and applications

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound sources in visual scenes: Analysis and applications. IEEE transactions on pattern analysis and machine intelligence, 43(5), 2019

    2019

  32. [40]

    Learning sound localization better from semantically similar samples

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Learning sound localization better from semantically similar samples. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022

    2022

  33. [41]

    Sound source localization is all about cross-modal alignment

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Sound source localization is all about cross-modal alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    2023

  34. [42]

    Increasing importance of joint analysis of audio and video in computer vision: A survey

    Ahmed Shahabaz and Sudeep Sarkar. Increasing importance of joint analysis of audio and video in computer vision: A survey. IEEE Access, 2024

    2024

  35. [44]

    Learning audio-visual source localization via false negative aware contrastive learning

    Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning audio-visual source localization via false negative aware contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2023

  36. [45]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), 2018

    2018

  37. [46]

    Combining non-negative matrix factoriza- tion and deep neural networks for speech enhancement and automatic speech recognition

    Thanh T Vu, Benjamin Bigot, and Eng Siong Chng. Combining non-negative matrix factoriza- tion and deep neural networks for speech enhancement and automatic speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016

    2016

  38. [47]

    Document clustering based on non-negative matrix factorization

    Wei Xu, Xin Liu, and Yihong Gong. Document clustering based on non-negative matrix factorization. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, 2003

    2003

  39. [48]

    Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion

    Naoto Yokoya, Takehisa Yairi, and Akira Iwasaki. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50(2), 2011. 12

    2011

  40. [49]

    Matrix co-factorization on compressed sensing

    Jiho Yoo and Seungjin Choi. Matrix co-factorization on compressed sensing. In Proceedings of the Twenty-Second international joint conference on Artificial Intelligence-Volume Volume Two, 2011

    2011

  41. [50]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36, 2023

    2023

  42. [51]

    Semantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127, 2019

    2019

  43. [52]

    Audio-visual segmentation with semantics

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, et al. Audio-visual segmentation with semantics. International Journal of Computer Vision, 2024

    2024

  44. [53]

    Audio–visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio–visual segmentation. In European Conference on Computer Vision. Springer, 2022. 13 The appendix is organized as follows: the first part...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.