Pith. sign in

REVIEW 3 major objections 5 minor 30 references

Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen audio-visual retrieval model can be turned into an accurate sound-source localizer by replacing global pooling with audio-informed spatial pooling.

desk verdict A plausible and clever method for unlocking localization from frozen retrieval backbones, with genuine gains, but the missing validation split and inconsistent equations keep me from trusting the exact numbers. read the letter →

arxiv 2607.24786 v1 pith:FPVYWRYT submitted 2026-06-22 cs.IR cs.AIcs.MMcs.SDeess.AS

Hugo Malard , Michel Olvera , Sanjeel Parekh , Gaël Richard , Slim Essid , Stéphane Lathuilière This is my paper
classification cs.IRcs.AIcs.MMcs.SDeess.AS
keywords soundsourcelocalizationaudio-visualretrievalweaklysupervisedlearningaudio-informedpoolingcross-modalattentionlarge-scalepretrainingspatialgroundingcontrastive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper demonstrates that large-scale audio-visual retrieval models, trained only for global audio-video matching, already contain the spatial information needed for sound-source localization. The authors show that while upper layers of the frame encoder lose spatial detail due to global pooling, intermediate visual tokens retain structured locality. Their method, LAIP, uses the audio signal to query these intermediate tokens through a lightweight Audio-informed Spatial Pooling (AiSP) mechanism, producing attention maps that serve as localization maps. Trained with a global contrastive objective alone, LAIP achieves state-of-the-art results on AVSBench, AVATAR, and ADE-SP, nearly doubling prior performance on AVATAR. The central claim is that accurate localization can be 'unlocked' from existing retrieval representations rather than learned from scratch, offering a unified path for retrieval and localization.

What carries the argument

The central mechanism is Audio-informed Spatial Pooling (AiSP), a cross-attention pooling operator in which a frame-aligned audio token serves as the query over dense visual tokens from an intermediate layer of the frozen frame encoder. A learnable null token gives the audio an explicit fallback when no visual region corresponds to the sound. Three stacked AiSP modules with K={2,2,6} progressively halve and then steeply reduce the spatial grid, producing multi-scale attention maps; at inference these maps are averaged after selecting the best-matching head of the final stage. This design recovers localized audio-visual correspondences while keeping the pooled token compatible with the frozen

What would settle it

Train LAIP on a different large-scale retrieval backbone (e.g., a CLIP-style model or a video-text model) using the same recipe, and check whether audio-informed pooling at an intermediate layer still yields localization maps. If the performance drop is severe or requires re-tuning the layer for each backbone, the claim that localization is a general emergent property of retrieval representations would be weakened; alternatively, a sharp performance cliff when swapping layer 16 for layer 15 or 17 without retuning would indicate the result rides on a layer-specific accident rather than a robust

Watch

Extended reading notes

Core claim

The paper's core discovery is that spatial grounding is not absent from large audio-visual retrieval models—it is merely hidden in intermediate layers and discarded by the final global pooling. By inserting an audio-conditioned pooling module between the frame and video encoders of the frozen PE-AV backbone, LAIP lets frame-aligned audio embeddings query dense visual tokens from layer 16, progressively down-sampling them into a single sound-conditioned token. The resulting attention maps, averaged over three resolutions, localize sounding objects with 27.63 CIoU on AVATAR (vs 13.42 for prior state of the art) and 65.18 F-score on AVSBench S4 (vs 45.33). The authors conclude that localization

Load-bearing premise

The method assumes that intermediate visual tokens of the frozen frame encoder (specifically those from layer 16) remain spatially queryable and compatible with the audio encoder's feature space; if that layer-specific spatial structure is absent or misaligned, the audio-informed pooling has nothing meaningful to extract.

Editorial extensions

If this is right

  • Retrieval and localization can share a single pretrained backbone: dropping the adapters restores the original PE-AV retrieval forward pass, so one model serves both tasks.
  • Accurate localization maps can be obtained with only a global contrastive objective plus two regularizers, without dense pixel-level supervision.
  • The approach transfers to sound-prompted segmentation benchmarks (ADE-SP) and multi-source audio-visual segmentation (AVSBench MS3), indicating the unlocked spatial representations generalize beyond the training setup.
  • Small, targeted adapters are sufficient to bridge the gap between the audio-informed token and the pretrained temporal stack, avoiding full fine-tuning of a 1.7B-parameter model.
  • The null-token regularization provides a mechanism for abstention when audio has no visible source, which is relevant to off-screen sound handling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The choice of layer 16 and the pooling schedule K={2,2,6} are empirical; the paper does not present a principled selection criterion. A natural extension would be to learn the layer index or pooling schedule end-to-end, which might improve robustness across backbones.
  • Because the method relies on intermediate-layer spatial structure, it may transfer to other large-scale retrieval models (e.g., image-text or video-text models) whose intermediate tokens also retain locality—if the same audio-informed pooling unlocks localization there, the 'unlocking' phenomenon is a general property of scaled contrastive training.
  • The sharp, peaky attention maps hurt fixed-threshold mask-IoU on AVSBench; a learned calibration or temperature-scaled binarization step could close the gap to F-score performance and make the maps directly usable for segmentation.
  • The explicit null token and multi-scale consistency losses may be reinterpreted as a form of weak abstention learning; this could be applicable to other weakly supervised grounding tasks where the input modality may be silent or absent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LAIP (Localization via Audio-Informed Pooling), a method for weakly supervised audio-visual sound source localization built on top of the frozen PE-AV retrieval backbone. The central idea is to replace the global frame-level pooling in PE-AV with a multi-stage Audio-informed Spatial Pooling (AiSP) module that operates on intermediate visual tokens (layer 16) and is queried by frame-aligned audio tokens. The module is trained with a SigLIP-style contrastive objective on 10k VGGSound videos, together with multi-resolution and null-token regularizers. The authors report large gains over previous state of the art on AVATAR (27.63 vs 13.42 CIoU), AVSBench S4 (65.18 vs 45.33 F-score), and ADE-SP, and provide extensive ablations over the pooling schedule, layer choice, and regularizers.

Significance. If the results hold, this is a significant finding: it suggests that large-scale retrieval models contain spatially queryable structure in intermediate layers that can be exploited by a lightweight audio-conditioned pooling module, avoiding dense spatial supervision. The paper's strengths are its extensive ablations, evaluation on multiple external benchmarks, and the architectural clarity of the proposed AiSP mechanism. The reported gains are large and the method is plausible. However, the central empirical claim currently rests on a missing validation protocol for key hyperparameters, and the stated training objective contains sign and definition errors. The significance is therefore conditional on resolving these issues.

major comments (3)
  1. [§3.3, Eq. (4)] The final objective is written as L_SIG + λ L_MRes − μ L∅, but the text states that L∅ is a penalty that encourages the model to route attention to the null token (i.e., abstain from visual tokens). Since training minimizes the objective, subtracting μL∅ would reward large null-token attention, implementing the opposite of the described regularization. This is internally inconsistent and load-bearing: the null-token term is credited with part of the performance gains in Tables 4 and 7. Please correct the sign to +μL∅ or explicitly justify the current sign if the intended optimization is different.
  2. [§3.3, Eq. (1)] Equation (1) defines L_SIG = 1/(1+exp(y(−τ⟨O,A_g⟩+b))), which is the probability of the incorrect label (σ(−y logit)) rather than the standard SigLIP loss (−log σ(y logit)). Minimizing this quantity is a valid but unusual surrogate; it is not the standard sigmoid contrastive loss, and its gradients behave differently, especially when the model is confident. If the implementation actually uses the standard negative-log-sigmoid loss, Eq. (1) is incorrect as written. Please provide the exact optimized loss and clarify which form is used in the reported experiments.
  3. [§4.1–4.2, Tables 4–9] The paper does not describe any validation split or a principled protocol for selecting the key hyperparameters: intermediate layer ℓ=16, pooling schedule K={2,2,6}, loss weights μ=0.01 and λ=100, adapter size 512, and the number of attention heads. The sensitivity analyses in Tables 4, 5, 7, 8, and 9 show that these choices have a very large effect (e.g., LAIP on the last layer drops from 26.22 to 15.92 CIoU; the inverted pooling schedule drops to 13.67). Because these ablations are reported on AVATAR test scenarios and the selection procedure is not described, it is not possible to rule out test-set hyperparameter tuning. The central claim that localization can be 'unlocked' from a frozen retrieval model requires either a held-out validation split used for model selection, a pre-registered protocol, or independent confirmation. Please add the selection protocol or provide an additional
minor comments (5)
  1. [Abstract and §5] The wording 'These findings prove...' is too strong for a benchmark-based empirical study. I suggest 'demonstrate' or 'provide evidence that'.
  2. [§4.2, Tables 2–3] The text says min-max normalization is applied to AVSBench results before evaluation. Please clarify explicitly whether this normalization is applied to all methods or only to LAIP; if it is LAIP-only, the comparison may be inconsistent.
  3. [§3.3] Minor typo: 'the norm of the attention payed to the null token' should be 'paid'.
  4. [Tables 4–9] No error bars or multiple-seed results are provided. Given the large reported gains, a reproducibility statement with code or at least multi-seed variance would strengthen the empirical claims.
  5. [§4.1] The paper adds about 35M parameters in AiSP plus 7M adapter parameters, described as 'lightweight'. Please quantify this relative to the baselines' trainable parameter counts, since 42M is not negligible in absolute terms.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: localization maps emerge from a global contrastive objective trained on unlabeled VGGSound pairs and are evaluated on external AVATAR/AVSBench/ADE-SP benchmarks; the sole self-citation (TACO) is a non-load-bearing baseline.

full rationale

The claimed derivation chain is empirical and self-contained with respect to the target masks. LAIP inserts an audio-informed pooling module between the frozen PE-AV frame encoder and video encoder, and trains it with the SigLIP-style contrastive loss L_SIG (Eq. 1) plus multi-resolution consistency and null-token regularizers (Eqs. 2-4). The localization heatmaps are the attention maps used by this pooling; they are not directly supervised by the AVATAR/AVSBench segmentation masks, because training uses only VGGSound video-audio pairs. The evaluation on AVATAR, AVSBench S4/MS3, and ADE-SP is against held-out external annotations, so the reported 27.63 CIoU and 65.18 F-score are not re-statements of the training objective. The layer-16 choice is justified by an external observation ([19]) and an ablation (Table 5), not by a self-citation. The only self-citation, TACO [18], appears as a baseline/competitor and in qualitative comparisons; it is not used to justify the central premise or to force the design, so it is not load-bearing. Possible concerns — absence of a described validation split for hyperparameters (layer 16, K={2,2,6}, lambda=100, mu=0.01), the sign of the mu L_null term in Eq. (4), and the non-standard SigLIP formula in Eq. (1) — are reproducibility/consistency issues, not reductions of the prediction to the fit. No equation in the paper is shown by construction to equal its input, and no fitted parameter is renamed as a prediction. Therefore no circularity step meets the required evidence bar.

Assumptions & free parameters 8 free parameters · 5 assumptions · 1 invented entities

The paper introduces a moderately sized set of architecture-specific choices (layer, pooling schedule, regularization weights, adapter size) that are all tuned by hand on the task. These are not derived from theory, so they count as free parameters. The null token is the only new 'entity', and it is a standard ML mechanism without independent physical evidence.

free parameters (8)
  • intermediate layer ℓ = 16
    Visual tokens are taken from the 16th layer of the 24-layer frame encoder; the paper cites [19] but does not justify this exact depth.
  • pooling factors K = {2,2,6}
    Spatial downsampling schedule for the three AiSP stages; chosen by hand, ablations show alternatives are worse.
  • null-attention weight μ = 0.01
    Weight on the null-token attention term in Eq. (4); chosen for training stability.
  • multi-resolution weight λ = 100
    Weight on the cross-resolution consistency loss; set so the two loss terms have similar magnitude.
  • adapter size = 512
    Dimension of the MLP adapters inserted into the video encoder.
  • number of attention heads in last AiSP = 8
    Last-stage heads; the best-matching head is chosen at inference (Eq. 3).
  • learned temperature τ and bias b = learned
    Parameters of the sigmoid contrastive loss in Eq. (1).
  • number of AiSP stages = 3
    Hierarchy depth; ablation shows two stages are worse.
assumptions (5)
  • standard math Softmax attention with cosine similarity provides a differentiable localization signal
    Used throughout Section 3.2.
  • domain assumption Intermediate visual tokens in PE-AV frame encoder retain local spatial information; global pooling in upper layers destroys it
    Section 3.1; supported only by citation [19] and the paper's own ablations.
  • domain assumption The audio encoder's representation is compatible with the visual token space, so that audio tokens can serve as queries
    Section 3.2; not explicitly verified beyond the reported results.
  • domain assumption A global contrastive objective over video-audio pairs induces correct spatial attention in the pooling module
    Section 3.3; the core weak-supervision hypothesis.
  • domain assumption The VGGSound 10k training subset is a valid proxy for AVATAR/AVSBench/ADE-SP evaluation domains
    Section 4.1; used for all training.
invented entities (1)
  • learnable null token x∅
    purpose: gives the audio query an explicit fallback so it can abstain from selecting any visual token when no sound source matches
    A learned latent; its behavior is only evaluated indirectly through TN on off-screen AVATAR and ablations. No external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlocking Spatial Grounding in Large Audio-Visual Retrieval models." pith.science (2026). https://pith.science/paper/FPVYWRYT

@misc{pith2026260724786,
  author       = {Pith},
  title        = {Pith review of: Unlocking Spatial Grounding in Large Audio-Visual Retrieval models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FPVYWRYT}},
  note         = {Machine review of arXiv:2607.24786}
}
read the original abstract

Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale. The task, however, remains challenging, as models must locate sound sources from temporally aligned audio-visual data without pixel-level supervision. Recent large-scale audio-visual retrieval models, trained at unprecedented scale, encode rich multimodal structure. We show their latent representations, though optimized for global alignment, can nonetheless enable fine-grained spatial grounding. While spatial detail is progressively lost in the upper layers of retrieval backbones due to global pooling, intermediate visual tokens retain highly structured spatial information. To exploit this, we introduce LAIP (\emph{Localization via Audio-Informed Pooling}), a framework that employs a lightweight \emph{Audio-informed Spatial Pooling} (AiSP) to replace the standard global aggregation module. By using frame-aligned audio to query intermediate visual tokens, LAIP recovers localized spatial information that is otherwise discarded by the frozen retrieval pipeline. Our approach achieves state-of-the-art performance on AVSBench and AVATAR, nearly doubling previous results on the latter. These findings prove that accurate localization does not need to be learned from scratch; instead, it can be unlocked from existing retrieval representations, providing a unified path for both retrieval and localization tasks.

Figures

Figures reproduced from arXiv: 2607.24786 by the authors.

Figure 1
Figure 1. Global cross-modal alignment training induces a loss of spatial information in both frame [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. LAIP: Localization via Audio-Informed Pooling. The audio tokens are first forwarded to a local aggregator to match visual frame rate, and then used to pool the frame representation through our proposed localization module. The pooled representations are used as input to the video encoder, which outputs the final representation used to compute the sigmoid loss. Consistent with the observations in [19], we posit that … view at source ↗
Figure 3
Figure 3. Overview of our localization and AiSP modules, with [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison between TACO and LAIP on the S4 dataset. While both methods [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: shows the attention maps extracted in each AiSP module in our model, as well as the aggregated prediction and the segmentation obtained by CAV-MAE Sync in the same example (from the S4 dataset). This example is particularly interesting as both models were trained (or p…
Figure 6
Figure 6. Figure 6: Impact of the thresholding on attention maps. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: PE-AV does not encode audio-visual spatial correspondences. The norm of the gradient of [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Qualitative samples of LAIP on the MS3 (first row) an S4 (second raw) dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Failure cases of LAIP. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 1 linked inside Pith

  1. [1]

    Localizing visual sounds the easy way

    Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. InEuropean Conference on Computer Vision, 2022

    2022

  2. [2]

    A closer look at weakly-supervised audio-visual source localization

    Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. Advances in Neural Information Processing Systems, 35, 2022

    2022

  3. [3]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PmLR, 2021

    2021

  4. [4]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, 2023

    2023

  5. [5]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023

    2023

  6. [6]

    Pushing the frontier of audiovisual perception with large-scale multimodal correspondence learning.arXiv preprint arXiv:2512.19687, 2025

    Apoorv Vyas, Heng-Jui Chang, Cheng-Fu Yang, Po-Yao Huang, Luya Gao, Julius Richter, Sanyuan Chen, Matt Le, Piotr Dollár, Christoph Feichtenhofer, et al. Pushing the frontier of audiovisual perception with large-scale multimodal correspondence learning.arXiv preprint arXiv:2512.19687, 2025

    arXiv 2025

  7. [7]

    Learning audio-visual source localization via false negative aware con- trastive learning

    Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning audio-visual source localization via false negative aware con- trastive learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023

    2023

  8. [8]

    What’s making that sound right now? video-centric audio-visual localization

    Hahyeon Choi, Junhoo Lee, and Nojun Kwak. What’s making that sound right now? video-centric audio-visual localization. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2025

    2025

Show all 30 references
  1. [9]

    Flair: Vlm with fine-grained language-informed image representations

    Rui Xiao, Sanghwan Kim, Mariana-Iuliana Georgescu, Zeynep Akata, and Stephan Alaniz. Flair: Vlm with fine-grained language-informed image representations. InProceedings of the computer vision and pattern recognition conference, 2025

    2025

  2. [10]

    Learning to localize sound source in visual scenes

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. InProceedings of the IEEE conference on computer vision and pattern recognition, 2018

    2018

  3. [11]

    Exploiting transformation invariance and equivariance for self-supervised sound localisation

    Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. InProceedings of the 30th ACM International Conference on Multimedia, pages 3742–3753, 2022

    2022

  4. [12]

    Marginnce: Robust sound localization with a negative margin

    Sooyoung Park, Arda Senocak, and Joon Son Chung. Marginnce: Robust sound localization with a negative margin. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

    2023

  5. [13]

    Sound source localization is all about cross-modal alignment

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Sound source localization is all about cross-modal alignment. InProceedings of the IEEE/CVF international conference on computer vision, 2023

    2023

  6. [14]

    Audio–visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio–visual segmentation. InEuropean Conference on Computer Vision, 2022

    2022

  7. [15]

    Av-sam: Segment anything model meets audio-visual localization and segmentation.arXiv preprint arXiv:2305.01836, 2023

    Shentong Mo and Yapeng Tian. Av-sam: Segment anything model meets audio-visual localization and segmentation.arXiv preprint arXiv:2305.01836, 2023

    2023 arXiv

  8. [16]

    Audio visual segmentation through text embeddings

    Kyungbok Lee, You Zhang, and Zhiyao Duan. Audio visual segmentation through text embeddings. In 2025 IEEE International Conference on Image Processing (ICIP). IEEE, 2025

    2025

  9. [17]

    Open-vocabulary audio-visual semantic segmentation

    Ruohao Guo, Liao Qu, Dantong Niu, Yanyu Qi, Wenzhen Yue, Ji Shi, Bowei Xing, and Xianghua Ying. Open-vocabulary audio-visual semantic segmentation. InProceedings of the 32nd ACM International conference on multimedia, 2024

    2024

  10. [18]

    Taco: Training-free sound prompted segmentation via semantically constrained audio-visual co-factorization.Transactions on Machine Learn- ing Research, 2026

    Hugo Malard, Michel Olvera, Stéphane Lathuilière, and Slim Essid. Taco: Training-free sound prompted segmentation via semantically constrained audio-visual co-factorization.Transactions on Machine Learn- ing Research, 2026. 10

    2026

  11. [19]

    Perception encoder: The best visual embeddings are not at the output of the network

    Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Abdul Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. InThe Thirty-ninth Annual Conference ...

    2025

  12. [20]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  13. [21]

    Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 2002

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 2002

    2002

  14. [22]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  15. [23]

    chirp" from the

    Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the" chirp" from the" chat": Self-supervised visual grounding of sound and language. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024

    2024

  16. [24]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE, 2020

    2020

  17. [25]

    Transfer learning from audio-visual grounding to speech recognition

    Wei-Ning Hsu, David Harwath, and James Glass. Transfer learning from audio-visual grounding to speech recognition. InProc. Interspeech 2019, 2019

    2019

  18. [26]

    Contrastive audio-visual masked autoencoder

    Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass. Contrastive audio-visual masked autoencoder. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  19. [27]

    Cav-mae sync: Improving contrastive audio-visual mask autoencoders via fine-grained alignment

    Edson Araujo, Andrew Rouditchenko, Yuan Gong, Saurabhchand Bhati, Samuel Thomas, Brian Kingsbury, Leonid Karlinsky, Rogerio Feris, James R Glass, and Hilde Kuehne. Cav-mae sync: Improving contrastive audio-visual mask autoencoders via fine-grained alignment. InProceedings of t...

    2025

  20. [28]

    Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip.Advances in Neural Information Processing Systems, 36, 2023

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip.Advances in Neural Information Processing Systems, 36, 2023

    2023

  21. [29]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF conference on computer vision ...

    2023

  22. [30]

    LAIP two poolers

    Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. Natural language supervision for general- purpose audio representations. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024. 11 A Multi-resolution analysis Figure...

    2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.