Pith. sign in

REVIEW 3 major objections 6 minor 50 references

CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Treating audio as a sequence of frame-aligned clips instead of one global clip lifts contrastive audio-video learning to state-of-the-art zero-shot retrieval and competitive sound-source localization.

desk verdict Well-executed CAV-MAE extension with solid ablations, but the state-of-the-art retrieval claim rests on a scoring protocol the baselines never got. read the letter →

arxiv 2505.01237 v2 pith:UYQ3SVH5 submitted 2025-05-02 cs.MM cs.CVcs.SDeess.AS

Edson Araujo , Andrew Rouditchenko , Yuan Gong , Saurabhchand Bhati , Samuel Thomas , Brian Kingsbury , Leonid Karlinsky , Rogerio Feris
show 2 more authors
James R. Glass Hilde Kuehne
This is my paper · ORCID
classification cs.MMcs.CVcs.SDeess.AS
keywords audio-visuallearningcontrastivemaskedautoencoderfine-grainedtemporalalignmentzero-shotretrievalsoundlocalizationregistertokensself-supervisedrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that two weaknesses of contrastive audio-visual masked autoencoders can be fixed with small architectural changes: matching a whole audio clip to a single random frame is too loose a learning signal, and forcing one shared representation to serve both reconstruction and cross-modal alignment sets up conflicting objectives. Its solution, CAV-MAE Sync, represents audio as a temporal sequence of short segments aligned to individual video frames, uses dedicated global tokens for the contrastive loss while patch tokens focus on reconstruction, and adds register tokens that absorb excess semantic load. The paper reports state-of-the-art zero-shot retrieval on AudioSet and VGGSound in both retrieval directions, improved linear-probe classification, and competitive sound-source localization on ADE20K Sound. The central message is that fine-grained temporal alignment and disentangled objectives, rather than larger or more complex models, drive better audio-visual representations.

What carries the argument

The load-bearing machinery is the frame-to-spectrogram alignment that creates fine-grained audio tokens: a linear map from frame index $i$ to spectrogram center $s_{\text{center}_i} = \lfloor iS/T \rfloor$ with a fixed window length, turning audio into a sequence of 16 temporally aligned instances instead of one global clip. Around this sit two architectural additions: dedicated global tokens $g_v$ and $g_a$, which are the only tokens used in the contrastive loss and in downstream retrieval, and 8 learnable register tokens appended to the patch sequences. Retrieval uses the diagonal-mean similarity of Eq. (5), averaging only the temporally matching entries of the token similarity matrix. Together these components let the reconstruction objective and the contrastive alignment objective operate in separate representational spaces rather than competing within a single token set.

What would settle it

Construct a retrieval or localization test set from videos with a known temporal offset between audio and visual tracks (for example, dubbed or delayed audio) or with off-screen sound sources, and measure R@1 under the diagonal scoring rule; if the model's advantage over global-audio representations disappears or reverses on such data, the linear synchronization assumption is the limiting factor.

Watch

Extended reading notes

Core claim

CAV-MAE Sync claims that a contrastive masked autoencoder learns a balanced joint embedding space when audio is treated as a temporal sequence of instances aligned with video frames rather than as a global representation. For each of 16 uniformly sampled frames, the model extracts a 4-second spectrogram window centered at $s_{\text{center}_i} = \lfloor iS/T \rfloor$, producing 16 audio tokens that correspond one-to-one with visual frames. A dedicated global token per modality carries the contrastive objective, while 8 learnable register tokens are appended to the patch sequences to reduce semantic load on patch tokens, which remain responsible for masked reconstruction. At inference, retrieval similarity between a query video and a target video is computed as the mean of the diagonal of the frame-token similarity matrix, so that only temporally corresponding audio-visual pairs contribute to the score. The paper reports that this achieves state-of-the-art zero-shot retrieval (AudioSet R@1 35.2 for visual-to-audio and 27.9 for audio-to-visual; VGGSound 27.9 and 23.2), 30.5 mAP on AudioSet-20K linear probing, and 22.7 mIoU on ADE20K Sound localization, while using fewer audio tokens than the original CAV-MAE.

Load-bearing premise

The fixed linear map from frame index to spectrogram center assumes every frame's correct audio counterpart is a 4-second window centered at the corresponding time point; if audio events are asynchronous with visual frames, off-screen, or span very different durations, the constructed positive pairs are misaligned and the contrastive objective is corrupted.

Editorial extensions

If this is right

  • Zero-shot cross-modal retrieval improves in both directions on AudioSet and VGGSound, indicating that the learned embedding space is balanced rather than dominated by one modality.
  • The same self-supervised backbone supports classification, retrieval, and sound-source localization without task-specific pretraining objectives, because the global token carries contrastive information while patch and register tokens preserve spatial structure.
  • Shorter 3-4 second audio segments outperform the standard 10-second clip, cutting the pretraining audio token count by about 60-70 percent and reducing compute.
  • The diagonal-mean retrieval aggregation is worth roughly 2.7 R@1 points over block-mean aggregation on AudioSet, so temporal correspondence itself is a usable retrieval signal.
  • Increasing the number of register tokens improves localization in the tested range, supporting their role as a buffer between local patch features and the global contrastive representation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the fixed linear alignment could be replaced by a learned or event-driven alignment mechanism, such as soft attention between frame tokens and audio segments, to handle non-linear audio-visual dynamics; the paper's mapping is a natural first step rather than the endpoint.
  • I infer that the diagonal-scoring retrieval rule could be generalized to soft assignment or optimal-transport matching between frame tokens and audio tokens, preserving temporal sensitivity while tolerating small asynchronies.
  • I infer that the global-token, register-token, patch-token separation is a general recipe for multi-task masked autoencoders beyond audio-visual learning, since it decouples generative reconstruction from discriminative alignment without adding parameters.
  • I infer that a direct stress test on dubbed or re-synchronized videos would quantify how much of the reported gain depends on true audio-visual synchrony rather than on per-video semantic similarity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CAV-MAE Sync, an extension of the CAV-MAE contrastive masked autoencoder for audio-visual representation learning. The method introduces three modifications: (1) representing audio as a temporal sequence of 4-second segments aligned to video frames instead of using a single global audio representation, (2) adding dedicated global tokens for the contrastive loss to disentangle it from the reconstruction objective, and (3) incorporating learnable register tokens to reduce the semantic load on patch tokens. The model is pretrained on AudioSet-2M and evaluated on zero-shot audio-visual retrieval (AudioSet, VGGSound), linear-probe classification (AudioSet-20K, VGGSound), and sound-prompted segmentation (ADE20K Sound). The paper reports state-of-the-art retrieval results, strong classification performance, and competitive localization accuracy, with a controlled ablation chain in Table 4 showing progressive gains from the proposed components.

Significance. If the results hold under a uniform evaluation protocol, the paper offers a simple and effective recipe for improving contrastive audio-visual masked autoencoders: fine-grained temporal audio tokens, disentangled global tokens, and register tokens. The controlled ablation in Table 4 is a genuine strength, showing that each ingredient contributes under matched hyperparameters. The code is publicly released, which supports reproducibility. However, the headline claim of state-of-the-art zero-shot retrieval is currently undermined by a mismatch between the evaluation protocol used for the proposed model (diagonal-mean temporal similarity) and that used for the baselines (global-embedding cosine), as detailed below. The method's internal gains are credible, but the external comparisons need to be made fair before the SOTA claim can be accepted.

major comments (3)
  1. [§3.5.1, §4.2, Table 1] The evaluation protocol for retrieval is inconsistent between the proposed model and the baselines. Section 4.2 states that the authors 'follow the evaluation protocol and subsampling from CAV-MAE [14], using cosine similarity between embeddings to rank candidates,' but Section 3.5.1 defines a different scoring function for CAV-MAE Sync: it constructs a T×T similarity matrix between per-frame visual tokens and per-frame audio tokens and averages only the diagonal (Eq. 5). The external baselines in Table 1 (ImageBind, LanguageBind, AVSiam, CAV-MAE) were evaluated in their original papers using a single global embedding per video and ordinary cosine similarity. This is not a fair comparison, and the diagonal-mean temporal protocol can raise scores purely from temporal correspondence. Indeed, Table 4 shows that adding 16 temporal tokens plus diagonal similarity raises AudioSet V→A R@1 from 19.7 to 23.9 with no other architectural change. To support the claim in Section 4.3 that the model 'achieves state-of-the-art performance in both directions,' the authors must either (a) apply the same temporal-token/diagonal-mean protocol to all baselines or (b) report global-embedding retrieval for the proposed model, so that all methods are evaluated under the same protocol. Without this, the SOTA claim is not supported.
  2. [§3.3, Eq. (1)–(5)] The linear frame-to-spectrogram mapping scenter_i = floor(i·S/T) assumes that the semantic audio counterpart of each frame is a fixed-duration window centered at the corresponding linear time point. For videos with asynchronous, off-screen, or duration-mismatched audio events, the constructed positive pairs will be misaligned, which can corrupt the contrastive objective during pretraining and the diagonal-mean retrieval scores during evaluation. The manuscript does not validate this assumption on data with known temporal offsets or non-linear audio-visual structure. Please add a discussion of this limitation and, if possible, an experiment (e.g., using videos with annotated event boundaries) to quantify the sensitivity of the method to this alignment assumption.
  3. [Table 2 vs. Tables 6 and 7] The reported final-model classification numbers are inconsistent. Table 2 reports 30.5 mAP on AudioSet-20K for 'Ours,' but Tables 6 and 7 report 30.8 mAP for the configuration with 8 registers and the global token, which appears to be the same final configuration. Please reconcile these numbers or clarify whether the difference arises from a different model variant, random seed, or a typographical error.
minor comments (6)
  1. [§3.3] Typo: 'a rather loos contrastive objective' should be 'a rather loose contrastive objective.'
  2. [§4.1] The description of ADE20K Sound states it contains 106 images, but it would be helpful to clarify whether this is the full dataset or a subset, and to provide the number of audio clips and classes more precisely.
  3. [Table 1] The table header and caption do not clearly indicate that Ours uses a different retrieval scoring (diagonal mean) than the baselines. Please add a footnote in the table to avoid misleading readers.
  4. [§4.3] The sentence 'For baselines, we compare against state-of-the-art audio-visual models including CAV-MAE, ImageBind, AVSiam, and VAB' does not mention LanguageBind, which appears in Table 1. Please correct the list or the table.
  5. [§6.3] The supplementary lists batch size 8×64 (512) for pretraining, which is consistent with Table 4, but the main text should mention that the 512 batch size is used in the final model for clarity.
  6. [§8, Table 11] The aggregation comparison in Table 11 is informative, but it would be improved by reporting also the global-embedding retrieval (e.g., mean-pooling over temporal tokens) to directly address the protocol question raised in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are grounded in external benchmark evaluations and ablations, and self-citations to CAV-MAE are baseline/protocol references rather than load-bearing justifications.

full rationale

The paper's central claims (zero-shot retrieval R@1, linear-probe classification, sound-prompted segmentation) are evaluated against external datasets (AudioSet, VGGSound, ADE20K) and compared with independently published baselines. No predicted quantity is defined in terms of a fitted parameter or the model's own outputs: the retrieval score in Eq. 5 is computed from the model's global tokens on held-out test sets, and Table 11 reports that diagonal-mean aggregation outperforms block/max alternatives for the same model, indicating the metric reflects learned temporal alignment rather than being a forced identity. The linear mapping scenter_i = floor(i*S/T) is an input assumption about audio-visual correspondence, not a quantity the paper claims to predict; using the same temporal indexing for retrieval does not guarantee success on unseen videos, as the model must still learn meaningful cross-modal token similarities. Self-citations to CAV-MAE [14] supply the baseline architecture, initialization, and evaluation subsampling, but the contributions (temporal audio tokens, global tokens, register tokens) are ablated and tested independently. No uniqueness theorem, ansatz-by-citation, or renaming of known results is found; the diagonal-mean retrieval protocol is a task-specific evaluation choice, and any concern about comparability with baselines is a correctness/fairness issue rather than circularity. Hence no circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities; global tokens and register tokens are standard transformer mechanisms. The central results rest on several hyperparameters chosen by ablations on the evaluation subsets themselves, and on the domain assumption that linear frame-to-spectrogram alignment yields valid fine-grained audio-visual pairs.

free parameters (5)
  • audio segment length = 4 seconds
    Chosen from ablation in Figure 4; 3-4s optimal, 10s baseline worse. The final model uses 4s.
  • number of register tokens = 8
    Selected in Table 6 as balanced across retrieval, classification, and localization; 0 registers gives slightly higher AudioSet R@1 (35.5 vs 35.2).
  • masking ratio = 0.75
    Selected in Table 8 as a trade-off between retrieval (best at 0.6) and localization (best at 0.9).
  • contrastive loss weight lambda_c = 0.1
    Set higher than CAV-MAE's 0.01; justified by the more challenging fine-grained alignment task, but no systematic ablation is provided.
  • number of frames and audio segments T = 16
    Selected in Table 5; 16/16 gives the best retrieval results.
assumptions (4)
  • domain assumption Linear frame-to-spectrogram mapping scenter_i = floor(i*S/T) correctly aligns audio segments to visual frames.
    Used in Sec 3.3 to construct positive training pairs and in Eq. 5 for retrieval scoring; assumes constant frame rate and linear time correspondence, with no handling of asynchronous or off-screen audio events.
  • domain assumption Contrastive and reconstruction losses from CAV-MAE are effective learning objectives and their disentanglement improves representations.
    Assumed throughout; the paper argues and ablates the architecture but does not prove that the disentanglement, rather than other confounds, is the cause of the gains.
  • ad hoc to paper Register tokens behave in the audio-visual joint layer as they do in vision transformers [10].
    Assumed without direct verification of the high-norm artifact claim; in Sec 9 the authors note they do not observe high-norm tokens, so the mechanism transferring to the AV setting is not established.
  • domain assumption Diagonal-mean scoring in Eq. 5 is a valid retrieval metric that rewards temporal alignment.
    Used for downstream retrieval; assumes matched videos have similar temporal ordering of events so that diagonal token pairs are the semantically correct matches.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment." pith.science (2026). https://pith.science/paper/UYQ3SVH5

@misc{pith2026250501237,
  author       = {Pith},
  title        = {Pith review of: CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYQ3SVH5}},
  note         = {Machine review of arXiv:2505.01237}
}
read the original abstract

Recent advances in audio-visual learning have shown promising results in learning representations across modalities. However, most approaches rely on global audio representations that fail to capture fine-grained temporal correspondences with visual frames. Additionally, existing methods often struggle with conflicting optimization objectives when trying to jointly learn reconstruction and cross-modal alignment. In this work, we propose CAV-MAE Sync as a simple yet effective extension of the original CAV-MAE framework for self-supervised audio-visual learning. We address three key challenges: First, we tackle the granularity mismatch between modalities by treating audio as a temporal sequence aligned with video frames, rather than using global representations. Second, we resolve conflicting optimization goals by separating contrastive and reconstruction objectives through dedicated global tokens. Third, we improve spatial localization by introducing learnable register tokens that reduce semantic load on patch tokens. We evaluate the proposed approach on AudioSet, VGG Sound, and the ADE20K Sound dataset on zero-shot retrieval, classification and localization tasks demonstrating state-of-the-art performance and outperforming more complex architectures.

Figures

Figures reproduced from arXiv: 2505.01237 by the authors.

Figure 1
Figure 1. By representing audio with multiple finer-grained repre [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our approach. Our model processes video frames and audio segments in parallel through separate encoders [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of our downstream tasks: (1) Classification: [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Impact of audio segment length on model performance. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Sound-prompted segmentation results showing localization maps generated from audio prompts from VGGSound classes like [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Temporal segmentation results showing audio-visual event boundaries across different scenarios. Each row shows frame se [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 45 canonical work pages

  1. [14]

    Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R

    Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representa- tions, 2023. 1, 2, 3, 4, 6, 7

    work page 2023

  2. [1]

    Self-supervised learning of audio-visual objects from video

    Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pages 208–224. Springer, 2020. 1

    2020

  3. [2]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In IEEE International Conference on Computer Vi- sion, pages 609–617, 2017. 1, 2

    work page 2017

  4. [3]

    Objects that sound

    Relja Arandjelovic and Andrew Zisserman. Objects that sound. In European Conference on Computer Vision, pages 435–451, 2018. 2, 3

    work page 2018

  5. [4]

    Sound- net: Learning sound representations from unlabeled video

    Yusuf Aytar, Carl V ondrick, and Antonio Torralba. Sound- net: Learning sound representations from unlabeled video. Advances in Neural Information Processing Systems , 29,

    work page

  6. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 3

    2021

  7. [6]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zis- serman. Vggsound: A large-scale audio-visual dataset. In ICASSP, pages 721–725, 2020. 2, 6, 1

    work page 2020

  8. [7]

    Localiz- ing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Na- grani, Andrea Vedaldi, and Andrew Zisserman. Localiz- ing visual sounds the hard way. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16867– 16876, 2021. 3

    work page 2021

Show all 50 references
  1. [8]

    Distilling audio-visual knowledge by com- positional contrastive learning

    Yanbei Chen, Yongqin Xian, A Koepke, Ying Shan, and Zeynep Akata. Distilling audio-visual knowledge by com- positional contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7016–7025, 2021. 1

    2021

  2. [9]

    Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning

    Ying Cheng, Ruize Wang, Zhihao Pan, Rui Feng, and Yue- jie Zhang. Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3884–3892, 2020. 1

    2020

  3. [10]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representa- tions, 2024. 2, 5, 8

    2024

  4. [11]

    Audio set: An ontology and human- labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In ICASSP, pages 776–780,

  5. [12]

    Audiovisual masked autoencoders

    Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab. Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16144–16154, 2023. 2, 4

    2023

  6. [13]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 3, 6, 7

    2023

  7. [15]

    Cross- mae: Cross-modality masked autoencoders for region-aware audio-visual pre-training

    Yuxin Guo, Siyang Sun, Shuailei Ma, Kecheng Zheng, Xi- aoyi Bao, Shijie Ma, Wei Zou, and Yun Zheng. Cross- mae: Cross-modality masked autoencoders for region-aware audio-visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2024

  8. [16]

    Separating the” chirp” from the” chat”: Self-supervised visual grounding of sound and language

    Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the” chirp” from the” chat”: Self-supervised visual grounding of sound and language. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 13117–13127, 20...

    2024

  9. [17]

    Jointly dis- covering visual objects and spoken words from raw sensory input

    David Harwath, Adria Recasens, D ´ıdac Sur ´ıs, Galen Chuang, Antonio Torralba, and James Glass. Jointly dis- covering visual objects and spoken words from raw sensory input. In Proceedings of the European conference on com- puter vision (ECCV), pages 649–665, 2018. 3, 7

    2018

  10. [18]

    Multi- modal attention for fusion of audio and spatiotemporal fea- tures for video description

    Chiori Hori, Takaaki Hori, Gordon Wichern, Jue Wang, Teng-Yok Lee, Anoop Cherian, and Tim K Marks. Multi- modal attention for fusion of audio and spatiotemporal fea- tures for video description. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition...

    2018

  11. [19]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29...

    2021

  12. [20]

    Mavil: Masked audio-video learners

    Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al. Mavil: Masked audio-video learners. Advances in Neural Information Processing Sys- tems, 36, 2024. 1, 2, 7

    2024

  13. [21]

    EquiA V: Leveraging Equivari- ance for Audio-Visual Contrastive Learning

    Jongsuk Kim, Hyeongkeun Lee, Kyeongha Rho, Junmo Kim, and Joon Son Chung. EquiA V: Leveraging Equivari- ance for Audio-Visual Contrastive Learning. InICML, pages 24327–24341, 2024. 3

    2024

  14. [22]

    Coopera- tive learning of audio and video models from self-supervised synchronization

    Bruno Korbar, Du Tran, and Lorenzo Torresani. Coopera- tive learning of audio and video models from self-supervised synchronization. Advances in Neural Information Process- ing Systems, 31, 2018. 2

    2018

  15. [23]

    Cross-attentional audio-visual fusion for weakly- supervised action localization

    Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross-attentional audio-visual fusion for weakly- supervised action localization. In International conference on learning representations, 2020. 1

    2020

  16. [24]

    Siamese vision transform- ers are scalable audio-visual learners

    Yan-Bo Lin and Gedas Bertasius. Siamese vision transform- ers are scalable audio-visual learners. In ECCV, 2024. 1, 2, 6, 8

    2024

  17. [25]

    Vision transformers are parameter-efficient audio- visual learners

    Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers are parameter-efficient audio- visual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2299– 2309, 2023. 1

    2023

  18. [26]

    Active contrastive learning of audio-visual video representa- tions

    Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video representa- tions. In International Conference on Learning Representa- tions, 2020. 3

    2020

  19. [27]

    Active contrastive learning of audio-visual video representa- tions

    Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video representa- tions. In International Conference on Learning Representa- tions, 2021. 1

    2021

  20. [28]

    Robust audio-visual instance discrimination

    Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimination. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 12934–12945, 2021. 3

    2021

  21. [29]

    Audio- visual instance discrimination with cross-modal agreement

    Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio- visual instance discrimination with cross-modal agreement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475–12486, 2021. 2

    2021

  22. [30]

    Audio-visual scene analysis with self-supervised multisensory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. InEuro- pean Conference on Computer Vision, pages 631–648, 2018. 2

    2018

  23. [31]

    Ambient sound provides supervision for visual learning

    Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In European Conference on Computer Vision, pages 801–816, 2016. 2

    2016

  24. [32]

    On compositions of transformations in contrastive self-supervised learning

    Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, Jo ˜ao F Henriques, Geoffrey Zweig, and Andrea Vedaldi. On compositions of transformations in contrastive self-supervised learning. In IEEE/CVF International Con- ference on Computer Vision, pages 9577–9587, 2021. 3

    2021

  25. [33]

    Broaden your views for self-supervised video learning

    Adria Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica P ˘atr˘aucean, Florent Altch ´e, Michal Valko, et al. Broaden your views for self-supervised video learning. In IEEE/CVF International Conference on Compu...

    2021

  26. [34]

    Avlnet: Learning audio-visual language representa- tions from instructional videos

    Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Au- dhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, et al. Avlnet: Learning audio-visual language representa- tions from instructional videos. In Interspeech, 2021. 1, 2

    2021

  27. [35]

    Self-supervised audio- visual representation learning with relaxed cross-modal syn- chronicity

    Pritam Sarkar and Ali Etemad. Self-supervised audio- visual representation learning with relaxed cross-modal syn- chronicity. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, pages 9723–9732, 2023. 3

    2023

  28. [36]

    Event-specific audio-visual fusion layers: A simple and new perspective on video understanding

    Arda Senocak, Junsik Kim, Tae-Hyun Oh, Dingzeyu Li, and In So Kweon. Event-specific audio-visual fusion layers: A simple and new perspective on video understanding. In Pro- ceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 2237–2247, 2023. 1

    2023

  29. [37]

    From vision to au- dio and beyond: A unified model for audio-visual representa- tion and generation

    Kun Su, Xiulong Liu, and Eli Shlizerman. From vision to au- dio and beyond: A unified model for audio-visual representa- tion and generation. InInternational Conference on Machine Learning, pages 46804–46822, 2024. 2, 6, 7

    2024

  30. [38]

    Learning audio-visual source localization via false negative aware contrastive learning

    Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning audio-visual source localization via false negative aware contrastive learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision...

    2023

  31. [39]

    Multimodal self- supervised learning of general audio representations

    Luyu Wang, Pauline Luc, Adria Recasens, Jean-Baptiste Alayrac, and Aaron van den Oord. Multimodal self- supervised learning of general audio representations. arXiv preprint arXiv:2104.12807, 2021. 3

    2021 arXiv

  32. [40]

    Temporal cue guided video highlight detection with low-rank audio-visual fusion

    Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang. Temporal cue guided video highlight detection with low-rank audio-visual fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 7950–7959, 2021. 1

    2021

  33. [41]

    Con- trastive learning of global and local video representations

    Zhaoyang Zeng, Daniel McDuff, Yale Song, et al. Con- trastive learning of global and local video representations. Advances in Neural Information Processing Systems , 34: 7025–7040, 2021. 3

    2021

  34. [42]

    The sound of pixels

    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl V on- drick, Josh McDermott, and Antonio Torralba. The sound of pixels. In European Conference on Computer Vision, pages 570–586, 2018. 1

    2018

  35. [43]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641,

  36. [44]

    Languagebind: Extending video-language pretraining to n- modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, W ANG HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zong- wei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining to n- modality by language-based semantic alignment. In Th...

    2024

  37. [45]

    Implementation Details In this section, we provide details on our data preprocessing, model architecture, and training hyperparameters. 6.1. Data Preprocessing For input, we sample 16 frames uniformly from each video, along with corresponding 4-second audio segments with tempo...

  38. [46]

    Modality-Specific Linear Probing Table 10 presents the results of our modality-specific linear probing experiments. We compare the performance of models trained with audio-only, video-only, and audio-visual inputs on Pretraining Probing Dataset AS-2M AS-20K VGG Optimizer Adam,...

  39. [47]

    di- agonal mean

    Retrieval Aggregation Methods We evaluate different strategies for aggregating similarity scores in cross-modal retrieval, as shown in Table 11. For any pair of videos, we compute a similarity matrix where each element represents the similarity between a visual token from the ...

  40. [48]

    Table 12 shows the performance comparison between register to- kens, patch tokens, and the global token

    Register Tokens Analysis In this section, we analyze the information captured by different token types through linear probing on the AudioSet-20k dataset. Table 12 shows the performance comparison between register to- kens, patch tokens, and the global token. Our findings reve...

  41. [49]

    writing on blackboard with chalk

    Sound Prompted Segmentation Examples Figure 5 shows our model’s sound-prompted segmentation re- sults. As described in Section 3, we compute localization maps by calculating cosine similarities between the global audio token and visual patch tokens. Using VGGSound audios from ...

  42. [50]

    For this experiment, we manually annotate the occurrence of the classes throughout the video

    Intra-Instance Temporal Segmentation To investigate how finer-grained audio representations impact the understanding of video clips, we conduct a qualitative analysis of temporal segmentation within samples from the AudioSet dataset. For this experiment, we manually annotate t...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.