{"id":"dfd46048-88b5-450b-b8af-a2810f91f547","arxiv_id":"2505.01237","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CAV-MAE Sync improves audio-visual representation learning by aligning fine-grained audio segments with video frames and separating contrastive and reconstruction objectives, achieving state-of-the-art zero-shot retrieval.","lead":"This paper extends CAV-MAE, a self-supervised audio-visual model, by aligning short audio segments to individual video frames, using separate global tokens for contrastive and reconstruction objectives, and adding register tokens. The resulting CAV-MAE Sync shows large gains on zero-shot retrieval and classification on AudioSet and VGGSound, and competitive sound-source localization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval SOTA claim rests on a protocol mismatch: Table 1 compares diagonal-mean temporal-token scoring (Eq. 5) against baselines using global-embedding cosine; uniform protocol needed before SOTA is supported.","rationale":"The reader's weakest-assumption (linear frame-to-spectrogram mapping in §3.3) is a modeling concern, but it is not the most load-bearing for the headline claim. The mapping is a standard uniform-time correspondence; the 4-second window is an empirical design choice, and the ablation in Fig. 4 shows it works. Off-screen or asynchronous audio is a known limitation of all audio-visual methods and does not invalidate the relative gains over the authors' own controlled baselines. The more serious issue is that the retrieval numbers in Table 1 are not computed under a common protocol. The paper explicitly says it follows CAV-MAE's protocol, yet §3.5.1 introduces a temporal diagonal-mean scoring that differs from the global-embedding cosine used by all cited baselines. This is an internal inconsistency and a direct threat to the SOTA claim. It is also testable: one can apply the same temporal-token protocol to baselines or global-embedding protocol to Ours. The paper's own Table 4 quantifies that the temporal-token protocol alone adds roughly 4 to 10 points, so the comparison is confounded. I therefore recommend keeping the CONDITIONAL verdict, with the condition being a uniform protocol evaluation. The reader's rationale already requested apples-to-apples comparison, so this is a partial agreement.","tokens_in":16558,"tokens_out":7557,"duration_ms":73534,"concrete_test":"Re-run zero-shot retrieval on the same AudioSet and VGGSound eval subsets with a uniform protocol. For each baseline that can produce per-frame/per-segment embeddings (ImageBind with its image encoder on the 16 frames and audio encoder on the 4-second segments; CAV-MAE with per-frame inputs), compute the same diagonal-mean similarity of Eq. 5 and report R@1/R@5/R@10. Conversely, evaluate CAV-MAE Sync using a single global token (e.g., CLS or mean-pooled temporal tokens) and ordinary cosine similarity. If the gap between Ours and ImageBind/CAV-MAE shrinks below significance or reverses under either uniform protocol, the state-of-the-art claim in §4.3 is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim in §4.3 rests on an evaluation protocol mismatch. Section 4.2 states that the authors \"follow the evaluation protocol and subsampling from CAV-MAE [14], using cosine similarity between embeddings to rank candidates,\" but §3.5.1 defines a different scoring function for CAV-MAE Sync: it computes a T×T similarity matrix between per-frame visual tokens and per-frame audio tokens and averages only the diagonal (Eq. 5). External baselines in Table 1 (ImageBind, LanguageBind, AVSiam, CAV-MAE) were evaluated with a single global embedding per video and ordinary cosine similarity, as reported in their original papers. The diagonal-mean temporal protocol is a more powerful matching rule that can raise scores purely from temporal correspondence, independent of the learned representation. Table 4 shows that adding 16 temporal tokens plus diagonal similarity to CAV-MAE Scale++* raises AudioSet V→A R@1 from 19.7 to 23.9 with no other architectural change, and Scale+++ reaches 30.1. The 35.2 R@1 for Ours is therefore partly attributable to the retrieval protocol, not solely to the proposed global and register tokens. Without applying the same temporal-token/diagonal-mean protocol to the external baselines, or reporting global-embedding retrieval for all methods, the claim that CAV-MAE Sync achieves state-of-the-art retrieval in a balanced joint embedding space is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAV-MAE Sync, an extension of the CAV-MAE contrastive masked autoencoder for audio-visual representation learning. The method introduces three modifications: (1) representing audio as a temporal sequence of 4-second segments aligned to video frames instead of using a single global audio representation, (2) adding dedicated global tokens for the contrastive loss to disentangle it from the reconstruction objective, and (3) incorporating learnable register tokens to reduce the semantic load on patch tokens. The model is pretrained on AudioSet-2M and evaluated on zero-shot audio-visual retrieval (AudioSet, VGGSound), linear-probe classification (AudioSet-20K, VGGSound), and sound-prompted segmentation (ADE20K Sound). The paper reports state-of-the-art retrieval results, strong classification performance, and competitive localization accuracy, with a controlled ablation chain in Table 4 showing progressive gains from the proposed components.","tokens_in":16809,"tokens_out":3315,"duration_ms":30626,"significance":"If the results hold under a uniform evaluation protocol, the paper offers a simple and effective recipe for improving contrastive audio-visual masked autoencoders: fine-grained temporal audio tokens, disentangled global tokens, and register tokens. The controlled ablation in Table 4 is a genuine strength, showing that each ingredient contributes under matched hyperparameters. The code is publicly released, which supports reproducibility. However, the headline claim of state-of-the-art zero-shot retrieval is currently undermined by a mismatch between the evaluation protocol used for the proposed model (diagonal-mean temporal similarity) and that used for the baselines (global-embedding cosine), as detailed below. The method's internal gains are credible, but the external comparisons need to be made fair before the SOTA claim can be accepted.","major_comments":[{"comment":"The evaluation protocol for retrieval is inconsistent between the proposed model and the baselines. Section 4.2 states that the authors 'follow the evaluation protocol and subsampling from CAV-MAE [14], using cosine similarity between embeddings to rank candidates,' but Section 3.5.1 defines a different scoring function for CAV-MAE Sync: it constructs a T×T similarity matrix between per-frame visual tokens and per-frame audio tokens and averages only the diagonal (Eq. 5). The external baselines in Table 1 (ImageBind, LanguageBind, AVSiam, CAV-MAE) were evaluated in their original papers using a single global embedding per video and ordinary cosine similarity. This is not a fair comparison, and the diagonal-mean temporal protocol can raise scores purely from temporal correspondence. Indeed, Table 4 shows that adding 16 temporal tokens plus diagonal similarity raises AudioSet V→A R@1 from 19.7 to 23.9 with no other architectural change. To support the claim in Section 4.3 that the model 'achieves state-of-the-art performance in both directions,' the authors must either (a) apply the same temporal-token/diagonal-mean protocol to all baselines or (b) report global-embedding retrieval for the proposed model, so that all methods are evaluated under the same protocol. Without this, the SOTA claim is not supported.","section":"§3.5.1, §4.2, Table 1"},{"comment":"The linear frame-to-spectrogram mapping scenter_i = floor(i·S/T) assumes that the semantic audio counterpart of each frame is a fixed-duration window centered at the corresponding linear time point. For videos with asynchronous, off-screen, or duration-mismatched audio events, the constructed positive pairs will be misaligned, which can corrupt the contrastive objective during pretraining and the diagonal-mean retrieval scores during evaluation. The manuscript does not validate this assumption on data with known temporal offsets or non-linear audio-visual structure. Please add a discussion of this limitation and, if possible, an experiment (e.g., using videos with annotated event boundaries) to quantify the sensitivity of the method to this alignment assumption.","section":"§3.3, Eq. (1)–(5)"},{"comment":"The reported final-model classification numbers are inconsistent. Table 2 reports 30.5 mAP on AudioSet-20K for 'Ours,' but Tables 6 and 7 report 30.8 mAP for the configuration with 8 registers and the global token, which appears to be the same final configuration. Please reconcile these numbers or clarify whether the difference arises from a different model variant, random seed, or a typographical error.","section":"Table 2 vs. Tables 6 and 7"}],"minor_comments":[{"comment":"Typo: 'a rather loos contrastive objective' should be 'a rather loose contrastive objective.'","section":"§3.3"},{"comment":"The description of ADE20K Sound states it contains 106 images, but it would be helpful to clarify whether this is the full dataset or a subset, and to provide the number of audio clips and classes more precisely.","section":"§4.1"},{"comment":"The table header and caption do not clearly indicate that Ours uses a different retrieval scoring (diagonal mean) than the baselines. Please add a footnote in the table to avoid misleading readers.","section":"Table 1"},{"comment":"The sentence 'For baselines, we compare against state-of-the-art audio-visual models including CAV-MAE, ImageBind, AVSiam, and VAB' does not mention LanguageBind, which appears in Table 1. Please correct the list or the table.","section":"§4.3"},{"comment":"The supplementary lists batch size 8×64 (512) for pretraining, which is consistent with Table 4, but the main text should mention that the 512 batch size is used in the final model for clarity.","section":"§6.3"},{"comment":"The aggregation comparison in Table 11 is informative, but it would be improved by reporting also the global-embedding retrieval (e.g., mean-pooling over temporal tokens) to directly address the protocol question raised in the major comments.","section":"§8, Table 11"}],"recommendation":"major_revision","confidential_remarks":"The core methodological contribution (temporal audio tokens, global tokens, registers) appears sound and the internal ablations are well controlled. The main obstacle is the unfair retrieval comparison in Table 1: the paper changes both the architecture and the retrieval protocol relative to all baselines, so the SOTA claim is not yet established. The authors should be asked to provide a uniform-protocol comparison. Also, the inconsistency between the AS20K mAP numbers in Table 2 (30.5) and Tables 6/7 (30.8) should be fixed. I would not reject the paper; with the protocol issue addressed, it could be a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a well-executed empirical extension of CAV-MAE, and the individual ingredients (per-frame audio segments, global tokens, register tokens) are sensible and clearly ablated. The paper's own Table 4 is the most honest part of it. But the headline SOTA retrieval claim in Section 4.3 does not survive contact with that same table: temporal tokens plus diagonal-mean similarity, with no other architectural change, lifts CAV-MAE Scale++ V→A R@1 from 19.7 to 23.9 on AudioSet, and scaling gets to 30.1. The final 35.2 comes partly from a retrieval protocol baselines never used. That doesn't make the architecture pointless, but it makes the state-of-the-art statement unsupported until the diagonal-mean protocol is applied to ImageBind, LanguageBind, AVSiam, etc., or global-embedding numbers are reported for all methods.\n\nWhat's genuinely new: the combination of frame-aligned audio segments with dedicated global tokens for contrastive loss, plus register tokens to keep patch tokens reconstruction-focused. The ablations in Tables 4–8 are careful and the progression from baseline to full model is controlled. I also like the supplementary analysis of token roles and the aggregation-strategy comparison; that is useful evidence that diagonal-mean is doing real work. Code and pretrained checkpoints are promised, which is creditable.\n\nSoft spots, in proportion: (1) the protocol mismatch is the load-bearing one, as above; (2) no standard deviations or seeds reported anywhere, and several hyperparameters (batch size, λ_c, masking ratio) are tuned on the eval subsets; (3) the linear frame-to-spectrogram mapping assumes a fixed 4-second window centered at each frame's timestep, which is reasonable for well-correlated AudioSet/VGGSound videos but is never stress-tested on asynchronous or off-screen audio; (4) the comparison to VAB-Encodec is labeled as fine-tuned upper bound, which is fine, but the table's formatting makes it look like a baseline. Minor.\n\nOverall: the architecture is worth building on and the ablations are a genuine contribution for anyone working on audio-visual retrieval. The SOTA claim needs a re-run with a uniform protocol. I'd send it to review—it will improve with a serious referee asking for apples-to-apples comparisons and variance estimates. Not a desk reject.","headline":"Well-executed CAV-MAE extension with solid ablations, but the state-of-the-art retrieval claim rests on a scoring protocol the baselines never got.","tokens_in":17442,"tokens_out":2084,"would_cite":true,"duration_ms":20170,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating audio as a sequence of frame-aligned clips instead of one global clip lifts contrastive audio-video learning to state-of-the-art zero-shot retrieval and competitive sound-source localization.","keywords":["audio-visual learning","contrastive learning","masked autoencoder","fine-grained temporal alignment","zero-shot retrieval","sound localization","register tokens","self-supervised representation learning"],"falsifier":"Construct a retrieval or localization test set from videos with a known temporal offset between audio and visual tracks (for example, dubbed or delayed audio) or with off-screen sound sources, and measure R@1 under the diagonal scoring rule; if the model's advantage over global-audio representations disappears or reverses on such data, the linear synchronization assumption is the limiting factor.","tokens_in":16295,"feed_emoji":"🎬","tokens_out":8986,"duration_ms":76468,"temperature":0.7,"pith_summary":"The paper argues that two weaknesses of contrastive audio-visual masked autoencoders can be fixed with small architectural changes: matching a whole audio clip to a single random frame is too loose a learning signal, and forcing one shared representation to serve both reconstruction and cross-modal alignment sets up conflicting objectives. Its solution, CAV-MAE Sync, represents audio as a temporal sequence of short segments aligned to individual video frames, uses dedicated global tokens for the contrastive loss while patch tokens focus on reconstruction, and adds register tokens that absorb excess semantic load. The paper reports state-of-the-art zero-shot retrieval on AudioSet and VGGSound in both retrieval directions, improved linear-probe classification, and competitive sound-source localization on ADE20K Sound. The central message is that fine-grained temporal alignment and disentangled objectives, rather than larger or more complex models, drive better audio-visual representations.","feed_headline":"Frame-aligned audio lifts zero-shot audio-video retrieval","feed_subtitle":"Sixteen aligned audio tokens and separate contrastive tokens lift zero-shot retrieval, classification, and localization.","key_machinery":"The load-bearing machinery is the frame-to-spectrogram alignment that creates fine-grained audio tokens: a linear map from frame index $i$ to spectrogram center $s_{\\text{center}_i} = \\lfloor iS/T \\rfloor$ with a fixed window length, turning audio into a sequence of 16 temporally aligned instances instead of one global clip. Around this sit two architectural additions: dedicated global tokens $g_v$ and $g_a$, which are the only tokens used in the contrastive loss and in downstream retrieval, and 8 learnable register tokens appended to the patch sequences. Retrieval uses the diagonal-mean similarity of Eq. (5), averaging only the temporally matching entries of the token similarity matrix. Together these components let the reconstruction objective and the contrastive alignment objective operate in separate representational spaces rather than competing within a single token set.","core_discovery":"CAV-MAE Sync claims that a contrastive masked autoencoder learns a balanced joint embedding space when audio is treated as a temporal sequence of instances aligned with video frames rather than as a global representation. For each of 16 uniformly sampled frames, the model extracts a 4-second spectrogram window centered at $s_{\\text{center}_i} = \\lfloor iS/T \\rfloor$, producing 16 audio tokens that correspond one-to-one with visual frames. A dedicated global token per modality carries the contrastive objective, while 8 learnable register tokens are appended to the patch sequences to reduce semantic load on patch tokens, which remain responsible for masked reconstruction. At inference, retrieval similarity between a query video and a target video is computed as the mean of the diagonal of the frame-token similarity matrix, so that only temporally corresponding audio-visual pairs contribute to the score. The paper reports that this achieves state-of-the-art zero-shot retrieval (AudioSet R@1 35.2 for visual-to-audio and 27.9 for audio-to-visual; VGGSound 27.9 and 23.2), 30.5 mAP on AudioSet-20K linear probing, and 22.7 mIoU on ADE20K Sound localization, while using fewer audio tokens than the original CAV-MAE.","pith_inferences":["I infer that the fixed linear alignment could be replaced by a learned or event-driven alignment mechanism, such as soft attention between frame tokens and audio segments, to handle non-linear audio-visual dynamics; the paper's mapping is a natural first step rather than the endpoint.","I infer that the diagonal-scoring retrieval rule could be generalized to soft assignment or optimal-transport matching between frame tokens and audio tokens, preserving temporal sensitivity while tolerating small asynchronies.","I infer that the global-token, register-token, patch-token separation is a general recipe for multi-task masked autoencoders beyond audio-visual learning, since it decouples generative reconstruction from discriminative alignment without adding parameters.","I infer that a direct stress test on dubbed or re-synchronized videos would quantify how much of the reported gain depends on true audio-visual synchrony rather than on per-video semantic similarity."],"forward_implications":["Zero-shot cross-modal retrieval improves in both directions on AudioSet and VGGSound, indicating that the learned embedding space is balanced rather than dominated by one modality.","The same self-supervised backbone supports classification, retrieval, and sound-source localization without task-specific pretraining objectives, because the global token carries contrastive information while patch and register tokens preserve spatial structure.","Shorter 3-4 second audio segments outperform the standard 10-second clip, cutting the pretraining audio token count by about 60-70 percent and reducing compute.","The diagonal-mean retrieval aggregation is worth roughly 2.7 R@1 points over block-mean aggregation on AudioSet, so temporal correspondence itself is a usable retrieval signal.","Increasing the number of register tokens improves localization in the tested range, supporting their role as a buffer between local patch features and the global contrastive representation."],"supporting_citations":[{"why":"It provides the base CAV-MAE framework and evaluation protocol that this work extends.","marker":"[14]"},{"why":"It introduces register tokens, the mechanism the paper adopts to reduce semantic load on patch tokens.","marker":"[10]"},{"why":"It serves as a strong zero-shot retrieval baseline on AudioSet and VGGSound that the model must beat.","marker":"[13]"},{"why":"It supplies the ADE20K Sound localization protocol and the DenseAV baseline for sound-prompted segmentation.","marker":"[16]"},{"why":"It provides the AudioSet-2M pretraining data and the AudioSet-20K evaluation split.","marker":"[11]"},{"why":"It provides the VGGSound dataset used for retrieval and classification evaluation.","marker":"[6]"},{"why":"It serves as a fine-tuned upper-bound reference for retrieval performance.","marker":"[37]"},{"why":"It provides the AVSiam CAV-MAE-family baseline that the paper compares against.","marker":"[24]"}],"fun_headline_variants":["Fine-grained audio-video alignment lifts zero-shot retrieval","Frame-aligned audio tokens improve audio-video contrastive learning","Separate contrastive tokens fix audio-video reconstruction conflict","CAV-MAE Sync: frame-level audio tokens beat global embeddings","Sixteen aligned audio tokens lift zero-shot audio-video benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fixed linear map from frame index to spectrogram center assumes every frame's correct audio counterpart is a 4-second window centered at the corresponding time point; if audio events are asynchronous with visual frames, off-screen, or span very different durations, the constructed positive pairs are misaligned and the contrastive objective is corrupted.","fun_headline_variants_meta":{"raw":{"variants":["Fine-grained audio-video alignment lifts zero-shot retrieval","Frame-aligned audio tokens improve audio-video contrastive learning","Separate contrastive tokens fix audio-video reconstruction conflict","CAV-MAE Sync: frame-level audio tokens beat global embeddings","Sixteen aligned audio tokens lift zero-shot audio-video benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2957,"prompt_tokens":1003,"completion_tokens":1954,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1870}},"tokens_in":619,"tokens_out":1954,"duration_ms":14618,"temperature":1.0,"reasoning_tokens":1870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:22:22.961764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a retrieval or localization test set from videos with a known temporal offset between audio and visual tracks (for example, dubbed or delayed audio) or with off-screen sound sources, and measure R@1 under the diagonal scoring rule; if the model's advantage over global-audio representations disappears or reverses on such data, the linear synchronization assumption is the limiting factor.","supporting_citations":[{"cited_title":"From vision to au- dio and beyond: A unified model for audio-visual representa- tion and generation","cited_arxiv_id":null,"evidence_quote":"It serves as a fine-tuned upper-bound reference for retrieval performance."},{"cited_title":"Siamese vision transform- ers are scalable audio-visual learners","cited_arxiv_id":null,"evidence_quote":"It provides the AVSiam CAV-MAE-family baseline that the paper compares against."},{"cited_title":"Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R","cited_arxiv_id":null,"evidence_quote":"It provides the base CAV-MAE framework and evaluation protocol that this work extends."},{"cited_title":"Vision transformers need registers","cited_arxiv_id":null,"evidence_quote":"It introduces register tokens, the mechanism the paper adopts to reduce semantic load on patch tokens."},{"cited_title":"Imagebind: One embedding space to bind them all","cited_arxiv_id":null,"evidence_quote":"It serves as a strong zero-shot retrieval baseline on AudioSet and VGGSound that the model must beat."},{"cited_title":"Separating the” chirp” from the” chat”: Self-supervised visual grounding of sound and language","cited_arxiv_id":null,"evidence_quote":"It supplies the ADE20K Sound localization protocol and the DenseAV baseline for sound-prompted segmentation."},{"cited_title":"Audio set: An ontology and human- labeled dataset for audio events","cited_arxiv_id":null,"evidence_quote":"It provides the AudioSet-2M pretraining data and the AudioSet-20K evaluation split."},{"cited_title":"Vggsound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"It provides the VGGSound dataset used for retrieval and classification evaluation."}],"review_version":1}