{"id":"68311990-336c-4bec-90cf-53b05de3d486","arxiv_id":"2411.12270","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KDC-MAE pretrains an audio-video transformer with two complementary masks plus KL self-distillation and reports small, partly inconsistent accuracy gains over CAV-MAE.","lead":"This paper combines three self-supervised learning tricks (contrastive learning, masked reconstruction, and knowledge distillation) into one audio-visual model called KDC-MAE, which feeds two complementary masked views of the same clip through a shared encoder. The reported gains over the CAV-MAE baseline are small and inconsistent across tasks, so the paper is a modest empirical step rather than a breakthrough.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Complementary mask generation in §2.2 is arithmetically impossible as stated: two disjoint 384-token masks cannot be drawn from a 512-token set, so the central mechanism may never have been instantiated as described.","rationale":"The reader's weakest assumption concerns whether two disjoint visible subsets should produce identical pooled embeddings. That is a substantive behavioral assumption, but it presupposes that the complementary masks are well-defined. The paper's own definition of complementary masks is arithmetically inconsistent: with 75% masking and a 512-token audio stream, two disjoint sets of 384 unmasked tokens cannot exist. This is a more fundamental, load-bearing flaw because the proposed architecture, the KL distillation loss, and all 'No Overlap' experiments depend on this construction. If the authors instead meant disjoint visible sets of 25% each, the text must be corrected and the reported numbers re-verified. If they meant disjoint masked sets, the method cannot be instantiated at all. Either way, the central claim as written is not reproducible. This is not a mere wording issue: the impossibility arises directly from the sizes stated in the paper. The verdict should move to REJECT because the method description cannot support the experiments, unless the authors supply code or a corrected definition that resolves the contradiction. I do not claim the empirical result is false; I claim the paper, as written, does not describe a realizable procedure.","tokens_in":14986,"tokens_out":6034,"duration_ms":64878,"concrete_test":"Implement the mask-generation procedure implied by §2.2: for Ua = 512 and mr = 0.75, sample M1 of size 384 uniformly, then attempt to sample M2 of size 384 from Ua − M1. The second draw fails because Ua − M1 has only 128 elements. If the authors' actual code instead samples disjoint visible sets of size 128 (the only consistent interpretation), rerun Table 1 and Table 2 with that corrected definition and confirm that the reported gains over CAV-MAE remain. If the code cannot be provided, the paper's central claim is unsupported by its own method description.","verdict_should_be":"REJECT","load_bearing_attack":"Section 2.2 defines M1 and M2 as 'sets of unmasked tokens' with M1 ∩ M2 = ∅, and then requires |M1| = |M2| = mr · |Ua|. With mr = 0.75 and |Ua| = 512, this demands two disjoint subsets of size 384 inside a 512-element set, which is impossible because the residue Ua − M1 has only 128 elements. The same contradiction holds for video: |Uv| = 196, mr · |Uv| = 147, and the residue has 49 elements. The paper also states that 75% masking leaves 49/196 video and 128/512 audio patches as visible, i.e., 25% visible. Thus either 'mask' means the masked tokens, in which case two 75% masks cannot be disjoint since 1.5 × 512 > 512, or 'mask' means the visible tokens, in which case the size formula |M| = mr · |Ua| is off by a factor of three. The central contribution—complementary masking with self-distillation—is therefore not well defined from the text. Reproducing the described mask generation is impossible, and the reported 'No Overlap' results cannot be verified against the description. This is an internal inconsistency, not a disagreement with the field.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KDC-MAE, a self-supervised audio-visual pretraining method that combines a masked autoencoder reconstruction loss, a contrastive audio-visual loss, and a symmetric KL-divergence self-distillation loss between the joint embeddings of two masked views. The two views are generated by a proposed 'complementary masking' strategy intended to force the encoder to be mask-agnostic. The authors compare against CAV-MAE on VGG-Sound, AudioSet-20k, Kinetics-400, and ImageNet for classification, retrieval, inpainting, and localization, and report small accuracy gains in several configurations, which they attribute to the distillation objective and complementary masking.","tokens_in":15321,"tokens_out":6961,"duration_ms":66161,"significance":"If the central claims hold, the paper contributes a modest but potentially useful combination of the three main SSL paradigms for audio-visual representation learning, and it provides a broad set of ablations across datasets and downstream tasks. The breadth of experiments—including classification, retrieval, inpainting, and localization, and comparisons with adaptive masking and three-stream variants—is a strength. However, the manuscript is currently not solidly supported: the complementary-mask construction as written is arithmetically impossible, several tables contain mutually inconsistent numbers for the same configurations, and some written conclusions contradict the paper's own results. The claimed 'modular correspondence' is enforced by the loss by construction rather than discovered, so that interpretive claim needs to be reframed.","major_comments":[{"comment":"The complementary mask construction is arithmetically inconsistent as stated. The text defines M1 and M2 as sets of unmasked tokens with M1 ∩ M2 = ∅, then requires |M1| = |M2| = mr · |Ua| with mr = 0.75 and |Ua| = 512; two disjoint subsets of size 384 cannot both fit in a 512-element set, because the residue Ua − M1 has only 128 elements. The same contradiction holds for video with |Uv| = 196 and mr · |Uv| = 147. If 'mask' instead means the masked tokens, then the two 75% masks necessarily overlap by construction. The paper states that 75% masking leaves 128 audio and 49 video visible patches, so the size formula appears to be off by a factor of three. This makes the proposed 'No Overlap' mechanism undefined and prevents reproduction of the central contribution.","section":"Section 2.2"},{"comment":"The claim that the encoder learns 'modular correspondence' from the distillation loss is not independently supported. Lkd(p1,p2) is the symmetric KL divergence between the pooled joint embeddings of the two masked views and is minimized by making the two embeddings agree. Agreement between views is therefore enforced by construction and cannot be cited as evidence that a 'mutual ground truth' was discovered. The downstream finetuning results use independent labels and provide some grounding for the accuracy claim, but the interpretive conclusion about modular correspondence should be either removed or tested by probing the learned representations on tasks that do not optimize the same agreement objective.","section":"Section 2.3, Eq. (5)"},{"comment":"The hyperparameter and model selection procedure raises fair-comparison concerns. λkd is swept on the same VGG-Sound and AudioSet-20k finetuning sets used to report the final results, and the final configuration is chosen as the best in that sweep; no held-out validation is described. In addition, the row 'Proposed No Overlap Dual Mask λkd=5' gives better AS-20k A-V accuracy (41.62) than the selected λkd=10 row (41.03), so the selection rule is unclear. Please specify how many seeds were run, whether the test sets were used for selection, and why λkd=10 is the headline configuration.","section":"Section 3.3, Table 2"},{"comment":"The numbers for the same configurations are not consistent across tables. Table 1 reports CAV-MAE on AS-20k FT A-V as 39.61 and AS-20k FT V-only as 31.00, while Table 5 reports 37.02 and 28.86 for the same CAV-MAE AS-20k rows. Table 5 also shows no-overlap dual mask AS-20k (A only) at 37.01, below CAV-MAE at 37.21, and Kinetics-400 (A V) with overlap at 71.53, below CAV-MAE at 71.55, which contradicts the sentence 'classification accuracy only improves when using the proposed dual mask method in all datasets.' Please reconcile these numbers and adjust the claims accordingly.","section":"Tables 1 and 5; Section 3.3"},{"comment":"The abstract's claim of 'better learning ... over multiple tasks' is not supported by Table 6 for the headline 'No Overlap' configuration. Compared with CAV-MAE, No Overlap is worse on audio→visual retrieval (R@1 0.0506 vs 0.0580), visual→audio retrieval (R@1 0.0416 vs 0.0662), and visual sound source localization (Avg Cos Sim 0.2798 vs 0.2884); only inpainting loss improves. The sentence 'performs better in the case of audio-to-visual retrieval' is true only for the 'With Overlap' row. Please report the results for the final configuration consistently and temper the task-level claims.","section":"Table 6"},{"comment":"The statement that Lc and Lkd affect only 'later two' components is inconsistent with Eq. (1), which computes the contrastive loss on mean-pooled outputs of the modality-specific encoders (cv_i, ca_j). If Lc and Lkd intentionally stop gradients before the audio/video encoders, that architectural detail is missing; if not, the sentence is wrong. This affects which modules are trained by each objective and is important for reproducing the method.","section":"Section 2.4"}],"minor_comments":[{"comment":"The text contains 'duel head MAE', which should be 'dual head MAE'.","section":"Section 2.2"},{"comment":"The text contains an unresolved 'refer Fig. ??' reference; Figure 2 is presumably intended.","section":"Section 2.3"},{"comment":"The dataset name is misspelled as 'ILSRVC' and should be 'ILSVRC'; capitalization of 'VGGsound' is also inconsistent.","section":"Section 3.1"},{"comment":"The definitions of overlapped and non-overlapped masks give the same condition for both: 'Mi ∩ Mj = φ ∀ i,j, i≠j' appears twice; the overlapping case should presumably have Mi ∩ Mj ≠ φ.","section":"Table 2 text"},{"comment":"No error bars or multiple-seed results are reported for the finetuning accuracy numbers, several of which differ by less than 0.5 percentage points; this should be addressed in the final version.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper does not provide code or checkpoints, which makes it hard to verify the experiments given the table inconsistencies. The central idea is plausible and the paper contains extensive ablations, but the manuscript is currently too internally inconsistent to accept. The arithmetic problem in the masking definition and the contradictory tables will need to be resolved before the claims can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know about KDC-MAE is that it tries to add a third pillar to CAV-MAE: self-distillation between two views that see disjoint subsets of tokens. The exact combination (complementary dual masks plus symmetric KL on joint embeddings) is new, and the authors did run a fairly wide ablation: overlap vs. no-overlap, different λkd values, triple masks, dynamic weights, and several downstream tasks. That is real work and it is not a rehash of prior papers.\n\nBut the central mechanism is not well defined. Section 2.2 defines M1 and M2 as sets of unmasked tokens, requires |M1| = |M2| = mr·|U|, and also requires them to be disjoint. With mr = 0.75 and |Ua| = 512, that demands two disjoint sets of 384 tokens inside a 512-element set—impossible. The same holds for video (147 + 147 > 196). The paper also states that 75% masking leaves 128 audio and 49 video tokens visible, which is 25% visible. So either the size formula is wrong or the definition of M is wrong. The “No Overlap” dual-mask results, which are the heart of the paper, cannot be reproduced from the description. That is a load-bearing internal inconsistency, not a minor typo.\n\nThe empirical record is also shaky. Several prose conclusions contradict the tables: the abstract says the method helps multiple tasks, but Table 1 shows the proposed method below CAV-MAE on both audio-only and video-only AudioSet-20k; Table 6 shows No Overlap worse than CAV-MAE on both retrieval directions; Table 5 shows No Overlap worse than CAV-MAE on AS-2M (V only). The best configuration in Table 2 is λkd=5 for AudioSet, but the paper claims λkd=10 is best without explanation. Hyperparameters are tuned on the same datasets used for evaluation, and there are no error bars anywhere. The gains that do appear are often sub-percentage-point.\n\nCredit where due: the authors are honest about some negative results (e.g., triple masks underperform), they cite the relevant prior work, and the idea of enforcing mask-agnostic joint embeddings is worth exploring. But as it stands, the paper is not reproducible and the evidence does not support the strong claims.\n\nI would send this to peer review—the idea is salvageable and a good referee could force the authors to fix the mask definitions and provide code—but I would not cite it in its current form. It belongs in a reading group as a case study in how a promising recipe can sink on internal contradictions and loose evaluation.","headline":"The paper has a good idea—complementary-mask self-distillation on CAV-MAE—but the mask recipe as written is arithmetically impossible, and the empirical support is weaker than the prose suggests.","tokens_in":15818,"tokens_out":3726,"would_cite":false,"duration_ms":39592,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding a symmetric KL self-distillation loss between two complementary masked views of the same audio-video clip improves joint audio-visual representation learning over the contrastive masked autoencoder baseline.","keywords":["self-supervised learning","masked autoencoder","contrastive learning","knowledge distillation","audio-visual representation learning","complementary masking","KL divergence"],"falsifier":"Train KDC-MAE with the same complementary masks and reconstruction and contrastive losses, but replace the distillation target with a fixed or randomly shuffled pooled embedding while keeping all hyperparameters identical. If downstream accuracy stays at the reported level, the KL term is not the cause of the gains; if accuracy falls back toward the CAV-MAE baseline, the mutual-agreement loss is doing the work.","tokens_in":14770,"feed_emoji":"🎬","tokens_out":6578,"duration_ms":60670,"temperature":0.7,"pith_summary":"The paper tries to establish that the three major self-supervised learning paradigms—contrastive learning, masked data modelling, and knowledge distillation—can be joined in one audio-visual encoder, and that the distillation piece adds something the other two do not provide. Its model, KDC-MAE, sends two complementary 25%-visible masks of the same clip through a weight-shared encoder and adds a symmetric KL divergence between the pooled joint embeddings, on top of the usual reconstruction and contrastive losses. On downstream classification this yields small but consistent gains over the CAV-MAE baseline on VGG-Sound and AudioSet-20k, and the paper also reports improvements on Kinetics-400 and ILSVRC plus several retrieval, inpainting, and localization measures. The authors' conclusion is that the distillation loss teaches 'modular correspondence': the encoder should produce nearly the same pooled code no matter which visible subset of the clip it sees.","feed_headline":"Knowledge-distilled masked autoencoder beats audio-video baseline","feed_subtitle":"A small KL self-distillation term lifts audio-video classification on VGG-Sound and AudioSet-20k","key_machinery":"The load-bearing object is the complementary dual-mask pair combined with a symmetric KL divergence on origin-corrected, mean-pooled joint embeddings. For each modality, two masks are generated so that the visible token sets are disjoint (M1 ∩ M2 = ∅) and each exposes 25% of the tokens at the 75% masking ratio; the two masked views pass through weight-shared audio, video, and joint encoders, and the loss Lkd = (D(p1||p2) + D(p2||p1))/2 is backpropagated only through the encoders. To make KL applicable, the pooled embedding is shifted by its minimum to be non-negative and then linearly normalized to a probability distribution, deliberately avoiding softmax. This mechanism is what gives the paper's 'modular correspondence': the same modality content should yield the same pooled code regardless of which visible subset is fed in.","core_discovery":"The central claim is that making the encoder mask-agnostic improves audio-visual representation learning. Concretely, an encoder trained so that two disjoint subsets of visible patches from the same clip yield almost identical pooled joint embeddings—enforced by a symmetric KL divergence propagated only through the encoder—learns representations that transfer better to downstream tasks than an encoder trained only with reconstruction and contrastive losses. Because the decoder is discarded for downstream use, the paper argues that the encoder alone should carry the invariance, and the two complementary views serve as mutual ground truth in the absence of labels. The reported finetuning results support the claim: 64.23% versus 63.89% on VGG-Sound audio-visual classification, 41.03% versus 39.61% on AudioSet-20k audio-visual classification, and further gains on Kinetics-400 and ILSVRC in several configurations.","pith_inferences":["A direct way to test whether the mutual-agreement objective is doing the work is to evaluate the pretrained encoder on heavily occluded or randomly cropped clips: if the distillation loss truly creates mask-agnostic embeddings, KDC-MAE should degrade less than CAV-MAE under such input corruption.","Because the KL loss operates on pooled embeddings, it could in principle encourage the encoder to discard patch-level information that pooling would wash out anyway; measuring performance on dense or patch-level downstream tasks, or removing the distillation term while keeping the dual-mask setup, would separate the effect of distillation from the effect of the extra augmented view.","The shift-and-normalize projection to a probability distribution is not theoretically justified in the paper; replacing it with softmax or a learned projection head is a natural ablation that would show whether the specific projection matters for the gains."],"forward_implications":["On audio-visual classification, KDC-MAE outperforms CAV-MAE on VGG-Sound (64.23% versus 63.89%) and on AudioSet-20k (41.03% versus 39.61%), and the paper reports gains in most audio-only and video-only finetuning configurations.","After pretraining on Kinetics-400, the no-overlap dual-mask model reaches 71.90% on Kinetics action recognition and 74.76% on ILSVRC classification, above the CAV-MAE baselines of 71.69% and 73.73%.","Complementary (non-overlapping) masks help more when the downstream annotation is audio-centric, while overlapping masks help more for video-centric annotation, because complementary masks expose more video tokens and can overfit when video is the label source.","Using three parallel masked streams degrades performance relative to two; the paper attributes this to overexposure of data and overfitting, so two streams are the recommended configuration.","On VGG-Sound, the paper reports that the proposed variants improve audio-to-visual retrieval and inpainting over CAV-MAE, while remaining competitive on localization; visual-to-audio retrieval metrics are mixed or lower."],"supporting_citations":[{"why":"Supplies the CAV-MAE baseline that KDC-MAE extends and the architecture and loss setup (reconstruction plus contrastive) that the paper builds on.","marker":"[16]"},{"why":"Establishes masked autoencoding and random masking as the reconstruction backbone the paper inherits.","marker":"[20]"},{"why":"Deep Mutual Learning is the self-distillation idea the paper adapts into a symmetric KL loss on complementary masked streams.","marker":"[65]"},{"why":"DINO is cited as the popular self-distillation model that the paper contrasts with its chosen mutual two-stream scheme.","marker":"[6]"},{"why":"Provides Kinetics-400, used for video-only and audio-visual finetuning experiments that support the claims.","marker":"[24]"},{"why":"Provides VGG-Sound, the main downstream classification and retrieval benchmark for the reported comparisons.","marker":"[8]"},{"why":"Provides AudioSet, used for large-scale pretraining and for the AudioSet-20k finetuning benchmark.","marker":"[14]"}],"fun_headline_variants":["KL self-distillation boosts masked autoencoder","Contrastive masking plus KL distillation sharpens AV learning","Encoder-only invariance improves audio-video transfer","Self-distilled masked AE edges out baselines on VGG-Sound","Knowledge-distilled MAE: better AV representations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that two disjoint 25%-visible views of the same clip should produce nearly identical pooled joint embeddings, and that enforcing this agreement during pretraining transfers to better performance on fully visible new clips.","fun_headline_variants_meta":{"raw":{"variants":["KL self-distillation boosts masked autoencoder","Contrastive masking plus KL distillation sharpens AV learning","Encoder-only invariance improves audio-video transfer","Self-distilled masked AE edges out baselines on VGG-Sound","Knowledge-distilled MAE: better AV representations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1311,"prompt_tokens":842,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":458,"tokens_out":469,"duration_ms":5448,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:44:35.903911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train KDC-MAE with the same complementary masks and reconstruction and contrastive losses, but replace the distillation target with a fixed or randomly shuffled pooled embedding while keeping all hyperparameters identical. If downstream accuracy stays at the reported level, the KL term is not the cause of the gains; if accuracy falls back toward the CAV-MAE baseline, the mutual-agreement loss is doing the work.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Establishes masked autoencoding and random masking as the reconstruction backbone the paper inherits."},{"cited_title":"Deep mutual learning","cited_arxiv_id":null,"evidence_quote":"Deep Mutual Learning is the self-distillation idea the paper adapts into a symmetric KL loss on complementary masked streams."},{"cited_title":"Emerg- ing properties in self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"DINO is cited as the popular self-distillation model that the paper contrasts with its chosen mutual two-stream scheme."},{"cited_title":"Vggsound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"Provides VGG-Sound, the main downstream classification and retrieval benchmark for the reported comparisons."},{"cited_title":"Audio set: An ontology and human- labeled dataset for audio events","cited_arxiv_id":null,"evidence_quote":"Provides AudioSet, used for large-scale pretraining and for the AudioSet-20k finetuning benchmark."}],"review_version":1}