{"id":"307645cc-a11f-48b8-89bf-68572b51cb2a","arxiv_id":"2501.09608","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A self-distillation training scheme for audio-visual embeddings progressively replaces labeled triplets with model-generated soft alignments, improving cross-modal retrieval MAP by roughly 2 percent on AVE and VEGAS.","lead":"Researchers propose a training method for audio-visual retrieval that makes a model learn from its own predictions over time, using a mix of labeled and unlabeled samples in each batch. The method reports small gains over prior state-of-the-art on two benchmarks, but the paper as written has technical inconsistencies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1)'s soft-alignment construction is undefined as written: the stated matrix product has mismatched inner dimensions, so the central method cannot be executed from the paper.","rationale":"The paper's headline is empirical (2.13% and 1.82% MAP gains), but the claimed gains are only meaningful if the training objective is the one described. Eq. (1)-(3) are the only formal specification of that objective, and as printed they are not executable: the matrix product has mismatched inner dimensions, the indexing in Eq. (2) is inconsistent, and AA is undefined. This is an internal correctness risk, not a disagreement with consensus; a reader cannot tell whether the implementation used projected logits, raw features, or some other soft-label construction. The reader's teacher/student concern is real and closely related, but it presupposes that the teacher's soft labels can be produced at all; resolving Eq. (1) first is necessary. The lack of released code, seeds, and error bars means the dimension issue cannot be dismissed as a harmless typo from the paper alone. I would keep the CONDITIONAL verdict: the empirical claim is plausible and the ablations give some internal support, but acceptance should require a corrected, executable definition of Eq. (1)-(3) and ideally code that reproduces Table I. No change from the reader's verdict.","tokens_in":7192,"tokens_out":7949,"duration_ms":85685,"concrete_test":"Take N=2 and instantiate Eq. (1) literally with A-hat in R^{2 x 128} and V-hat in R^{2 x 1024}; the matrix multiply V-hat @ A-hat^T raises a shape error. The decisive check is to obtain the authors' code, or a forward-pass trace, and print the tensor shapes entering Eq. (1). If the shapes are not N x c for both modalities, the published description is not the implemented model. Independently, re-derive Eq. (2) on a two-sample batch using the text's column/row definitions; if the argmax comparisons are not between corresponding audio and visual instances, the positive/negative selection rule is undefined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central method claim is that soft-alignment labels defined in Eq. (1) are generated from output logits A-hat in R^{N x 128} and V-hat in R^{N x 1024} (Section III.A/B). The matrix product V-hat times A-hat^T is then an (N x 1024) times (128 x N) product, whose inner dimensions 1024 and 128 do not match; the equation is undefined as written. Eq. (2) is also internally inconsistent: it compares argmax(L^a_j) with argmax(L^v_i), while the text defines L^a_i as the i-th column and L^v_j as the j-th row. If the intended tensors are actually projections into the c-dimensional label space (c=15 for AVE, c=10 for VEGAS, Section IV.B), then Section III.A/B misstates their dimensions and never defines the projection. Since these soft labels select the positive and negative visual samples in the core triplet loss Eq. (3), the reported SOTA improvements cannot be attributed to the described algorithm until the forward pass is specified and executable. This is also logically prior to the teacher/student update ambiguity: before asking who updates whom, the equation that produces the teacher's labels must be well-defined. The AA proxy used in Eq. (3) is likewise only cited, not defined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a metric-learning method for audio-visual embedding that combines a cross-modal triplet loss with progressive self-distillation. In each batch, a subset N1 is trained with ground-truth labels, and the resulting model generates soft-alignment labels for the remaining subset N2; these soft labels are then used to construct soft cross-modal triplets that supervise the model on N2. The partition ratio r decreases during training so that the model gradually becomes its own teacher. The final objective combines a label-embedding loss, the soft cross-modal triplet loss, and a cross-modal dependency loss. Experiments on the AVE and VEGAS datasets report improvements over prior methods in average MAP, and ablation studies examine the effect of the dependency loss, the AA proxy, and the decay schedule.","tokens_in":7503,"tokens_out":4848,"duration_ms":50414,"significance":"The problem addressed is relevant: audio-visual metric learning is often limited by rigid label-based sampling, and using learned soft alignments to construct triplets is a plausible way to exploit distributional structure beyond labels. If the method were fully specified and reproducible, the reported consistent gains over the strong AADML baseline and the accompanying ablations would make a useful contribution. The paper also has strengths in experimental breadth: it compares against nine methods, reports results on two datasets, and includes ablation variants that isolate the contributions of ldis, self-distillation, the AA proxy, and the decay schedule. However, as written, the central formulation is not executable from the text because Eq. (1) is dimensionally inconsistent, Eq. (2) has an indexing inconsistency, the AA proxy in Eq. (3) is not defined, and the teacher-student gradient flow is unspecified. These issues are load-bearing for the claimed state-of-the-art results, so the contribution is currently conditional on a substantial rewrite of the method section.","major_comments":[{"comment":"The soft-alignment construction is undefined as written. The text states that the audio logits are A-hat in R^{N x 128} and the visual logits are V-hat in R^{N x 1024}. The first equation, La = sigma(V-hat * A-hat^T), is the product of an N x 1024 matrix with a 128 x N matrix, whose inner dimensions 1024 and 128 do not match. The analogous product Lv = sigma(A-hat * V-hat^T) is also invalid. The paper later says the projected features have dimension equal to the number of labels (c = 15 or c = 10), but Eq. (1) does not use such a projection and none is defined. Since these soft labels determine the positive and negative visual samples in the core triplet loss Eq. (3), the method cannot be implemented or evaluated from the paper as it stands.","section":"Section III.B, Eq. (1)"},{"comment":"The indexing in Eq. (2) is internally inconsistent with the surrounding text. The text says 'L^a_i refers to the i-th column of L^a and L^v_j corresponds to the j-th row of L^v', but the displayed equation uses argmax(L^a_j) and argmax(L^v_i). If L^a and L^v are N x N matrices, then the row/column convention determines which sample is the anchor and which is the candidate positive or negative; swapping the indices changes the meaning of the adjacency matrix. The notation must be made consistent, and the dimensions of L^a and L^v should be stated explicitly.","section":"Section III.B, Eq. (2)"},{"comment":"The teacher-student gradient flow is not specified. The paper says the teacher network generates soft-alignment labels for the unlabeled subset N2 and that these labels supervise the student, but it never states whether gradients from the student loss on N2 are stopped from updating the teacher, whether an EMA teacher is used, or whether the teacher and student share parameters. If the same network is updated by the student loss, the soft labels are simply the model's current predictions and the method reduces to self-training with pseudo-labels, which can reinforce errors. The authors should specify the gradient flow and, ideally, add an ablation that detaches the teacher's labels or uses a momentum teacher to demonstrate that the reported gains are not an artifact of this choice.","section":"Section III.C"},{"comment":"The AA(·) proxy is cited to reference [27] but is never defined. Since the triplet loss is computed on AA(a_i), AA(v_+_i), and AA(v_-_i), the representation used for distance computation is not specified. Without a definition of AA(·), Eq. (3) is incomplete and the reader cannot reproduce the training procedure or understand the contribution of the AA proxy beyond the citation.","section":"Section III.B, Eq. (3)"},{"comment":"The final loss includes llab = 1/n ||f_a(a_i) - Y(a_i)||_F + 1/n ||f_v(v_i) - Y(v_i)||_F, but the notation is ambiguous. It uses n instead of the batch size N used elsewhere, and it does not specify whether f_a(a_i) and f_v(v_i) are raw features, projected features, or output logits. The term Y(a_i) is called 'label representations' but is not defined; it is unclear whether this is a one-hot vector, a label embedding, or something else. Since this loss is part of the final objective that produces the reported results, the notation must be clarified.","section":"Section III.C, Eq. (4)"},{"comment":"The empirical central claim is based on improvements of 1.9-2.4 percentage points in MAP, but the paper reports no error bars, no number of runs, and no significance tests. Given that the gains over AADML are modest in absolute terms, the authors should report the mean and standard deviation over multiple seeds, or at least provide a clear statement that the table reports a single run. Without this information, the claim of state-of-the-art performance is not statistically supported.","section":"Section IV.C, Table I"}],"minor_comments":[{"comment":"The abstract contains the typo 'probabilistic aligns' where 'probabilistic alignments' is intended.","section":"Abstract"},{"comment":"The notation for the dataset is typeset awkwardly as '{(a_i, v_i)}n-1_i=0'; standard set notation would be clearer, e.g., D = {(a_i, v_i)}_{i=0}^{n-1}.","section":"Section III.A"},{"comment":"The implementation section states that the dimensionality of the projected features matches the number of labels, but it does not reconcile this with Eq. (1), where the logits are described as having dimensions 128 and 1024. This ambiguity should be resolved in the rewritten method section.","section":"Section IV.B"},{"comment":"The caption and text refer to comparisons of loss and MAP over epochs, but the figure is not described clearly; the reader cannot tell which curves correspond to which model or what metric is plotted on each axis.","section":"Section IV.D.1, Fig. 3"},{"comment":"Several references are formatted inconsistently (e.g., volume and page ranges are not uniform), and reference [30] is used for the data partitioning and feature extraction details, but the paper does not summarize those details in the text; a brief description would improve self-containedness.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's method section needs a substantial rewrite to make the equations executable; this is not a matter of polish. The authors may also consider releasing code or a supplementary appendix with exact tensor shapes and the training loop, since the current text leaves several load-bearing details unspecified. The manuscript is heavily based on the authors' prior AADML work; a more explicit statement of what is new relative to that work would help the editor assess novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper has a simple, plausible idea and the ablations are consistent, but the core equation is undefined as written, so I can't verify the method actually does what they claim.\n\nThe idea—generate soft-alignment labels from the model's own predictions and use them to form cross-modal triplets, with the labeled fraction shrinking over training—is a reasonable extension of AADML and self-training. The ablation table shows each piece (ldis, self-distillation, AA proxy) helps, and the step-wise schedule beats linear and cosine schedules. That's real work, and the comparison against nine baselines is thorough.\n\nThe problem is that Eq. (1) cannot be computed. They define A-hat in R^{N x 128} and V-hat in R^{N x 1024}, then set La = sigma(V-hat A-hat^T). The inner dimensions are 1024 and 128, which don't match. Same for Lv. So the soft labels are never actually defined. Eq. (2) also mixes up indices: the text says La_i is the i-th column but the equation uses argmax(La_j). And the teacher/student update rule is unspecified—does gradient flow from the student loss update the teacher? If so, this is just self-training with the model's own predictions, which can reinforce errors. There's also no code, no data release, and no error bars, so the 2% MAP gains over AADML are just point estimates.\n\nThis is a workshop-level draft. The idea might be salvageable, but as written it is not reproducible. I'd tell the authors to fix the equations, clarify the training dynamics, and ideally release code. A serious referee would spend most of the report asking for clarifications rather than evaluating the actual contribution.\n\nRecommendation: don't send to peer review yet. Desk reject with encouragement to resubmit after fixing.","headline":"Plausible incremental idea undone by an undefined core equation; needs a major rewrite before it can be evaluated.","tokens_in":7942,"tokens_out":2552,"would_cite":false,"duration_ms":24924,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that integrating cross-modal triplet loss with progressive self-distillation, where a model generates soft audio-visual alignment labels for the unlabeled part of each batch and then gradually relies on those labels…","keywords":["audio-visual cross-modal retrieval","metric learning","triplet loss","self-distillation","soft alignment labels","embedding learning","mean average precision","multimodal representation learning"],"falsifier":"Fix a copy of the teacher every few epochs so that the student's loss on unlabeled samples never changes the network generating the soft labels. If average MAP stays at 0.908 on AVE and 0.914 on VEGAS, the progressive self-distillation story is supported; if it falls back to the previous best of 0.887 and 0.896, the reported gain comes from reusing the model's own current predictions rather than from a genuine teacher-student separation.","tokens_in":7022,"feed_emoji":"🎵","tokens_out":6697,"duration_ms":60986,"temperature":0.7,"pith_summary":"This paper tries to establish that label-guided metric learning for audio-visual retrieval leaves latent distributional relationships unused, and that a model can exploit them by distilling its own knowledge into soft audio-visual alignment labels. In each batch, part of the data is trained with ground-truth labels; the model then generates probabilistic soft labels for the rest, constructs soft cross-modal triplets from those labels, and progressively shrinks the labeled fraction so the model becomes its own teacher. The authors report that this raises mean average precision to 0.908 on AVE and 0.914 on VEGAS, outperforming the previous best method by 2.13% and 1.82% on average MAP. A sympathetic reader would care because it suggests a way to relax annotation dependence in multimodal metric learning without adding a separate teacher network.","feed_headline":"Self-distillation lifts audio-visual retrieval MAP by 2.13%","feed_subtitle":"The model becomes its own teacher, refining soft audio-visual alignments and beating prior methods on AVE and VEGAS.","key_machinery":"The load-bearing object is the soft-alignment label: a probability distribution over the other modality computed as the softmax of a cross-modal similarity matrix (Equations 1-2), which replaces hard positive and negative choices with probabilistic ones. The cross-modal triplet loss in Equation 3 uses these soft labels through an adjacency matrix and its complement, and it is combined with the AA anchor-aware proxy from the baseline method. The progressive schedule on the partition ratio $r$, which shrinks the labeled subset from all samples to 20% of each batch, is what converts the network into its own teacher. These pieces together turn fixed annotated categories into a continuously refined soft structure between audio and visual samples.","core_discovery":"The authors' central claim is that probabilistic soft alignments between audio and visual samples, generated by the model itself from a labeled subset of each batch, carry information beyond the fixed labels and improve cross-modal retrieval. The teacher network produces label distributions $L_a = \\sigma(\\hat{V}\\hat{A}^T)$ and $L_v = \\sigma(\\hat{A}\\hat{V}^T)$; these distributions are converted into an adjacency matrix $A$ and its complement $\\bar{A}$ that decide which visual samples act as soft positives and soft negatives for each audio anchor. Those soft triplets are optimized with the AA proxy of the anchor-aware metric learning baseline, while an additional loss pulls each audio-visual pair together and a label-space loss keeps the projected features aligned with label representations. The proportion $r$ of labeled samples per batch decreases stepwise from 1.0 to 0.2, so the network gradually supervises itself on a larger fraction of each batch. On the AVE and VEGAS benchmarks the resulting MAP values are 0.908 and 0.914 on average, exceeding the previous best values of 0.887 and 0.896.","pith_inferences":["A natural extension is to apply the same progressive self-distillation to other modality pairs, such as image-text retrieval, since the soft-alignment construction does not depend on audio- or visual-specific details.","The paper does not report a variant that stops gradients from the student loss before they reach the teacher that generates the soft labels; testing such a variant would clarify whether the gain comes from self-distillation or from using the model's current predictions as smoothed targets.","Because the soft labels are produced by a softmax over the whole batch, the method's behavior may change with batch size; sweeping batch size could reveal whether the improvement is partly a high-batch-size effect."],"forward_implications":["On AVE, average MAP rises from 0.887 to 0.908, an improvement of 2.13% over the previous best method.","On VEGAS, average MAP rises from 0.896 to 0.914, an improvement of 1.82% over the previous best method.","Ablations show that all three components matter: removing the cross-modal dependency loss, the self-distillation, or the AA proxy each lowers average MAP on AVE.","The stepwise decay of the labeled fraction outperforms both linear and cosine-annealing schedules for the partition ratio."],"supporting_citations":[{"why":"Supplies the AA anchor-aware proxy used in the triplet loss and serves as the main baseline the method improves on.","marker":"[27]"},{"why":"Introduces the triplet loss formulation that the cross-modal triplet builds on.","marker":"[6]"},{"why":"Provides the self-knowledge distillation idea of a network training itself.","marker":"[3]"},{"why":"Baseline cross-modal triplet method in label space that the paper compares against.","marker":"[28]"},{"why":"Baseline two-stage triplet training with curriculum augmentation for audio-visual retrieval.","marker":"[25]"},{"why":"Baseline using explicit and implicit dual common subspaces for audio-visual retrieval.","marker":"[29]"},{"why":"Provides the AVE dataset used for evaluation.","marker":"[18]"},{"why":"Provides the VEGAS dataset used for evaluation.","marker":"[32]"}],"fun_headline_variants":["Progressive self-distillation boosts audio-visual retrieval MAP","Soft audio-visual alignments via self-distillation set new SOTA on AVE and VEGAS","Model-teaches-itself method tops audio-visual retrieval benchmarks","Progressive self-distillation refines soft alignments for audio-visual embedding","Self-distilled soft triplets improve audio-visual retrieval by 2.13%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the same network's current predictions are reliable enough to supervise itself on the unlabeled part of each batch, and the paper does not say whether the training signal from those unlabeled samples is blocked from updating the teacher that produced the soft labels.","fun_headline_variants_meta":{"raw":{"variants":["Progressive self-distillation boosts audio-visual retrieval MAP","Soft audio-visual alignments via self-distillation set new SOTA on AVE and VEGAS","Model-teaches-itself method tops audio-visual retrieval benchmarks","Progressive self-distillation refines soft alignments for audio-visual embedding","Self-distilled soft triplets improve audio-visual retrieval by 2.13%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1621,"prompt_tokens":945,"completion_tokens":676,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":573}},"tokens_in":561,"tokens_out":676,"duration_ms":6579,"temperature":1.0,"reasoning_tokens":573,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:50:07.091042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix a copy of the teacher every few epochs so that the student's loss on unlabeled samples never changes the network generating the soft labels. If average MAP stays at 0.908 on AVE and 0.914 on VEGAS, the progressive self-distillation story is supported; if it falls back to the previous best of 0.887 and 0.896, the reported gain comes from reusing the model's own current predictions rather than from a genuine teacher-student separation.","supporting_citations":[{"cited_title":"Anchor-aware deep metric learning for audio-visual retrieval","cited_arxiv_id":null,"evidence_quote":"Supplies the AA anchor-aware proxy used in the triplet loss and serves as the main baseline the method improves on."},{"cited_title":"Born again neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the self-knowledge distillation idea of a network training itself."},{"cited_title":"Com- plete cross-triplet loss in label space for audio-visual cross-modal retrieval","cited_arxiv_id":null,"evidence_quote":"Baseline cross-modal triplet method in label space that the paper compares against."},{"cited_title":"Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval","cited_arxiv_id":"2310.13451","evidence_quote":"Baseline two-stage triplet training with curriculum augmentation for audio-visual retrieval."},{"cited_title":"Learning explicit and implicit dual common subspaces for audio-visual cross-modal retrieval","cited_arxiv_id":null,"evidence_quote":"Baseline using explicit and implicit dual common subspaces for audio-visual retrieval."},{"cited_title":"Audio-visual event localization in unconstrained videos","cited_arxiv_id":null,"evidence_quote":"Provides the AVE dataset used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VEGAS dataset used for evaluation."}],"review_version":1}