{"id":"31a619c0-70b0-4e26-9a86-208ab216fdd7","arxiv_id":"2412.00049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A review that categorizes deep audio-visual correlation learning methods by architectures, objective functions, datasets, and evaluation metrics, and points to missing standardized benchmarks.","lead":"This paper is a survey that organizes recent deep-learning methods for connecting audio and visual data, grouped by model type, loss function, and learning paradigm. It is a starting point for researchers who want a map of the field, including its benchmarks and open problems.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 labels AudioSet mAP values as ACC, so the numeric evidence for the paper's headline trends is not internally consistent and needs correction.","rationale":"The reader's weakest assumption was that published performance numbers from different tasks and metrics are being treated as evidence despite the paper's own disclaimer. My pass found a concrete instance of that problem inside the paper itself: Table 1 and Table 2 disagree on whether AudioSet rows are ACC or mAP. This matters because Sections 4 and 5 explicitly cite Table 1 as the basis for the two central field trends. Correcting the labels would not require new experiments, but it would require softening or removing the claim that the trends are 'reflected in Table 1' as numeric evidence. The same correction also strengthens the reader's recommendation to add explicit search and inclusion criteria and matched benchmarks. The errors are presentation-level rather than conceptual, so the appropriate verdict remains CONDITIONAL rather than REJECT. I also note that several equations are misstated (e.g., Eq. 12 repeats the same hinge term twice), which reinforces the need for careful revision but is secondary to the Table 1 evidence problem.","tokens_in":54330,"tokens_out":10279,"duration_ms":88649,"concrete_test":"Audit the AudioSet rows of Table 1 against the original papers: retrieve the official AudioSet metric for refs [44] (Audiovisual Masked Autoencoders), [46] (UAVM), and [62] (MAViL), and check whether each value is mAP rather than accuracy. If the values are mAP (0.518, 0.504, 0.533) being presented as ACC (51.8, 50.4, 53.3), relabel those cells, remove the numeric trend claim from Section 4, and decide whether Table 1 should be restricted to controlled comparisons instead of mixed metrics and datasets.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The survey's two qualitative trends are explicitly anchored to Table 1: Section 4 states that, 'As reflected in Table 1, recent research has intensively demonstrated that i) self-supervised learning methods can achieve more discriminative features ... and ii) attention-based methods are designed to improve the synchronisation between audio and visual sequences.' That makes Table 1 load-bearing evidence for the central claim of a systematic, comparative survey. The table is not internally reliable for that role. For three AudioSet rows, the metric is labeled ACC: ref [46] is listed as 'ACC 50.4', ref [44] as 'ACC 51.8', and ref [62] as 'ACC 53.3'. The paper's own Table 2 reports AudioSet for those same methods as mean Average Precision, with values 0.504, 0.518, and 0.533 respectively. ACC and mAP are different metrics; presenting one as the other changes the meaning of the number. Since the table also mixes WER, SDR, FID, Recall@10, AUC, and ACC across datasets and protocols, the trend statements cannot be read as quantitative demonstrations. The survey itself concedes in Section 4 that 'it is not possible to make a direct quantitative comparison between them'; the table should be reframed as an index rather than evidence, or restricted to matched comparisons.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a survey of deep audio-visual correlation learning (AVCL), organizing recent work by feature extraction models (attention, auto-encoders, GANs, diffusion), objective functions, datasets, and downstream tasks. It claims to fill a gap in prior surveys by systematically categorizing methods across application use cases, and it identifies two field trends: self-supervised methods produce more discriminative features, and attention-based methods improve audio-visual synchronization. The paper includes a large comparison table (Table 1), dedicated benchmark tables for audio and video encoders (Tables 2 and 3), a dataset catalog (Table 4), and a discussion of evaluation metrics and research challenges.","tokens_in":54594,"tokens_out":4796,"duration_ms":44529,"significance":"If the survey's organization were reliable, it would be a useful entry point for researchers, especially because it covers a broad corpus (199 references) and offers a taxonomy that spans model families, losses, and datasets. The paper also contributes a useful perspective on structured knowledge and self-supervised proxy tasks as a route toward more interpretable AVCL. The benchmark appendices are a practical resource. However, the quantitative claims that anchor the survey's central trends are undermined by metric mislabeling and by the paper's own admission that direct comparisons are not possible; the survey's value currently lies in its qualitative taxonomy, not in the numerical evidence, and several objective-function equations are garbled. These issues are correctable, and the underlying synthesis remains plausible.","major_comments":[{"comment":"The two headline trends are introduced as \"As reflected in Table 1, recent research has intensively demonstrated that i) self-supervised learning methods ... and ii) attention-based methods ...\". However, Table 1 is not internally reliable for that role: the AudioSet rows for refs [46], [44], and [62] are labeled \"ACC\" with values 50.4, 51.8, and 53.3, whereas Table 2 reports the same methods as mean Average Precision (mAP) with values 0.504, 0.518, and 0.533. ACC and mAP are different metrics, and the table also mixes WER, SDR, FID, Recall@10, AUC, and ACC across different datasets and protocols. In addition, Section 4 concedes that \"it is not possible to make a direct quantitative comparison between them\". The table should be relabeled, or preferably reframed as an index of representative methods rather than as quantitative evidence for the trends; the trend statements should be presented as qualitative observations consistent with the cited works, not as results demonstrated by Table 1.","section":"Section 4 and Table 1"},{"comment":"Several objective-function formulas are garbled. Eq. (12) is written as the sum of two identical terms, max(0, cos(c,i)+α) + max(0, cos(c,i)+α), which cannot be the intended hinge loss. Eq. (13) has inconsistent index use: M_ij appears in a sum over j, but the numerator and denominator use cos(ii) and cos(ij) without clearly separating the positive and negative pairs, and L_va is written with the same cos(ij) structure rather than the transposed comparison. Eq. (16) uses cos(x_a, x_v) inside the sum over negative samples, where it should be cos(x_a, x') for x' in the negative set. Eq. (5) defines A_ij = 1/2 cos(u_a, u_v), which does not depend on i and j as written. These errors compromise the reference value of the objective-function catalog, which is a central component of the survey.","section":"Section 3.2, Eqs. (5), (12), (13), (16)"},{"comment":"The survey does not state its search strategy, inclusion criteria, or temporal coverage for the 199 cited works, even though it claims to offer a \"comprehensive study\" that \"fills this gap\". Without this information, the representativeness and completeness of the review cannot be independently checked. The authors should add a short methodology paragraph describing the databases searched, the keywords used, the inclusion/exclusion criteria, and the period covered, or explicitly position the survey as a structured narrative review rather than a systematic one.","section":"Section 1.2 and Section 4"}],"minor_comments":[{"comment":"The abstract contains a sentence fragment: \"The rapid growth of Deep Learning propelled the development of proposals that process audio-visual data and can be observed in the number of proposals in the past years. Thus encouraging the development of a comprehensive survey.\" This should be rewritten as a complete sentence.","section":"Abstract and Section 1"},{"comment":"The itemized lists in Section 3.1.2 and Section 3.2 contain duplicate numbering: two items are labeled \"vi\" in Section 3.1.2, and the objective-function list in Section 3.2 uses \"v\" three times (Correlation Loss, Adversarial Loss, ELBO) in an order that does not match the subsections. Renumber the lists for clarity.","section":"Section 3.1.2 and Section 3.2"},{"comment":"Eq. (10) writes KL(q(Z|x)||p(Z|x)) = log(p(x)) - E_{Z∼q}[p_θ(x,Z)/q_θ(Z|x)], but the expectation should contain the log ratio, i.e., E[log(p_θ(x,Z)/q_θ(Z|x))]. The missing logarithm makes the identity incorrect.","section":"Section 3.2, Eq. (10)"},{"comment":"One row in Table 3 reads \"VicTR† 72.4 51 RGB 32 [67]\" with the second number likely a typo for 95.8, given the later row \"VicTR† 70.7 95.8 RGB 32 [67]\" and the corresponding paper. Please verify all entries against the source.","section":"Table 3"},{"comment":"The description of SDR is inaccurate: SDR (signal-to-distortion ratio) measures the ratio of desired signal to distortion, not the Euclidean distance between representations or the amount of background noise. The text conflates SDR with other distance-based metrics.","section":"Section D.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a survey and does not derive new empirical results, so the bar for acceptance rests on the accuracy and usefulness of its synthesis. The metric-label inconsistency in Table 1 is a load-bearing issue because the paper explicitly uses that table to support its two main trends; the garbled equations in Section 3.2 further reduce the reference value of the survey. All issues appear locally fixable without changing the paper's scope. I would not support rejection, because the taxonomy and dataset collection are potentially useful, but the current version needs careful correction and reframing before it can serve as a reliable survey."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: this is a genuinely useful survey of deep audio-visual correlation learning. It organizes the literature by model family (attention, auto-encoders, GANs), objective function, and downstream task, and it does a serviceable job of connecting method choices to application areas. For a newcomer, it's a decent map. It also makes a fair point that cross-method comparison is nearly impossible given the field's habit of cherry-picking datasets and metrics. I'd want this on the desk of anyone starting in AVCL.\n\nThe soft spots are real but fixable. The biggest one is Table 1. The paper explicitly says that 'as reflected in Table 1,' self-supervised methods give more discriminative features and attention methods improve synchronization. But Table 1 is not internally consistent enough to carry that weight. For three AudioSet rows (refs 46, 44, 62) the metric is listed as ACC, while Table 2 in the appendix reports the same numbers as mAP (0.504, 0.518, 0.533). ACC and mAP are different measurements; presenting one as the other changes the meaning. The table also mixes WER, SDR, FID, AUC, Recall@10, and ACC across different datasets and protocols. The authors themselves concede in Section 4 that direct quantitative comparison is impossible. With that concession, the table should be reframed as a qualitative index, not evidence for a trend, or restricted to matched comparisons.\n\nThere are also mechanical errors that a referee would want cleaned up: the loss numbering in Section 3.2 is inconsistent (two 'v' items, then 'ii', 'iii', etc.), and several equations (5, 12, 13, 15, 16) have garbled or repeated terms. Eq. (11) looks wrong as written — the KL term should be between the posterior and prior, not the posterior and itself. And Eq. (5) has a mangled expression. These are typos, not conceptual flaws, but they undermine the careful reader's trust.\n\nOne thing I'd push back on lightly: the novelty framing. The survey's taxonomy is a re-organization of existing material, and they do draw from prior taxonomies (H. Zhu et al., Y. Yang et al.). That's fine for a survey, but the 'fills this gap' claim in Section 1.2 is stronger than what they deliver, given the lack of any systematic selection criteria for the 199 cited papers. A short paragraph on search strategy and inclusion/exclusion rules would fix this.\n\nFor the right reader — a grad student or practitioner wanting a structured overview — this paper is worth the read after revision. The core synthesis is sound and the limitations are honestly stated, despite the presentation issues. I'd send it to peer review, but with the clear expectation that Table 1 and the equation glitches get fixed before publication. I wouldn't cite it in its current form.","headline":"Useful survey of AV correlation learning, but the table that anchors its main claims mislabels mAP as accuracy; needs a revision before I'd trust the comparisons.","tokens_in":55102,"tokens_out":2418,"would_cite":false,"duration_ms":22439,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims that deep audio-visual correlation learning (AVCL) is best organized by model family and objective function, and that two trends dominate: self-supervised objectives produce more discriminative features, while attention…","keywords":["audio-visual correlation learning","multimodal machine learning","self-supervised learning","attention mechanisms","contrastive learning","video understanding","sound separation","representation learning"],"falsifier":"Apply explicit inclusion criteria and re-benchmark a stratified sample of the surveyed methods on one shared protocol with the same datasets, metrics, and training budget; if self-supervised methods do not show more discriminative features than equally trained supervised baselines, the headline trend is false.","tokens_in":54138,"feed_emoji":"🎬","tokens_out":6778,"duration_ms":57189,"temperature":0.7,"pith_summary":"The paper is a systematic survey of deep audio-visual correlation learning (AVCL), the task of projecting audio and visual features into a common space so that shared semantics can be measured and maximized. It aims to fill a gap left by earlier surveys by grouping roughly 199 recent methods according to model family (projectors, attention mechanisms, auto-encoders, GANs and diffusion models) and objective function (cross-entropy, correlation, adversarial, ELBO, hinge, triplet, contrastive, and NCE), across four application families: sound separation and localization, retrieval, recognition, and generation. Its central observations are that self-supervised learning methods produce more discriminative features by exploiting large unlabeled corpora, and that attention-based methods are the main engine for synchronizing audio and visual sequences. A sympathetic reader would care because the survey converts a scattered literature into a map that lets a researcher choose a method archetype and see the open problems, namely benchmark comparability, explainability, and structured knowledge injection.","feed_headline":"Self-supervision and attention lead audio-visual learning","feed_subtitle":"A systematic map of 199 methods shows which loss functions and architectures drive sound-image correlation.","key_machinery":"The organizing device is a two-axis categorization: multimodal encoding models (attention mechanisms, auto-encoders, GANs, and diffusion) on one axis, and objective functions (cross-entropy, correlation loss, adversarial loss, ELBO, hinge, triplet, contrastive, NCE) on the other, applied to the four application groups of sound separation and localization, retrieval, recognition, and generation. The paper's Table 1 is the load-bearing structure: each surveyed method is placed by model type, loss, learning paradigm, metric, and benchmark, which is what makes the cross-application comparison and the two field trends visible. The audio-visual correlation formalization, paired features $x_1, x_2$ projected into a shared or separate common space $S$, defines the problem that every method in the table attacks.","core_discovery":"On its own terms, the paper's discovery is taxonomic: it finds that methods aimed at different audio-visual applications share structural properties, so the field can be described by a small set of encoding models and loss families. It synthesizes two field-level trends: (i) self-supervised methods yield more discriminative features by leveraging large volumes of unlabeled data, and (ii) attention-based methods improve alignment and synchronization of audio and visual sequences. It also positions the reviewed methods inside deep knowledge representation, meaning they encode raw signals into abstract, non-explainable embeddings, and argues that injecting human-understandable structured knowledge, such as proxy tasks and pseudo-labels, is the natural route to interpretability and reliability.","pith_inferences":["Because the survey does not run its own experiments, a controlled comparison that varies only the learning paradigm would be needed to confirm that self-supervision, not just scale, produces the reported discriminative advantage.","The absence of explicit inclusion criteria means the comprehensiveness of the 199-method table is not independently checkable; a preregistered search could shift the reported distribution of model families.","The paper's observation that attention interaction functions, such as addition versus dot-product, constrain what can be correlated suggests that matching interaction function to representation structure is a design variable future methods should report.","Graph-based text semantics, mentioned as a way to reduce language-model cost, could also serve as a source of structured knowledge and as a principled way to choose negative samples for contrastive losses in AVCL."],"forward_implications":["A newcomer can choose an AVCL method by looking up model family and loss in the survey's overview table instead of reading the field application by application.","Self-supervised pretraining on large unlabeled corpora will continue to be the main source of discriminative audio-visual features, with fine-tuning on smaller labeled sets for downstream tasks.","Attention-based models, especially co-attention and cross-attention, will remain the default mechanism for fine-grained audio-visual alignment and synchronization.","Without shared benchmarks and metrics, reported results across the surveyed methods cannot be ranked, so standard evaluation protocols are a prerequisite for the field's progress.","Injecting human-understandable structured knowledge, such as pseudo-labels, proxy tasks, text, or graphs, is the paper's proposed direction toward interpretability and reliability of deep AVCL models."],"supporting_citations":[{"why":"Supplies the four-category application taxonomy that the paper adopts for organizing audio-visual correlation methods.","marker":"[192]"},{"why":"Provides the deep versus canonical knowledge representation taxonomy used to classify AVCL methods and motivate structured knowledge.","marker":"[171]"},{"why":"Frames the multimodal machine learning challenges against which the survey defines its contribution.","marker":"[79]"},{"why":"Supplies the speech, music, and ambient audio taxonomy used in the feature extraction analysis.","marker":"[104]"},{"why":"Provides baseline audio feature extractors and AudioSet benchmark results against which newer audio encoders are compared.","marker":"[69]"},{"why":"Serves as the exemplar of text-supervised vision-language pretraining used for visual encoding and alignment extensions.","marker":"[108]"},{"why":"Is the source of the multimodal transformer interaction taxonomy shown in Figure 6 and discussed in the attention section.","marker":"[167]"},{"why":"Supplies the self-attention, multi-head attention, and Transformer machinery at the center of the attention-based trend.","marker":"[141]"},{"why":"Defines the AudioSet baseline dataset chosen for comparing audio encoding models throughout the survey.","marker":"[43]"}],"fun_headline_variants":["Survey maps 199 audio-visual methods, flags self-supervision and attention","Audio-visual learning trends: self-supervision and attention lead","200 audio-visual models reviewed: what drives sound-image correlation","Self-supervision and attention dominate audio-visual correlation learning","Audio-visual correlation survey: self-supervision, attention, and future paths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's summary trends rest on published performance numbers collected from different datasets, metrics, and training setups, while the paper itself concedes that direct quantitative comparison between them is not possible.","fun_headline_variants_meta":{"raw":{"variants":["Survey maps 199 audio-visual methods, flags self-supervision and attention","Audio-visual learning trends: self-supervision and attention lead","200 audio-visual models reviewed: what drives sound-image correlation","Self-supervision and attention dominate audio-visual correlation learning","Audio-visual correlation survey: self-supervision, attention, and future paths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000152,"raw_usage":{"total_tokens":1149,"prompt_tokens":833,"completion_tokens":316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":449,"tokens_out":316,"duration_ms":3162,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:59:54.158459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply explicit inclusion criteria and re-benchmark a stratified sample of the surveyed methods on one shared protocol with the same datasets, metrics, and training budget; if self-supervised methods do not show more discriminative features than equally trained supervised baselines, the headline trend is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the deep versus canonical knowledge representation taxonomy used to classify AVCL methods and motivate structured knowledge."}],"review_version":1}