{"id":"65c8f36c-9600-4a3e-97ac-55cc27df1708","arxiv_id":"1908.07673","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding triplet neural networks to supervised deep CCA raises audio-visual retrieval mean average precision from 70.34% to 72.69% on VEGAS.","lead":"This short paper adds a triplet neural network to a supervised canonical correlation analysis pipeline for audio-visual retrieval, and reports a small accuracy gain on the VEGAS dataset. It is a work-in-progress note relevant to researchers tracking CCA-based cross-modal retrieval methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II's 2.4-point TNN gain is not interpretable without a stated train/test split; if triplets are trained on the same VEGAS videos later used as queries, the mAP may reflect concept memorization rather than retrieval generalization.","rationale":"The reader's weakest assumption concerns temporal alignment: the mean layer collapses time, so per-second pretrained features plus a mean may not generalize to longer or more complex temporal dynamics. That is a real limitation, but it is about generalization beyond VEGAS. The more load-bearing issue for the paper's actual central claim is that the reported mAP comparison may not be valid even on VEGAS, because the paper never specifies whether evaluation queries are disjoint from the training pairs used to build the triplet loss. Since the triplet network is trained using concept correspondences, using the same videos as queries could let the model exploit category labels instead of learning a true cross-modal embedding. This would directly undermine the headline result. The reader's rationale does mention missing training details and error bars, which is related, but not the specific train/test split concern. I therefore mark agreement as partial. I keep the verdict at CONDITIONAL rather than moving to REJECT or UNVERDICTED because the flaw is an unstated protocol detail that could be resolved by the authors; if the split is provided and the gain survives, the claim would be supported, which is exactly the conditional acceptance path.","tokens_in":3573,"tokens_out":5451,"duration_ms":59585,"concrete_test":"Ask the authors to release the exact video-level train/test split, or re-run the pipeline with a leave-one-out or 80/20 split, and recompute Table II using only queries whose audio-visual pairs were not used to construct triplets, reporting mAP with variance across splits. If the TNN vs S-DCCA gap disappears or reverses under this protocol, Table II does not support the claimed enhancement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The numerical headline depends entirely on Table II's 2.4-point mAP gain of TNN over S-DCCA, but the paper never states whether the VEGAS pairs used to train the triplet network are disjoint from the queries used for evaluation. Section II-B says correspondences are built 'based on their concepts' (category labels), and Section III reports retrieval results over the dataset with no train/test split, cross-validation, or held-out query set. VEGAS is a small, category-structured collection; if triplets are trained on the same videos that are later used as queries, the TNN can inflate mAP by memorizing concept-level correspondences rather than by learning a general cross-modal embedding. The claim that 'TNN outperforms S-DCCA' therefore rests on an unstated evaluation protocol that is essential to the comparison: without a disjoint test set, the 2.4-point difference cannot be attributed to the method's retrieval ability. This is more immediately load-bearing than the temporal-alignment question, because it affects the reported numbers themselves, not just generalization to other datasets. The absence of error bars or variance estimates makes it impossible to tell whether the gain is even statistically distinguishable from noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This short forum paper proposes a triplet neural network (TNN) enhancement over a supervised deep canonical correlation analysis (S-DCCA) architecture for audio-visual cross-modal retrieval. The method extracts high-level features (VGGish/Inception), applies a mean layer, learns a CCA-based joint embedding, and then uses triplet ranking to incorporate dissimilar pairs into the training objective. Preliminary results on the VEGAS dataset are reported in two tables: Table I compares S-DCCA under different feature combinations, and Table II compares CCA, DCCA, S-DCCA, and TNN, with TNN achieving 72.69% mAP for audio-visual and 71.84% for visual-audio, about 2.4 points above S-DCCA. The paper concludes that the proposed architecture achieves the best result when supervised data representations are used.","tokens_in":3704,"tokens_out":3059,"duration_ms":109032,"significance":"If the reported gains are reproducible and statistically reliable, the paper demonstrates a simple, potentially generalizable way to improve supervised CCA for cross-modal retrieval by adding triplet-based ranking. The evaluation uses the external VEGAS benchmark, so the central claim is not normalized by construction and is in principle falsifiable. However, the contribution is incremental, and the paper's value depends entirely on the credibility of the empirical comparison, which is currently not established because key experimental details are missing. The architecture is simple enough that the work could be a useful baseline if fully specified, but as written it does not support the 'state-of-the-art' claim.","major_comments":[{"comment":"The evaluation protocol is unspecified: the paper never states how the VEGAS data is split into training and test sets, whether queries are disjoint from the videos used to train the triplet network, or whether any cross-validation or held-out set is used. Section II-B says correspondences are built 'based on their concepts' and Section III reports retrieval results over the dataset, but no split is described. If the triplets are trained on the same videos later used as queries, the reported mAP gain could reflect memorization of concept-level correspondences rather than retrieval generalization. The authors must specify the train/test protocol, report results on a disjoint test set, and provide error bars or variance estimates so the 2.4-point difference can be assessed.","section":"Section III and Table II"},{"comment":"The TNN architecture and training procedure are not specified in enough detail to reproduce the method. The paper does not give the triplet loss formulation, margin, sampling strategy, number of negative samples per query, optimizer, learning rate, number of epochs, or regularization. The only architectural details are a mean layer and an output dimension of 10. Without these details, the comparison in Table II is not interpretable as a fair evaluation of a well-defined method, and the claim that 'TNN model outperforms S-DCCA' cannot be independently verified.","section":"Section II-B and III"},{"comment":"The comparison to 'state-of-the-art' methods is incomplete and partly self-referential. Table II includes only CCA, DCCA, S-DCCA, and TNN, where S-DCCA is the authors' own prior method and TNN is described as 'our proposed architecture [11], [12],' citing two previous papers by the same group. The relation between the current TNN and the earlier deep triplet network [12] is not clarified, so it is unclear what new contribution this paper makes beyond those earlier works. A stronger comparison should include recent cross-modal retrieval methods that have published results on VEGAS or on comparable audio-visual benchmarks, and the text should state explicitly what is new relative to [11] and [12].","section":"Table II and Section III"}],"minor_comments":[{"comment":"The phrase 'based on their concepts' is vague; the authors should state how many concepts VEGAS contains, how one-to-one correspondences are constructed from concept labels, and whether every video is paired with all other videos in the same concept or only with its own pair.","section":"Section II-B"},{"comment":"Table I reports mAP for S-DCCA under four feature combinations, but it is not stated whether these features were taken from the official VEGAS release or re-extracted by the authors. This matters because feature extraction details can affect the reported numbers.","section":"Section III"},{"comment":"The sentence 'Once The similarity matrix is generated' contains a capitalization error; it should read 'Once the similarity matrix is generated.'","section":"Section III"},{"comment":"The paper refers to the proposed architecture as 'our proposed architecture [11], [12]' but does not specify which components are inherited from the prior works and which are newly introduced here. Adding one sentence to delimit the novelty would clarify the contribution.","section":"Section I and III"},{"comment":"The notation 'ai ∈ R^{128×L×M}' is used, but L is later described as 'the number of extracted frames for video i' and M as 'the number of videos.' Since i indexes the video, the notation should make it clear whether L and M are global constants or vary per video; a brief clarification would help.","section":"Section II-A"}],"recommendation":"major_revision","confidential_remarks":"The paper reads more like an extended abstract or workshop note than a full research article: no code, no experimental protocol, no ablation, and very little architectural detail. The self-citation pattern in [11] and [12] is understandable for a forum paper, but the lack of a clear statement of what is new relative to those works weakens the contribution. The most serious issue is the absent train/test split, which makes the headline numerical claim unverifiable. If the authors can provide a proper evaluation protocol and the missing training details, the paper could be acceptable as a short empirical contribution; otherwise, the central claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a short, honest work-in-progress note. What is actually new is a small increment on the author's S-DCCA line: add a triplet loss to the supervised CCA objective and evaluate on VEGAS across different feature extractors. Table I gives a useful internal consistency check (VGGish/Inception beats SoundNet/I3D by a wide margin), and the external VEGAS benchmark is the right kind of evaluation. The gain direction is consistent for both audio-visual and visual-audio, which is at least a sign that the effect is not entirely arbitrary. That should be credited.\n\nThe soft spots, in order of severity. First, the stress-test concern is real and load-bearing: the paper does not state whether the VEGAS videos used to train the triplet network are disjoint from the queries used for retrieval. Section II-B constructs correspondences from the dataset using concept labels, and Section III reports retrieval results with no train/test split, held-out query set, or cross-validation. VEGAS is small and category-structured; if training and querying share videos, the 2.4-point TNN gain could reflect concept memorization rather than retrieval generalization. This affects the reported numbers themselves, not just transfer to other datasets.\n\nSecond, there are no error bars or variance estimates, so we cannot tell whether 2.4 points is above noise. Third, no training details, data splits, hyperparameters, or code are provided, so the experiments are not reproducible. Fourth, Table II compares only against CCA and DCCA, and DCCA at 14.09% being far below linear CCA at 32.43% is odd enough that the setup needs explanation. Fifth, the relation to the author's reference [12], which already announces deep triplet networks with cluster-CCA, is not clarified; the incremental novelty here seems limited to feature representation and a slightly different loss application. The temporal-alignment assumption in Section II-B (mean layer over per-second features) is a real limitation but secondary to the split problem.\n\nI would not publish this as is, but I would not desk reject it. A referee could ask for a clear split, error bars, and a direct comparison to [12], and if the gain survives on held-out queries this becomes a legitimate incremental result. I would send it to peer review with a conditional recommendation: fix the evaluation protocol before acceptance.","headline":"A plausible 2.4-point mAP gain on VEGAS from adding triplet loss to S-DCCA, but the missing train/test split makes the headline result uninterpretable as reported.","tokens_in":4295,"tokens_out":4633,"would_cite":false,"duration_ms":46381,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Triplet-enhanced supervised CCA lifts audio-visual retrieval to 72.69% mAP on VEGAS.","keywords":["cross-modal retrieval","joint embedding","canonical correlation analysis","triplet neural networks","audio-visual","deep learning","VEGAS dataset"],"falsifier":"Run the TNN and S-DCCA models on a dataset of longer, temporally varying videos (e.g., full-length movie scenes or unedited event videos) where the per-second features are not aligned by concept; if the mAP gain over S-DCCA disappears or reverses, the temporal-alignment assumption is the cause.","tokens_in":3279,"feed_emoji":"🎵","tokens_out":2066,"duration_ms":546136,"temperature":0.7,"pith_summary":"This paper claims that adding triplet neural networks (TNN) to a supervised deep canonical correlation analysis (S-DCCA) architecture improves cross-modal retrieval between audio and video. On the VEGAS dataset, the TNN model reaches 72.69% mean average precision for audio-to-visual retrieval and 71.84% for visual-to-audio, outperforming S-DCCA and other baselines by about 2.4 points. The authors argue that explicitly pulling similar pairs together and pushing dissimilar pairs apart in the shared embedding space enhances the correlation learned by CCA. They further claim the approach generalizes to other paired multimedia data, such as image-text and video-text retrieval.","feed_headline":"Triplet-enhanced CCA pushes audio-visual retrieval to 72.7% mAP","feed_subtitle":"Adding negative samples to supervised CCA on VEGAS beats prior best by 2.4 points.","key_machinery":"The central machinery is a two-branch triplet neural network attached to a supervised deep CCA (S-DCCA) embedding. Raw audio is represented with VGGish features (128-dimensional per second) and raw video with Inception features (1024-dimensional per second); a mean layer collapses the time dimension before correlation learning. The CCA branch learns linear projections that maximize audio-visual correlation in a shared space, while the triplet branch uses anchor-positive and anchor-negative samples to pull matching pairs together and push non-matching pairs apart. This combined objective generates the final embeddings used for similarity matrix computation and ranking.","core_discovery":"The central claim is that a triplet neural network, operating on top of a CCA-based joint embedding, yields a better cross-modal retrieval model than supervised CCA alone. The authors show that when audio and visual features are first mapped into a common space by maximizing correlation, adding triplet loss terms—which enforce that matching audio-visual pairs are closer than mismatched ones—improves retrieval accuracy. On the VEGAS dataset, their TNN model achieves 72.69% mAP for audio-to-visual and 71.84% for visual-to-audio, compared to 70.34% and 69.27% for S-DCCA, and much lower scores for plain CCA and DCCA. The paper presents this as evidence that incorporating category-level similarity and dissimilarity information into correlation learning strengthens the joint embedding.","pith_inferences":["The mean layer that collapses time likely works because VEGAS clips are short and conceptually homogeneous; on longer videos where events shift over time, the fixed mean may wash out the temporal structure that links specific audio moments to specific visual moments.","A testable extension would be to replace the mean layer with temporal attention or a sequence encoder and measure whether retrieval on VEGAS or a longer-video dataset improves; this would directly test the paper's implicit assumption that temporal alignment is unnecessary.","Because the triplet objective only contrasts whole-video embeddings, it may underuse fine-grained correspondences; a frame-level or segment-level triplet loss could be a stronger constraint, but it would require temporally aligned positive pairs, which the current dataset and features do not provide.","The reported improvements are small relative to the gain from switching feature extractors, suggesting that future work should prioritize better audio and visual representations over tweaks to the correlation objective."],"forward_implications":["If the TNN improvement holds beyond VEGAS, cross-modal retrieval systems for audio, video, image, and text could adopt triplet-enhanced supervised CCA as a straightforward upgrade to plain CCA-based joint embeddings.","The reported gap between supervised (VGGish/Inception) and unsupervised (SoundNet/I3D) feature representations suggests that the choice of per-second high-level features is a major driver of retrieval performance, more so than the correlation learning method itself.","The architecture's reliance on one-to-one concept correspondences indicates it could be applied to other paired datasets that have category labels, such as video-text captioning pairs, though temporal alignment is not explicitly handled.","The authors' plan to combine generative adversarial networks with the triplet discriminative model points toward a future direction where generated features are used to enrich training data for cross-modal retrieval."],"supporting_citations":[{"why":"Provides the original CCA formulation and linear projection machinery that the supervised deep CCA builds on.","marker":"[5]"},{"why":"Introduces Deep Canonical Correlation Analysis (DCCA), the deep-learning extension that S-DCCA adapts with category supervision.","marker":"[7]"},{"why":"Introduces the VEGAS dataset of paired audio-visual clips used for all experiments and comparisons.","marker":"[10]"},{"why":"Describes the authors' prior S-DCCA architecture, which is the direct baseline the triplet enhancement is measured against.","marker":"[11]"},{"why":"Extends the same line of work with deep triplet networks and cluster-CCA, forming the basis for the TNN model's triplet loss design.","marker":"[12]"}],"fun_headline_variants":["Triplet-enhanced CCA boosts audio-visual retrieval to 72.7% mAP","Audio-visual mAP 72.7% via triplet-boosted CCA on VEGAS","Triplet loss on CCA: 72.7% mAP beats S-DCCA on VEGAS","Adding negative samples to CCA improves cross-modal retrieval to 72.7% mAP","Supervised correlation learning with triplets hits 72.7% mAP on VEGAS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that per-second high-level features (VGGish/Inception) collapsed by a mean layer are enough to make audio and visual sequences comparable in a shared CCA space, without any temporal alignment.","fun_headline_variants_meta":{"raw":{"variants":["Triplet-enhanced CCA boosts audio-visual retrieval to 72.7% mAP","Audio-visual mAP 72.7% via triplet-boosted CCA on VEGAS","Triplet loss on CCA: 72.7% mAP beats S-DCCA on VEGAS","Adding negative samples to CCA improves cross-modal retrieval to 72.7% mAP","Supervised correlation learning with triplets hits 72.7% mAP on VEGAS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2327,"prompt_tokens":868,"completion_tokens":1459,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":1335}},"tokens_in":484,"tokens_out":1459,"duration_ms":222426,"temperature":1.0,"reasoning_tokens":1335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:59:04.995419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the TNN and S-DCCA models on a dataset of longer, temporally varying videos (e.g., full-length movie scenes or unedited event videos) where the per-second features are not aligned by concept; if the mAP gain over S-DCCA disappears or reverses, the temporal-alignment assumption is the cause.","supporting_citations":[{"cited_title":"Canonical corre- lation analysis: An overview with application to learning methods,","cited_arxiv_id":null,"evidence_quote":"Provides the original CCA formulation and linear projection machinery that the supervised deep CCA builds on."},{"cited_title":"Deep canonical cor- relation analysis,","cited_arxiv_id":null,"evidence_quote":"Introduces Deep Canonical Correlation Analysis (DCCA), the deep-learning extension that S-DCCA adapts with category supervision."},{"cited_title":"Visual to sound: Generating natural sound for videos in the wild,","cited_arxiv_id":null,"evidence_quote":"Introduces the VEGAS dataset of paired audio-visual clips used for all experiments and comparisons."},{"cited_title":"Audio-visual embedding for cross- modal music video retrieval through supervised deep cca","cited_arxiv_id":null,"evidence_quote":"Describes the authors' prior S-DCCA architecture, which is the direct baseline the triplet enhancement is measured against."},{"cited_title":"Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval","cited_arxiv_id":"1908.03737","evidence_quote":"Extends the same line of work with deep triplet networks and cluster-CCA, forming the basis for the TNN model's triplet loss design."}],"review_version":1}