{"id":"22127991-bfa5-4a09-9414-b098bf4a38ef","arxiv_id":"1908.03737","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TNN-C-CCA, a two-stage pipeline of Cluster-CCA embeddings refined by a deep triplet network with cosine triplet loss, achieves state-of-the-art MAP on VEGAS and MV-10K audio-visual retrieval benchmarks.","lead":"The authors combine cluster canonical correlation analysis with a triplet neural network to embed audio and video clips in a shared space for cross-modal retrieval. The method reports higher retrieval accuracy than several CCA-based and adversarial baselines on two audio-video datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains likely inflated by test-set hyperparameter selection; fair comparison requires nested validation.","rationale":"The reader's verdict was CONDITIONAL, with the weakest assumption being that baselines were implemented and tuned fairly. My concern is closely related but more specific: the proposed method's own hyperparameters are selected on the test set, creating an asymmetric comparison. This is the single most load-bearing issue because it directly affects the validity of the headline MAP differences. The reader also noted this in the rationale ('hyperparameters are selected on the test set'), so there is partial agreement; however, the reader's stated weakest_assumption emphasized undertuned baselines, whereas I emphasize overtuning of the proposed method. Other issues (no error bars, no code release, internal table inconsistencies) are real but secondary. The concrete test—nested cross-validation with comparable tuning—would settle whether the reported gains survive a fair protocol. If the gains persist, the concern is resolved and the conditional acceptance stands; if not, the central claim fails. Therefore I recommend keeping the reader's CONDITIONAL verdict: the paper needs revision to provide a fair evaluation before the outperformance claim can be accepted. My read does not change the verdict, so verdict_should_be is UNCHANGED.","tokens_in":22388,"tokens_out":4169,"duration_ms":44808,"concrete_test":"Run a nested 5-fold cross-validation on VEGAS. For each outer fold, split the training fold into sub-train and validation; select margin (0.1-1.1), batch number (300-900), and correlation components (10-50) on the validation subset; then evaluate on the outer test fold. Apply the same inner validation procedure to tune C-DCCA's beta. Report mean and standard deviation of MAP over outer folds for both methods. If TNN-C-CCA's mean MAP drops below 72% on audio-to-visual retrieval, or its advantage over the tuned C-DCCA falls below 2 percentage points, the headline outperformance claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TNN-C-CCA outperforms six CCA-based and four state-of-the-art baselines on VEGAS and MV-10K, with the headline numbers in Table 2. The most load-bearing weakness is that these numbers are produced after selecting hyperparameters on the same test folds used for evaluation. Section 4.5.4 varies margin, batch number, and correlation components directly on VEGAS and reports the best MAP (e.g., margin 0.5, 400 batches, 10 components), and Fig. 10 shows component count selected by best test MAP. This is test-set tuning, which inflates the proposed method's reported performance. Meanwhile, baselines are re-implemented 'with the same dimension of outputs and the same parameters' (Section 4.3), but no comparable tuning is documented; C-DCCA's beta is explicitly described as 'very hard to set,' with no search reported. Thus the 4.3-point VEGAS gap over C-DCCA may reflect a tuning advantage rather than genuine method superiority. This is load-bearing because the paper's empirical claim is the entire basis for acceptance, and the comparison protocol does not support it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents TNN-C-CCA, an end-to-end two-branch architecture for audio-visual cross-modal retrieval. Audio clips are embedded with VGGish and visual frames with Inception-v3 plus PCA; Cluster-CCA projects the two views into a shared 10-dimensional space, and a triplet network with cosine distance and in-batch mining refines the embeddings using same-category positive pairs and different-category negative pairs. The method is evaluated on VEGAS and the authors' MV-10K dataset with MAP and PRC, and compared with six CCA-based methods and four state-of-the-art cross-modal retrieval methods. The paper claims consistent improvements, with the largest being 74.66% versus 70.34% audio-to-visual MAP over C-DCCA on VEGAS. Ablation studies cover triplet selection strategies, Euclidean versus cosine distance, and sensitivity to margin, batch size, and correlation components.","tokens_in":22563,"tokens_out":6986,"duration_ms":65765,"significance":"The audio-visual retrieval setting is underexplored, and the combination of cluster-level correlation learning with triplet ranking is a reasonable and clearly motivated idea. The paper's strengths include an end-to-end architecture, five-fold cross-validation, a broad comparison set, and an ablation study that probes several design choices. If the reported gains survive a statistically fair comparison, the method would be a useful contribution to multimedia retrieval. However, the quantitative claim is currently not supported by the evaluation protocol, because the proposed method's hyperparameters are selected on the test folds and the baselines are not shown to be comparably tuned. The contribution is incremental rather than conceptual, but it could be acceptable after a rigorous re-evaluation.","major_comments":[{"comment":"The margin (0.5), batch number (400), and correlation component count (10) are chosen by running the model on the VEGAS dataset and selecting the values with the highest MAP. Figure 10 plots MAP against the number of correlation components without any indicated validation split. Since the same folds are used for both selection and evaluation, the headline MAPs in Table 2 are optimistically biased. The authors should use a held-out validation split (or nested cross-validation) and report test MAP for the configuration selected without access to test labels.","section":"Section 4.2(3) and Section 4.5.4"},{"comment":"The baseline comparison is not sufficient to support the claimed improvement. The paper states that all baselines were implemented \"with the same dimension of outputs and the same parameters,\" which does not address hyperparameter tuning. C-DCCA's beta is explicitly described as \"very hard to set,\" yet no search is reported; the VEGAS advantage over C-DCCA is only 4.3 percentage points. To make the comparison interpretable, the authors should report validation-based hyperparameter selection for every baseline or use the numbers from the original publications.","section":"Section 4.3"},{"comment":"The reported numbers are internally inconsistent. Table 5, which is presented as the margin study with \"batch_num is 500,\" gives audio-visual MAP 75.31 and visual-audio MAP 73.26 at margin 0.5. Table 6, the batch-size study at margin 0.5, reports at 500 batches 74.51 and 72.98. The text states that the best configuration is 400 batches, yet the same MAP values appear in Table 5 for 500 batches. These inconsistencies must be corrected and explained.","section":"Tables 5 and 6"},{"comment":"No measure of variability is reported. The MV-10K improvements over C-DCCA are 1.55 and 1.24 percentage points, which may be within fold-to-fold noise. The authors should report per-fold results with standard deviations and perform a paired significance test across the five folds.","section":"Section 4.2(4) and Table 2"}],"minor_comments":[{"comment":"Equation (1) omits the squares in the denominator: the cosine distance should be 1 - sum(x_k y_k) / (sqrt(sum x_k^2) sqrt(sum y_k^2)).","section":"Section 3.1, Eq. (1)"},{"comment":"The MV-10K visual-to-audio value for LSTM_C_CCA is written as \"18,89\" with a comma; it should be 18.89.","section":"Table 2"},{"comment":"The list of VEGAS categories includes \"rail transport\" twice, although ten categories are claimed.","section":"Section 4.1"},{"comment":"The baseline LSTM_C_CCA appears in Table 2 and is mentioned in Section 4.3, but no reference, architecture description, or training procedure is given.","section":"Section 4.3 and Table 2"},{"comment":"There is an arithmetic slip: 6*63*70 is 26,460, not 264,640 as stated.","section":"Section 4.5.4"},{"comment":"The indices j and k are not defined under the summation; the triplet condition should be written with explicit (i,j,k) triples.","section":"Section 3.4, Eqs. (7)-(8)"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is an incremental combination of existing components, and its publication value depends entirely on whether the empirical comparison can be made fair. The MV-10K dataset comes from the authors' previous work; this is disclosed but should be clearly cited as their own dataset. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is a straightforward but sensible stack: Cluster-CCA to get a shared embedding, then a FaceNet-style triplet network with cosine triplet loss to push same-class cross-modal pairs together and different-class pairs apart. That combination for audio-visual retrieval is a legitimate extension of published techniques. Second, the reported gains over baselines are not trustworthy as-is, because the authors selected their hyperparameters on the test folds and did not document comparable tuning for the baselines.\n\nThe paper does some things well. The ablation study covers triplet selection strategies (batch all, batch hard, batch semi-hard), distance metrics, and correlation-component counts. The authors are candid about known difficulties, e.g., C-DCCA's beta being hard to set and random negative sampling being statistically unreliable. The MV-10K dataset is from their own prior work, but they disclose that.\n\nThe load-bearing problem is the evaluation protocol. Section 4.5.4 sweeps margin, batch number, and correlation components on VEGAS and picks the values with the best test MAP; those tuned values then produce the headline numbers in Table 2. That is test-set selection, and it inflates the proposed method's score relative to baselines that are re-implemented 'with the same parameters' but with no reported search. The paper even says C-DCCA's beta is very hard to set, and gives no evidence that beta was tuned. So the 4.3-point VEGAS gap over C-DCCA could easily be a tuning artifact.\n\nThere are also internal inconsistencies: Table 5 and Table 6 both report 75.31% audio-to-visual MAP for the configuration (margin 0.5, batch 400/500), but Table 2 reports 74.66% for the same setup. Table 2 also contains a typo (18,89). No error bars are given anywhere, and no code or data is released. These problems do not prove the method is wrong, but they mean the central empirical claim is not currently supported.\n\nWho is this for? Researchers working specifically on audio-visual cross-modal retrieval. For that niche, the idea is worth engaging with, but only after the evaluation is cleaned up. I would not cite the headline numbers in their current form.\n\nRecommendation: This deserves a serious referee, not a desk reject, because the core approach is plausible and the experiments are extensive. But the referee should demand major revision: a held-out validation set for hyperparameter selection, reported variance across folds, documented baseline tuning, corrected tables, and ideally code/data. With those changes, this could be a solid applied contribution.","headline":"A plausible stacking of Cluster-CCA with a cosine triplet network for audio-visual retrieval, but the headline MAP gains are unreliable because hyperparameters were tuned on the test set and baseline tuning is undocumented.","tokens_in":23132,"tokens_out":2294,"would_cite":false,"duration_ms":22693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Audio-visual retrieval improves when a deep triplet network refines Cluster-CCA embeddings by pulling same-category cross-modal pairs together in cosine space.","keywords":["audio-visual cross-modal retrieval","deep triplet neural network","cluster canonical correlation analysis","triplet loss","joint embedding learning","canonical correlation analysis","cross-modal retrieval","mean average precision"],"falsifier":"Retrain every baseline on the same VEGAS and MV-10K folds with per-method hyperparameter search, especially C-DCCA's beta and the regularization of the hashing and adversarial baselines, and check whether TNN-C-CCA's MAP margins survive; a modest grid search that closes the 4.28-point gap on VEGAS would falsify the outperformance claim.","tokens_in":22144,"feed_emoji":"🎬","tokens_out":9620,"duration_ms":89118,"temperature":0.7,"pith_summary":"The paper sets out to improve cross-modal retrieval between audio and video, a setting with little prior work, by learning a shared embedding where a sound and a video from the same semantic category are close while pairs from different categories are pushed apart. It proposes TNN-C-CCA, a supervised two-branch architecture: Cluster-CCA first projects VGGish audio features and Inception V3 visual features into a common low-dimensional space, and a deep triplet neural network then refines that space with triplet loss over cosine distance. On the VEGAS and MV-10K datasets the authors report that this combination beats six CCA-based baselines and four recent cross-modal retrieval methods on mean average precision, with the largest gains on VEGAS (74.66% audio-to-visual and 73.77% visual-to-audio). A sympathetic reader would care because the method is a concrete recipe for category-supervised audio-visual retrieval and because the triplet stage directly attacks the pairwise-only limitation of standard correlation learning.","feed_headline":"Audio-to-video retrieval hits 74.66% MAP with triplet-plus-CCA","feed_subtitle":"Pairing Cluster-CCA with a deep triplet network beats six CCA methods and four recent baselines on two audio-visual datasets.","key_machinery":"The architecture chains two stages. Cluster-CCA projects the two modalities into a shared subspace by treating every cross-modal pair inside a semantic category as a matching pair and applying canonical correlation analysis, which segregates categories while maximizing correlation. A deep triplet neural network then refines those embeddings with loss $d_{\\cos}(T(i),S(j)) + \\alpha < d_{\\cos}(T(i),S(k))$, where $T(i)$ is the anchor audio embedding, $S(j)$ a same-category visual embedding, $S(k)$ a different-category visual embedding, $d_{\\cos}$ is cosine distance, and $\\alpha=0.5$; in-batch triplet selection (batch all, batch hard, batch semi-hard) controls which negatives are used. The triplet fine-tuning stage is what turns pairwise and category-level correlation into a space where same-category cross-modal pairs are closer than different-category pairs.","core_discovery":"The paper claims that adding a deep triplet neural network on top of Cluster-CCA produces more discriminative audio-visual embeddings than either component alone or than existing CCA variants. The architecture maps audio and visual features to 10-dimensional embeddings via Cluster-CCA, then trains three fully connected branches (audio anchor, visual positive, visual negative, with the two visual branches sharing weights) using triplet loss with cosine distance and a margin of 0.5. In 5-fold cross-validation, TNN-C-CCA reaches MAP 74.66% and 73.77% for audio-to-visual and visual-to-audio retrieval on VEGAS, and 23.34% and 21.32% on MV-10K, surpassing the strongest CCA baseline C-DCCA by 4.28 and 4.5 percentage points on VEGAS. Ablation results show cosine distance beats Euclidean distance, batch-all triplet selection beats batch-hard and batch semi-hard, and the learned embedding clusters are more separated than Cluster-CCA's.","pith_inferences":["The two-stage recipe should transfer to other labeled modality pairs such as image-text or video-text, since the architecture does not depend on audio-specific machinery beyond the feature extractors.","Because performance drops sharply on MV-10K's automatically assigned labels, a testable extension is to corrupt a known fraction of VEGAS labels and measure how quickly the MAP advantage over C-DCCA erodes.","The paper's own batch-size table suggests that an external reader could check whether hard-negative mining at small batch sizes recovers most of the reported gain at a fraction of the 32-hour cost."],"forward_implications":["Audio-to-visual and visual-to-audio retrieval improve together, so the learned shared space is usable in both query directions.","Explicitly training on negative cross-modal pairs, rather than only on positive/category pairs, is the component the paper credits for the margin over C-DCCA.","The method's gains shrink on the noisily labeled MV-10K dataset, indicating that label quality is a binding constraint on the approach.","The reported batch-size experiments imply a practical cost curve: reaching the top MAP requires enough negatives per batch, and the paper reports training times up to 32 hours for the best configuration."],"supporting_citations":[{"why":"Supplies the original CCA objective and the linear baseline that Cluster-CCA extends and that all variants are compared against.","marker":"[8]"},{"why":"Introduces Cluster-CCA, whose category-wise one-to-one correspondences form the projection stage that the triplet network refines.","marker":"[25]"},{"why":"Provides DCCA, the deep nonlinear CCA baseline and the basis for the C-DCCA comparison.","marker":"[2]"},{"why":"Defines C-DCCA, the strongest baseline the paper must beat, and the category-correlation idea the method extends.","marker":"[55]"},{"why":"Supplies the triplet loss and triplet selection strategy that the deep TNN stage is built on.","marker":"[28]"},{"why":"Provides the triplet categorization and in-batch triplet mining details used in the ablation study.","marker":"[10]"},{"why":"Provides the VEGAS dataset, one of the two evaluation benchmarks and the source of the headline MAP numbers.","marker":"[60]"},{"why":"Provides the MV-10K dataset, the previous supervised deep CCA baseline, and prior comparative results.","marker":"[57]"}],"fun_headline_variants":["Deep triplet nets with Cluster-CCA boost audio-visual retrieval","TNN-C-CCA: Triplet loss + Cluster-CCA for audio-video retrieval","Triplet network beats CCA baselines in audio-video retrieval","Cluster-CCA plus triplet loss lifts audio-visual retrieval MAP","Audio-video retrieval improved by triplet Cluster-CCA embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported lead over baselines assumes that all comparison methods were implemented and tuned well enough that their hyperparameters are near-optimal, so the score gaps reflect method quality rather than undertuned competitors.","fun_headline_variants_meta":{"raw":{"variants":["Deep triplet nets with Cluster-CCA boost audio-visual retrieval","TNN-C-CCA: Triplet loss + Cluster-CCA for audio-video retrieval","Triplet network beats CCA baselines in audio-video retrieval","Cluster-CCA plus triplet loss lifts audio-visual retrieval MAP","Audio-video retrieval improved by triplet Cluster-CCA embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000773,"raw_usage":{"total_tokens":3478,"prompt_tokens":1056,"completion_tokens":2422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":2330}},"tokens_in":672,"tokens_out":2422,"duration_ms":16009,"temperature":1.0,"reasoning_tokens":2330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:03:28.143497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain every baseline on the same VEGAS and MV-10K folds with per-method hyperparameter search, especially C-DCCA's beta and the regularization of the hashing and adversarial baselines, and check whether TNN-C-CCA's MAP margins survive; a modest grid search that closes the 4.28-point gap on VEGAS would falsify the outperformance claim.","supporting_citations":[{"cited_title":"Hardoon, Sándor Szedmák, and John Shawe-Taylor","cited_arxiv_id":null,"evidence_quote":"Supplies the original CCA objective and the linear baseline that Cluster-CCA extends and that all variants are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Cluster-CCA, whose category-wise one-to-one correspondences form the projection stage that the triplet network refines."},{"cited_title":"Bilmes, and Karen Livescu","cited_arxiv_id":null,"evidence_quote":"Provides DCCA, the deep nonlinear CCA baseline and the basis for the C-DCCA comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines C-DCCA, the strongest baseline the paper must beat, and the category-correlation idea the method extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VEGAS dataset, one of the two evaluation benchmarks and the source of the headline MAP numbers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MV-10K dataset, the previous supervised deep CCA baseline, and prior comparative results."}],"review_version":1}