{"id":"1e766875-a464-4933-9969-96b8354c9876","arxiv_id":"2608.01310","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A frame-level audio-visual embedding that measures similarity over time-aligned frames beats pooled embedding and offset-prediction models on temporal and semantic retrieval, zero-shot event localization, and generation evaluation.","lead":"FATE is a new audio-visual model that keeps frame-by-frame information instead of squashing clips into a single vector, so it can tell both what a sound is and exactly when it happens. It outperforms existing methods on retrieving synced audio-video segments, localizing events without training on the target dataset, and scoring generated videos against human perception.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2) assumes index-proportional alignment equals physical-time alignment, but token timestamps are never measured; a constant offset would make FATE's temporal discrimination an artifact of its coordinate mapping.","rationale":"The paper's central contribution is that alignment on a physical timeline lets a single frame-level similarity encode both semantic and temporal discrimination. The weakest point is the alignment itself: Eq. (2) maps indices proportionally, but the paper never verifies that encoder tokens are uniformly spaced from a shared origin. The reader's weakest assumption identifies exactly this, and my reading sharpens it by noting that the temporal soft-contrastive loss is trained in the same index coordinate system, so a constant index-to-time bias would be invisible to training and to Fig. 3. This is a genuine validity gap for the claimed mechanism, but it is not a demonstrated failure: the model may still be well calibrated in practice, and the controlled ablations (w/o Frame-level, loss components, alignment strategy) are strong evidence that the frame-level design matters. Because the reader already returned CONDITIONAL and this concern is the same one, the verdict does not need to change; the concrete calibration probe would determine whether the concern lands. I found no more serious internal inconsistency or unsupported central claim than this alignment assumption.","tokens_in":16429,"tokens_out":12354,"duration_ms":123620,"concrete_test":"Run a calibration probe: take a 2s VGGSound clip and synthesize an aligned impulse pair (a single white frame and a 10ms audio click at the same time t, for t=0.25,0.5,...,1.75s). Feed the clip to the PE-AV encoders, locate the video token and the audio token whose features respond maximally, and record their physical timestamps. Then check whether Eq. (2) pairs tokens with equal timestamps for every t and compute the mean absolute time error. As a second test, apply known physical audio delays from -1s to +1s in 0.1s steps and find the delay at which S_AV is maximal. If the argmax deviates from 0 by more than half the audio token interval (or is clip-dependent), Eq. (2) is not a physical alignment and the central claim needs revision. If the error is within one audio hop, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Eq. (2): phi(i)=min(round(i*Tv/Ta), Tv), used as 'physical timeline' alignment before Eq. (3). This is a coordinate hypothesis, not a measured correspondence. It is valid only if video token centers and audio spectrogram-token centers are uniformly spaced with a common origin and token rates exactly Tv/duration and Ta/duration. The paper reports neither token timestamps nor receptive fields for the PE-AV encoders. If the audio encoder's first token center is offset by c (or if padding makes the offset clip-dependent), every 'aligned' pair in Eq. (3) is actually displaced by a constant physical delay. Since the temporal soft-contrastive loss (Eq. 7) labels candidates by their index shift in the same coordinate system, training is blind to a constant bias: FATE learns to peak at the biased offset, not at true synchrony. The temporal sensitivity curve (Fig. 3) also shifts in the same index space and therefore cannot reveal the bias. This does not invalidate the retrieval numbers, but it does mean the central claim that synchronization is encoded 'on the physical timeline' is unverified; the reported temporal discrimination could be an artifact of the index mapping. The w/o Frame-level ablation and other controlled comparisons are valuable, but they do not address this.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FATE, a dual-encoder audio-visual model that keeps frame-level token sequences instead of pooling to a global vector. It aligns the higher-rate video tokens to the audio token grid via nearest-neighbor downsampling (Eq. 2) and defines cross-modal similarity as the mean inner product over the aligned diagonal (Eq. 3). Training combines a cross-video semantic InfoNCE loss with a within-video temporal soft-contrastive loss using a raised-cosine target (Eqs. 4-7). The authors evaluate FATE on temporal retrieval (AVSync15, VGG-Sync), zero-shot event localization (AVE), and as a generation metric correlated with human ratings, reporting large gains over pooling-based embeddings and synchronization models, with ablations isolating the frame-level design, loss components, label softness, and alignment strategy.","tokens_in":16756,"tokens_out":11443,"duration_ms":96730,"significance":"If the results hold, FATE is a simple and potentially influential design: it shows that a general-purpose embedding space can carry synchronization without a dedicated offset prediction head, and the controlled 'w/o Frame-level' ablation, the loss-component ablation, and the code release are concrete strengths. The temporal discrimination is explicitly built into Eq. (3) and Eq. (6), so the sharp peak in Fig. 3 is a property of the design rather than an emergent discovery; the conclusion's 'emerge' wording should be softened. The central claims are not yet fully supported because the physical-timeline alignment in Eq. (2) is uncalibrated, the headline comparisons lack variance estimates, and the human-annotation protocol is described inconsistently.","major_comments":[{"comment":"The mapping phi(i)=min(round(i*Tv/Ta),Tv) assumes that the i-th video token and the i-th audio token are uniformly spaced from a common origin, with token rates exactly Tv/T and Ta/T for the same clip duration. The paper reports neither token timestamps nor receptive-field centers for the PE-AV encoders, so the 'aligned' pairs in Eq. (3) may be offset by a constant physical delay. Because the temporal loss in Eq. (7) labels candidates by index shift in the same coordinate system, training is blind to a constant bias, and the sensitivity curve in Fig. 3 cannot reveal it. This does not invalidate the relative retrieval results, but it means the central claim that synchronization is encoded 'on the physical timeline' is unverified; please measure or calibrate the token timestamps, or at least analyze sensitivity to a constant offset in phi.","section":"Method, 'Temporal alignment' (Eq. 2)"},{"comment":"The main text states that 'ten independent annotators rate the temporal synchronization of each video on a 1-5 scale, and we average their ratings into a Mean Opinion Score (MOS),' while the supplementary 'Human Annotation Protocol' describes a different procedure: annotators assign a full ranking from 1 (best) to 5 (worst) with no ties, the rankings are averaged into a mean rank, and the rank is negated before Spearman correlation. These are incompatible protocols, and Table 3 depends on which one was actually used. Please state the exact score used in Eq. (10), report inter-annotator agreement (e.g., Krippendorff's alpha), and make the main-text description consistent with the appendix.","section":"Joint Audio-Video Generation Evaluation / Appendix"},{"comment":"The abstract's 'surpasses supervised methods' claim rests on a 0.5-point average-accuracy margin over DAM (48.3% vs. 47.8% in Table 2), while FATE is actually worse on V2A (44.7% vs. 47.1%). With single runs and no significance tests, this margin cannot support 'surpasses'; please report variance over seeds or a paired test and, in the meantime, describe the result as matching or being on par with supervised methods.","section":"Abstract / Table 2"},{"comment":"All tables report a single run without error bars, confidence intervals, or significance tests. This is particularly limiting for Table 3, where the best sample-level correlation is 17.24 and the model-ranking correlation is computed over only five generation models; the paper acknowledges the latter limitation but not the absence of variance. Please report at least three seeds for the main experiments and a significance test for the comparisons that underlie the headline claims.","section":"Experimental Results / Tables 1-3"}],"minor_comments":[{"comment":"In Fig. 3, the shaded region is labeled ±1 std, but the text does not state whether this is across videos or across runs; please specify.","section":"Experimental Results, Fig. 3"},{"comment":"The phrase 'yields K=17 candidate audio clips' from a 10s video with a 2s window and 0.5s stride should show the arithmetic (10-2)/0.5+1.","section":"Training Strategy, temporal loss"},{"comment":"The loss weights in Eq. (4) are λ1=0.5 and λ2=1.0; a sentence justifying the 2:1 ratio or reporting sensitivity to it would help.","section":"Training Strategy, Eq. (4)"},{"comment":"The abstract and introduction both mention Omni LLMs as baselines, but the quantitative MLLM comparison is only in the appendix and is not summarized in a main-text table; please either add the numbers or move the claim to the appendix.","section":"Introduction / Appendix"},{"comment":"The table caption for Table 1 includes the ablation row 'w/o Frame-level' among the methods; please mark it explicitly as a controlled ablation rather than a baseline.","section":"Table 1 caption"},{"comment":"Eq. (6) cites Vatakis and Spence (2006) for the 0.5 s half-maximum point; please give the specific threshold reported there and note that the threshold may differ across stimulus types.","section":"Eq. (6) and psychophysical threshold"}],"recommendation":"major_revision","confidential_remarks":"The core idea is simple and the ablations are well controlled, so the paper is worth a major revision rather than rejection. The reviewers should be alert to the overclaiming around physical-time alignment and the 0.5-point 'surpasses' margin; both are fixable with additional measurements or softened claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The core idea is simple and mostly checks out: keep frame-level tokens instead of pooling, align video to audio with nearest-neighbor index mapping, and train with both semantic and temporal soft-contrastive losses. The ablations are clean and isolate the contribution. The main caveat is that the paper never verifies that the index mapping in Eq. (2) actually corresponds to physical time.\n\nThe combination is genuinely new relative to pooled embeddings and offset-only sync models. The w/o Frame-level ablation is a strong control: same backbone, data, and budget, only pooling vs. frame-level, and the gap is roughly 2x. The loss ablation shows the two objectives are complementary. The zero-shot localization result is impressive even if the margin over DAM is only 0.5 points on average.\n\nThe Eq. (2) concern from the stress test is real but not as fatal as it sounds. If the audio and video encoders have a constant offset, training is blind to it, and retrieval would systematically pick the wrong clip. Since FATE gets well above chance, the offset is probably small (tens of ms), but the paper doesn't measure it. A short paragraph with token timestamps or a sensitivity analysis would settle it. Also, the paper reports single runs without error bars; the 'surpasses' language on AVE should be softened to 'statistically indistinct' given 48.3 vs 47.8. The human annotation protocol is described as MOS rating in the main text but as ranking in the appendix; that needs to be fixed. Minor: the model-ranking rho is over only 5 generation models, which they do acknowledge.\n\nThis deserves a serious referee. The design is clean, the ablations are informative, and the code is promised. I'd condition acceptance on the calibration check, error bars, and fixing the MOS description. Anyone working on audio-visual retrieval, event localization, or generation evaluation should read it; I'd cite the frame-level similarity design.","headline":"FATE's frame-level design and ablations are the real contribution; the 'physical timeline' claim needs a calibration check, but the core result is solid.","tokens_in":17227,"tokens_out":6809,"would_cite":true,"duration_ms":62364,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FATE claims that frame-level embeddings, scored over physically aligned frame pairs, put semantic and temporal synchronization into one similarity measure.","keywords":["audio-visual embedding","frame-level representation","temporal synchronization","contrastive learning","cross-modal retrieval","event localization","generation evaluation","soft temporal labels"],"falsifier":"Take a controlled recording of a single percussive event with known audio and video timestamps, encode it with FATE, and shift the audio by known offsets before computing similarity. If the similarity peak appears at the true recorded offset rather than at the coordinate-defined zero, the alignment is physical; if it peaks at zero regardless of the true offset, Eq. (3) is measuring coordinate alignment rather than real-world simultaneity.","tokens_in":16264,"feed_emoji":"🎬","tokens_out":6606,"duration_ms":53467,"temperature":0.7,"pith_summary":"FATE aims to establish that one embedding space can carry both what a sound is and when it happens, without needing a task-specific prediction head. Instead of pooling each modality into a single vector, FATE keeps the full sequence of frame-level tokens, aligns video tokens to the audio timeline, and defines cross-modal similarity as the average inner product of corresponding frame pairs. That single score then does double duty: it identifies the semantically correct clip and also peaks only at zero temporal offset. The paper reports that this design outperforms both pooling-based embedding models and dedicated synchronization models on temporal retrieval, matches supervised methods on event localization without seeing that dataset's labels, and agrees better with human ratings than existing generation metrics.","feed_headline":"Frame-level embeddings synchronize audio and video in one score","feed_subtitle":"FATE keeps every frame instead of pooling, so matching content and matching timing collapse into one similarity score.","key_machinery":"The load-bearing mechanism is the frame-level diagonal similarity with nearest-neighbor temporal alignment. After discarding the global pooling head, the video encoder outputs $T_v$ tokens per clip and the audio encoder outputs $T_a$ tokens; the paper uses audio as the temporal anchor and downsamples video by nearest-neighbor selection to length $T_a$. Cross-modal similarity is then the mean diagonal of the pairwise inner-product matrix, $S_{AV} = \\frac{1}{T_a}\\sum_i (\\tilde{F}_V^{(i)})^\\top F_A^{(i)}$. This diagonal trace enforces temporal order by construction: a shift moves true correspondences off the diagonal, so the score decays with offset. The second piece is the training objective, which combines semantic InfoNCE across videos with a raised-cosine soft label over candidate offsets within a video, shaping a similarity landscape that peaks at zero offset and rolls off smoothly.","core_discovery":"FATE's central claim is that retaining frame-level embeddings and scoring similarity over temporally aligned frame pairs encodes semantic and temporal discrimination in one number. Concretely, given aligned visual features $\\tilde{F}_V$ and audio features $F_A$, similarity is $S_{AV} = \\frac{1}{T_a}\\sum_i (\\tilde{F}_V^{(i)})^\\top F_A^{(i)}$. A temporally shifted audio clip lowers $S_{AV}$ because the $i$-th video token no longer corresponds to the same physical moment, while a semantically different clip lowers $S_{AV}$ because the frame features do not match in content. Trained with a symmetric cross-video contrastive loss plus a within-video soft contrastive loss whose labels decay with absolute offset, FATE produces a similarity that peaks sharply at zero offset. The paper's empirical case is that this one representation surpasses the strongest baselines on temporal and semantic retrieval, matches fully supervised methods on zero-shot event localization, and is the best automatic proxy for human synchronization judgments among the metrics compared.","pith_inferences":["If the physical-timeline alignment is valid, the same diagonal-similarity construction could turn any pair of encoders with known token rates into a synchronization scorer, with no prediction head needed.","The soft-label kernel is anchored to a 0.5-second grid, so a finer test would probe whether FATE's similarity peak tracks human asynchrony thresholds below half a second, where the paper's chosen 1-second support begins to roll off.","Because FATE is initialized from a pretrained audio-visual encoder and trained on VGGSound, a transfer experiment to out-of-domain videos with independent ground-truth timestamps would test whether the learned alignment generalizes beyond the training distribution."],"forward_implications":["Temporal retrieval can be done with a reusable embedding: FATE reports gains of more than 13 points in R@3 over the strongest synchronization baseline while simultaneously raising semantic retrieval accuracy by over 20 points.","Zero-shot event localization reaches supervised-level accuracy: on the AVE dataset, FATE's 48.3% average exceeds the strongest supervised baseline's 47.8%, without using any AVE training labels.","Frame-level structure, not the encoder or training data alone, drives the gain: the controlled ablation that pools tokens instead of keeping frames loses roughly half of retrieval performance with everything else held fixed.","As an automatic evaluation metric for generated audio-video, frame-level synchronization scoring correlates better with human ratings (sample-level $\\rho=17.24$, model-ranking $\\rho=44.41$) than offset predictors or pooling-based similarity.","Semantic and temporal capabilities are complementary rather than in tension: ablations show each loss alone underperforms the joint objective, and the combined gain is super-additive."],"supporting_citations":[{"why":"Supplies the dual-encoder backbone that FATE keeps, with the global pooling head removed and frame tokens retained.","marker":"(Vyas et al. 2025)"},{"why":"Provides VGGSound, the large training set whose 2-second segments FATE fine-tunes on and whose categories define the semantics.","marker":"(Chen et al. 2020)"},{"why":"Defines the synchronization-model baseline and offset-predictor metric that FATE must beat or outperform.","marker":"(Iashin et al. 2024)"},{"why":"Provides CAVP, the embedding-based synchronization baseline and generation metric that FATE compares against.","marker":"(Luo et al. 2023)"},{"why":"Supplies AVSync-15, the test benchmark used for temporal retrieval and human-opinion correlation.","marker":"(Zhang et al. 2024)"},{"why":"Gives the InfoNCE contrastive objective that FATE's cross-video semantic loss is built on.","marker":"(Oord, Li, and Vinyals 2018)"},{"why":"Motivates the 1-second support of the soft temporal label, matching the psychophysical threshold for perceivable audio-visual asynchrony.","marker":"(Vatakis and Spence 2006)"},{"why":"Provides the AVE dataset and the supervised event-localization baselines that FATE is compared against in zero-shot mode.","marker":"(Tian et al. 2018)"}],"fun_headline_variants":["Frame-level embedding: one score for what and when","FATE: align audio and video frame by frame, one similarity","No pooling: FATE's frame-wise score nails timing and content","One similarity for semantic and temporal audio-visual match","Audio-visual embedding that keeps every frame: timing and meaning aligned"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The alignment step assumes that after nearest-neighbor downsampling, the $i$-th video token and the $i$-th audio token refer to the same physical instant for every $i$; if the two encoders' time origins or token centers are offset, the 'aligned' pairs are not simultaneous and the reported temporal discrimination would be an artifact of the coordinate mapping.","fun_headline_variants_meta":{"raw":{"variants":["Frame-level embedding: one score for what and when","FATE: align audio and video frame by frame, one similarity","No pooling: FATE's frame-wise score nails timing and content","One similarity for semantic and temporal audio-visual match","Audio-visual embedding that keeps every frame: timing and meaning aligned"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1402,"prompt_tokens":977,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":593,"tokens_out":425,"duration_ms":4210,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:08:35.067780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a controlled recording of a single percussive event with known audio and video timestamps, encode it with FATE, and shift the audio by known offsets before computing similarity. If the similarity peak appears at the true recorded offset rather than at the coordinate-defined zero, the alignment is physical; if it peaks at zero regardless of the true offset, Eq. (3) is measuring coordinate alignment rather than real-world simultaneity.","supporting_citations":[{"cited_title":"VGGSound: A Large-scale Audio-Visual Dataset","cited_arxiv_id":null,"evidence_quote":"Provides VGGSound, the large training set whose 2-second segments FATE fine-tunes on and whose categories define the semantics."}],"review_version":2}