{"id":"86ff6630-7bb1-4a27-8186-e765a1c1e719","arxiv_id":"2607.23855","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation improves downstream text-to-audio-video synchronization and quality.","lead":"The authors train a video and an audio VAE together so their latent spaces line up at the level of short time segments, then show that a text-to-audio-video generator built on these latents produces better synchronization and audio quality. The work matters because it tests whether cross-modal alignment should live in the tokenizer instead of being left to the generative model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contrastive gains are confounded by unequal training schedules (steps/clip length/freeze stage) and initialization, so the causal role of AVCLIP is not yet isolated.","rationale":"The reader's verdict (CONDITIONAL, medium risk) is reasonable, but their weakest_assumption focuses on data filtering quality. I agree that a noisy corpus could affect the contrastive signal, but even with perfect data, the experimental design has a more fundamental threat: the joint variants differ from the per-modality baselines in training length, clip context, and schedule, making it impossible to attribute the observed gains specifically to the contrastive objective. This is load-bearing because the paper's central claim is causal (alignment improves learnability). The proposed Joint-Recon baseline directly tests this by removing only the contrastive term while holding the joint schedule constant. This would settle whether the DeSync/LSE-C/WER improvements in Table 4 come from cross-modal alignment or from longer clips/joint optimization dynamics. The reader's other concerns (selection bias, lack of error bars, in-house data) are valid but secondary; the training confound is more decisive. Hence I recommend keeping CONDITIONAL but note that this additional ablation is mandatory for acceptance, not optional. My read partially agrees with the reader because we identify different weakest points; however, the data-quality concern and the training-confound concern are complementary, and both suggest the current evidence is not fully clean. I do not propose REJECT because the paper's method is plausible and the public code/models could enable the necessary checks.","tokens_in":14785,"tokens_out":7477,"duration_ms":82640,"concrete_test":"Run a 'Joint-Recon' baseline using the exact joint schedule (82k steps, 193-frame/8s clips, batch 256, two-stage freeze/unfreeze, same decoder fine-tuning) but set λ_avclip=0, keeping all other losses. Compare its Table 3/4 metrics to Recon+AVCLIP and OmniVAE; also retrain Recon+AVCLIP initialized from the same checkpoint as the other joint variants and report all initializations. If Joint-Recon matches the AVCLIP variants on DeSync/LSE-C/WER, the claimed benefit of contrastive alignment is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that the segment-level InfoNCE objective improves latent 'learnability'—is not cleanly identified by the ablation. In §4.1, the per-modality baselines (Recon, Recon+Distill) are trained for 250k steps on 121-frame clips and independently, whereas the joint variants (Recon+AVCLIP, OmniVAE) train for 82k steps on 193-frame clips with a two-stage freeze/unfreeze schedule and without audio discriminators during joint training. Longer clips and the joint schedule themselves could improve temporal coherence, audio quality, and downstream sync metrics, independent of the contrastive loss. The initialization also differs: OmniVAE starts from Recon+Distill checkpoints, while the paper is silent on what checkpoint initializes Recon+AVCLIP, so the full model's advantage over AVCLIP may be inherited from distillation rather than from joint optimization. Without a joint-training baseline that omits only the contrastive loss, the measured DeSync and LSE-C gains (Table 4) cannot be attributed to cross-modal alignment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OmniVAE, a jointly trained audio-video VAE that augments per-modality reconstruction with two training-only objectives: a segment-level bidirectional InfoNCE contrastive loss (Eq. 1) that aligns audio and video latents at fine temporal granularity, and modality-specific semantic distillation from frozen Qwen3-Omni encoders. The authors argue that these objectives make the latent spaces more 'learnable' for downstream generative models, and evaluate this through sync probing on VGS-Sp (§4.3) and by training frozen-VAE text-to-audio-video (T2AV) models (§4.4). The internal ablation compares Recon, Recon+Distill, Recon+AVCLIP, and OmniVAE, reporting improved cross-modal alignment (e.g., DeSync 0.884→0.570, LSE-C 1.479→2.093, WER 0.243→0.168 in Table 4) and better audio quality for OmniVAE. The paper also includes ablations of contrastive pool size/temporal granularity (§4.5.1) and loss-balancing strategies (§4.5.2), and releases code and model weights.","tokens_in":15020,"tokens_out":5626,"duration_ms":51664,"significance":"If the causal claims survive scrutiny, the contribution is valuable: it shows that tokenizer-level, training-only cross-modal alignment can reduce the burden on downstream joint audio-video generators, with no inference overhead. The paper is also commendable for its large-scale 23M-clip corpus, the two complementary objectives, and the evaluation spanning latent probing, reconstruction, and downstream generation. However, the central attribution of the measured gains to the contrastive objective is currently not cleanly identified: the ablation uses unequal training schedules, clip lengths, and initializations across conditions, and the sync-probe hyperparameters are selected on the same benchmark used for headline sync results. The significance is therefore conditional on additional controlled experiments.","major_comments":[{"comment":"The ablation does not isolate the contrastive objective. Recon and Recon+Distill are trained for 250k steps on 121-frame clips independently, while Recon+AVCLIP and OmniVAE train for 82k steps on 193-frame clips with a two-stage freeze/unfreeze schedule, without audio discriminators during joint training, and with an additional decoder fine-tuning stage. Larger temporal context, joint optimization, and the schedule itself are plausible causes of the DeSync/LSE-C/WER gains in Table 4, independent of Eq. (1). A control variant trained jointly with the same clip length and schedule but omitting ℒ_avclip is needed; as written, the headline comparisons cannot be attributed to cross-modal alignment.","section":"§4.1, §3.4"},{"comment":"The hyperparameters (FPS, clip duration, segment length) are selected using VGS-Sp A@1, which is the same benchmark used to report the headline sync-probing results in Table 3. This creates a selection-circularity risk; the reported 30.0 A@1 configuration may partly reflect fitting to this evaluation set. Please perform hyperparameter selection on a held-out validation subset and disclose the amount of selection. Also, row '24/—/0.17s' (A@1=7.0) versus '24/2s/0.17s' (A@1=6.0) shows that contrastive training with 2s clips is below the no-contrastive baseline; this non-monotonicity is not discussed and weakens the narrative that larger negative pools uniformly improve alignment.","section":"§4.5.1, Table 5"},{"comment":"The VGS-Sp probe is a temporal-offset classification task, which is essentially the same discrimination the segment-level InfoNCE loss optimizes: the objective already uses intra-clip negatives at nearby temporal positions. Table 3's large A@1 gains (6.4→20.2 frozen) may therefore largely reflect the probe measuring a direct readout of the training objective rather than an independent capability. The paper should rely primarily on the downstream T2AV metrics for the causality claim, or add a probe task that is not a test-time version of the trained objective, and temper the statement that 'contrastive learning provides the main gain in temporal alignment.'","section":"§4.3 vs. §3.2/Eq. (1)"},{"comment":"The 23M-clip training corpus is filtered using ImageBind semantic alignment, DeSync-based synchronization filtering, and BEATs event detection. If these filters are aggressive, the surviving training distribution is already strongly aligned, so a contrastive objective may preserve filtering-induced alignment rather than create it. Please report filtering retention rates and the distribution of DeSync/alignment scores before and after filtering, or train on an unfiltered (or less filtered) control subset, to show the objective is effective beyond cleaning artifacts.","section":"§4.1, Datasets"}],"minor_comments":[{"comment":"The abstract contains an orphan footnote marker '1' with no corresponding footnote text; please remove it or add the footnote.","section":"Abstract"},{"comment":"The GradNorm variant is described only as a 'last-layer proxy'; please give a one-sentence explanation or citation for the proxy, since it is not self-explanatory.","section":"§4.5.2, Table 6"},{"comment":"No confidence intervals, standard errors, or significance tests are reported; given that metrics are averaged over only three checkpoints, please add variance estimates or at least state whether the differences are stable across checkpoints.","section":"§4.4, Table 4"},{"comment":"The text says ℒ_avclip appears in both objectives but is evaluated only once; this is clear in context, but the wording could be tightened to avoid implying two separate contrastive computations.","section":"§3.4, Eq. (3)"},{"comment":"The caption 'The dash denotes the reconstruction-only baseline without contrastive training' is ambiguous: it is unclear whether the dash in the Clip column or elsewhere denotes the baseline, and the table is easier to read if the baseline row is explicitly labeled.","section":"§4.5.1, Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a strong engineering contribution with a plausible and interesting idea. My recommendation rests on the fact that the central causal claim is currently confounded by unequal training schedules and selection on the evaluation benchmark. If the authors add a joint-training control without the contrastive loss, perform hyperparameter selection on a held-out set, and report data-filtering retention rates, I would expect the paper to be acceptable. The downstream T2AV improvement is real and useful, but the attribution needs to be cleaner."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. You should know: this is a genuinely useful system paper, but the headline claim—the contrastive loss (AVCLIP) is what improves downstream sync—is not cleanly identified by the experiments.\n\nThe ablation has a confound: Recon and Recon+Distill run 250k steps on 121-frame clips; Recon+AVCLIP and OmniVAE run 82k steps on 193-frame clips, with a freeze/unfreeze schedule and no audio discriminator during joint training. There is no joint-training baseline that omits only the contrastive loss. So the large DeSync/LSE-C gains in Table 4 could come from longer clips, more temporal context, the schedule, or the contrastive loss. The direction is consistent across many metrics, which is reassuring, but for a paper centered on putting alignment in the tokenizer, this is the load-bearing comparison.\n\nSecond, the sync probe is close to the training objective: it asks a classifier to predict temporal offset, which is nearly the same task as the segment-level InfoNCE loss. Hyperparameters (fps, clip duration, segment length) were also selected on the same VGS-Sp probe used to report the headline sync gains in Table 3. That bias mostly inflates the sync-probe numbers, not the downstream T2AV numbers.\n\nWhat is genuinely good: as a system, the combination of semantic distillation and segment-level contrastive alignment in a jointly trained VAE is new. The two objectives look complementary—distillation helps audio quality, AVCLIP helps alignment, and the full model is best overall. Reconstruction stays competitive with strong external VAEs. Figure 2, tracking metrics over training steps, is a nice piece of evidence, and the code and model weights are released.\n\nThe T2AV table has no error bars or significance tests, and the downstream generator is in-house, so an external T2AV baseline would help put the absolute numbers in context. But the internal comparisons are directionally consistent, so that is a moderate concern, not a fatal one. The data-filtering pipeline is also in-house and not released, which is minor-to-moderate.\n\nSerious thinker: yes—the paper is coherent, the schedule limitations are stated, and the discussion of prior work is fair.\n\nRecommendation: send it to peer review. The system is novel, the code is out, and the main confound is fixable. Require a joint-trained baseline without AVCLIP (same clip length and schedule) as a condition for acceptance.","headline":"A useful system paper on jointly trained audio–video VAE with semantic distillation and segment-level contrastive alignment, but the ablation does not isolate the effect of the contrastive loss from the longer clips and different training schedule, so the causal claim is weaker than the abstract suggests.","tokens_in":15633,"tokens_out":4943,"would_cite":true,"duration_ms":54487,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that building cross-modal alignment directly into an audio-video VAE's latent spaces—via segment-level contrastive learning and semantic distillation—makes downstream joint generation better synchronized and higher quality,","keywords":["audio-video generation","cross-modal alignment","contrastive learning","semantic distillation","VAE tokenizer","latent space learnability","text-to-audio-video generation","synchronization"],"falsifier":"Train OmniVAE on deliberately shuffled or weakly aligned audio-video pairs under the same objectives; if sync-probing and downstream generation synchronization gains persist, the alignment objective is not the active ingredient. Alternatively, run the frozen OmniVAE latents through a temporal-offset probe on an out-of-distribution dataset with natural audio-video offsets and see whether the fine-grained discrimination survives.","tokens_in":14624,"feed_emoji":"🎬","tokens_out":8484,"duration_ms":68045,"temperature":0.7,"pith_summary":"The paper tries to establish a design principle: a tokenizer for joint audio-video generation should not only reconstruct well, but also structure its latent spaces semantically and align them across modalities. OmniVAE is a jointly trained audio-video VAE that adds two training-only objectives to reconstruction: segment-level audio-video contrastive learning, which pulls temporally matching audio and video segments together in a shared embedding space using a bidirectional InfoNCE loss, and per-modality semantic distillation, which injects features from frozen pretrained encoders into each branch. The authors show that these objectives improve the learnability of the latents: a downstream text-to-audio-video model trained on OmniVAE's frozen latents achieves better synchronization, lower word error rate, and better audio quality than models trained on reconstruction-only latents, while reconstruction quality barely drops. If the claim holds, the practical message is that cross-modal correspondence can be paid for once, at tokenizer training time, instead of being re-learned by every downstream generator.","feed_headline":"Aligned audio-video VAE cuts sync error in joint generation","feed_subtitle":"Aligned latent spaces give downstream models better synchronization and speech clarity with no extra inference cost.","key_machinery":"The two load-bearing mechanisms are (1) a segment-level audio-video contrastive head that aggregates latent frames into temporally aligned one-sixth-second segment embeddings and applies a bidirectional InfoNCE loss over a hierarchical negative pool, and (2) per-modality semantic distillation projectors that align each VAE latent to features from frozen pretrained semantic encoders via a sigmoid-cosine loss. The contrastive objective carries the cross-modal alignment claim; the distillation objective carries the per-modality learnability claim. Both are training-only: the contrastive head and projectors are discarded at inference, so the deployed tokenizer is just the two VAEs with no extra","core_discovery":"The central claim is that fine-grained semantic and temporal correspondence between audio and video can be made an intrinsic property of VAE latent representations rather than something a downstream generative model must infer from scratch. OmniVAE keeps separate audio and video encoders and decoders but trains them jointly with a segment-level bidirectional contrastive objective over one-sixth-second segments, using negatives drawn from the same clip, sibling clips of the same source video, and other videos; this forces the latents to encode temporal-semantic alignment. In parallel, distillation losses push each branch's latents toward the feature geometry of frozen pretrained semantic enco","pith_inferences":["The training corpus underwent aggressive filtering for semantic alignment and synchronization, so the measured gains may partly reflect cleaner training data rather than the contrastive objective per se; an ablation on unfiltered pairs would separate the two effects.","The claim is evaluated on the same type of concept-filtered web data used for training; a natural next test is whether the alignment transfers to in-the-wild videos with realistic audio-visual offsets and mixed sound sources.","The same recipe—distillation plus segment-level contrastive alignment—could generalize to other modality pairs as a strategy for making tokenizers generation-ready, but the paper only demonstrates the audio-video case.","The modest sync-probing gain from distillation alone suggests the modality-specific teachers contribute mainly to per-modality quality rather than temporal alignment; synchronization-aware teachers or multi-scale contrastive heads may be a useful extension."],"forward_implications":["A downstream text-to-audio-video generator can start from latents that already encode cross-modal correspondence, so it does not have to rediscover synchronization during diffusion training.","Video-to-audio systems that currently rely on externally computed synchronization features could instead use the tokenizer's aligned latents, shrinking the gap between training and inference conditioning.","The two objectives are complementary: contrastive alignment drives temporal synchronization gains, semantic distillation drives modality-specific quality, and using both yields the strongest aggregate result rather than either alone.","Reconstruction quality is largely preserved when the alignment objectives are added, so the approach is compatible with standard VAE-based generation pipelines and can serve as a drop-in replacement tokenizer."],"fun_headline_variants":["Joint audio-video VAE aligns latents for better sync","One VAE for audio and video: alignment beats separate training","OmniVAE: shared latent space cuts sync errors in A/V generation","Cross-modal VAE: contrastive training improves joint A/V quality","Audio-video VAE with segment contrastive learning boosts sync"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the roughly 23 million training clips are correctly synchronized and semantically paired in fine temporal detail, so the contrastive objective's positives are true positives; if a non-negligible share of clips is misaligned, the model learns false associations and the reported synchronization gains may be inflated by data cleaning.","fun_headline_variants_meta":{"raw":{"variants":["Joint audio-video VAE aligns latents for better sync","One VAE for audio and video: alignment beats separate training","OmniVAE: shared latent space cuts sync errors in A/V generation","Cross-modal VAE: contrastive training improves joint A/V quality","Audio-video VAE with segment contrastive learning boosts sync"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1101,"prompt_tokens":714,"completion_tokens":387,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":458,"tokens_out":387,"duration_ms":4006,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:44:50.042210+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train OmniVAE on deliberately shuffled or weakly aligned audio-video pairs under the same objectives; if sync-probing and downstream generation synchronization gains persist, the alignment objective is not the active ingredient. Alternatively, run the frozen OmniVAE latents through a temporal-offset probe on an out-of-distribution dataset with natural audio-video offsets and see whether the fine-grained discrimination survives.","supporting_citations":[],"review_version":3}