{"id":"87f22bc1-2ed0-4429-9b86-b5570d45ae3e","arxiv_id":"2602.12304","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A zero-shot model that generates a video of a reference face speaking user-chosen text with a reference voice timbre.","lead":"OmniCustom generates a short video of a specific person speaking new text in a specific voice, using one reference photo and one reference audio clip. It is the first system to combine identity-preserving video generation with timbre cloning and text-controlled speech in a single joint audio-video model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contrastive loss in Eq. (8) may rely on a degenerate no-reference baseline; Table 2's gains lack error bars, so the central SOTA claim is not yet established.","rationale":"The reader's weakest assumption correctly identifies the contrastive objective as the key risk. I add specificity: Eq. (8)'s negative is self-referential and stop-grad, and the paper provides no stability analysis of the no-reference flow. The benchmark is small, and the ablation numbers are the only evidence for the objective's efficacy. A bootstrap CI and a fixed-negative ablation would settle whether the gains are real or an artifact. These concerns do not overturn the reader's CONDITIONAL verdict; they reinforce it.","tokens_in":20389,"tokens_out":12976,"duration_ms":143971,"concrete_test":"Re-run the §6.4 ablation with the stop-gradient negative replaced by a frozen pretrained OVI (no LoRA, no references) as a fixed negative, and compute bootstrapped 95% CIs over the 100 benchmark identities for FaceSim-Arc and Speaker-Sim. If the '+ Contrastive' row no longer significantly exceeds '+ Face&Timbre' row, the claimed benefit is an artifact of the self-referential stop-gradient negative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is the contrastive objective (Eq. 8) which pushes the reference-conditioned velocity field away from the no-reference velocity field. For this to be meaningful, the no-reference flow must be a stable, non-degenerate baseline. The paper never characterizes this flow; it is computed by the same network with masked attention and zeroed embeddings, and a stop-gradient on the negative. If that baseline is high-variance or poorly calibrated, the L2 repulsion can inflate the distance to a noisy predictor, improving face-similarity and speaker-similarity metrics without actually transferring to new identities or timbres. Moreover, the reported ablation gains (FaceSim-Arc 0.39→0.48→0.60 in Table 2) come from a 100-example benchmark with 3 seeds and no confidence intervals or significance tests. Thus the evidence that the contrastive loss—rather than the added face/timbre embeddings or generic training dynamics—causes the improvement is not conclusive, and the SOTA claim rests on this unresolved dependency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 'sync audio-video customization,' a new task in which a model receives a reference image and a reference audio clip and must produce a video that preserves the identity of the image while imitating the timbre of the audio, with the spoken content specified by a free-form text prompt. The proposed system, OmniCustom, extends the OVI joint audio-video diffusion transformer by adding reference-image and reference-audio branches with LoRA adapters in self-attention layers, plus global face and timbre embeddings. A 'contrastive' regularizer (Eq. 8) pushes reference-conditioned flows away from no-reference flows. The model is trained on a new 1M-clip dataset derived from SpeakerVid-5M, and evaluated on a self-built 100-example benchmark with video, audio, and TTS baselines. The paper reports state-of-the-art identity preservation, timbre cloning, and video quality, with a user study showing preference over prior methods.","tokens_in":20719,"tokens_out":3833,"duration_ms":40983,"significance":"If the reported results hold, this is a genuinely new capability: simultaneous zero-shot identity and timbre customization with free text control, enabled by a clean architectural extension of an open audio-video generation model. The dataset construction and the use of an existing TTS benchmark for timbre comparison are useful contributions. The paper also provides ablations isolating the contribution of the embeddings and the contrastive loss, and includes a user study. However, the central SOTA claim currently rests on a small, unreleased benchmark with no statistical inference, and the novel objective in Eq. (8) is not characterized enough to establish that it provides the claimed benefit rather than a training artifact.","major_comments":[{"comment":"The paper's headline claim of state-of-the-art performance rests on a self-built benchmark of only 100 examples (300 videos with 3 seeds) and reports no confidence intervals, error bars, or significance tests. The differences in FaceSim-Arc (e.g., 0.60 vs. 0.49 for ConsisID) and FID/FVD may be within noise, especially given that the benchmark construction is not publicly released. I request error bars, statistical significance tests (e.g., paired bootstrap or Wilcoxon), and release of the benchmark to make the SOTA claim verifiable.","section":"§6.3, Table 2"},{"comment":"The 'contrastive' identity/timbre objective is a stop-gradient L2 repulsion from the no-reference flow. The paper does not characterize this no-reference flow: it is obtained by the same network with masked attention and zeroed embeddings, and the baseline may be high-variance or poorly calibrated. If so, the repulsion term can inflate distances to a noisy predictor and improve FaceSim/Speaker-Sim on the small benchmark without transferring to new identities. The paper should analyze the no-reference flow (e.g., variance, stability), test the objective without the stop-gradient, and verify that the gains persist on a larger or cross-session set. Also note that the objective is not 'contrastive' in the standard sense—there is no positive pair; the positive examples are only 'implied' in the flow-matching loss, so the terminology may mislead.","section":"§4.2.3, Eq. (8)"},{"comment":"Training reference and target clips are sampled from the same source video (first 4s vs. last 5s). This pairing may teach the model to exploit recording/session conditions rather than speaker identity and timbre alone. More importantly, the Speaker-Sim evaluation in Table 2 is ambiguous: it compares the synthesized speech with 'ground-truth speech segments,' but for a freely specified text prompt there is no natural ground truth. If the comparison is against audio from the same source video as the reference, the metric may be inflated by channel/room acoustics. Please clarify the exact protocol and, ideally, construct a benchmark where reference and target are from different sessions/recordings.","section":"§5.3 and §6.1"},{"comment":"The paper presents OmniCustom-1M as a core contribution, but the dataset is not released, and the 100-example benchmark is not released. This makes the reported results impossible to reproduce or compare against by other groups. Given that the central claim depends on this data, I consider a release (even a subset or detailed metadata) and the benchmark essential for the claim to be substantiated.","section":"§5 and §6.1"}],"minor_comments":[{"comment":"The equation has a mismatched parenthesis: `StopGrad(v_θ(Z_ti, φ, C, ti)||^2` is missing a closing `)`. Please fix.","section":"§4.2.3, Eq. (8)"},{"comment":"'Positive examples are implied in the flow matching objectives' is vague. Clarify how the flow-matching loss acts as a positive term and why this design is preferable to explicit positives.","section":"§4.2.3"},{"comment":"Typo: 'the first to propose sync' is missing a space. Also, the sentence 'proposes a more compelling new task' is subjective; consider rewording.","section":"§1, Table 1"},{"comment":"Reference [6] has the author name 'JianZhao JianZhao' duplicated. Please correct.","section":"§2, References"},{"comment":"The statement 'timbre embeddings and contrastive timbre loss boost timbre similarity by 31.0% and 23.7%' should specify whether this is a relative or absolute improvement and what the baseline value is, for reproducibility.","section":"§6.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and useful task, and the architecture is plausible. My main concern is that the empirical evidence for SOTA is not yet rigorous: a 100-example unreleased benchmark without error bars or significance tests, an ambiguous Speaker-Sim protocol, and an uncharacterized 'contrastive' objective. I would want to see either a larger and released evaluation, or at least a substantial clarification of the evaluation protocol and analysis of the negative flow, before accepting. The dataset release is also important for the community to build on this work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe first thing to know is that this paper defines a new task — sync audio-video customization — and builds a working system for it. Given a reference image, a reference audio, and a text prompt, it generates a video with the image's identity speaking the prompt's text in the audio's timbre. That's a real gap in the literature: prior video customization methods ignore timbre, and audio-driven methods lock the speech to the input audio. The task formulation alone is worth a paper.\n\nThe method is coherent. They start from OVI, add two LoRA branches for identity and timbre, and train with flow matching plus a contrastive regularizer that repels the reference-conditioned flow from a no-reference baseline. The ablations are internally consistent: adding face/timbre embeddings and then the contrastive loss steadily improve FaceSim and Speaker-Sim. The qualitative results look plausible, and the supplementary material shows cross-gender timbre transfer and a failure case with profile faces, which suggests honest reporting.\n\nThe soft spots are real but addressable. The evaluation is the biggest one: a self-built 100-example benchmark, 3 seeds, no confidence intervals, no significance tests. The FaceSim-Arc improvement from 0.39 to 0.60 is large, but I can't tell if it's robust. For a SOTA claim, that's insufficient. The second issue is the contrastive loss itself. Eq. (8) maximizes the distance to a no-reference flow computed with masked attention and zeroed embeddings from the same network. The paper never characterizes that baseline. If it's a noisy predictor, the repulsion could inflate the gap without improving transfer. That concern is not fatal — the flow-matching loss anchors the reference-conditioned output, and the repulsion weight is only 0.1 — but it deserves analysis. A third minor issue: reference and target clips come from the same video, so timbre similarity may partly measure recording consistency rather than voice identity.\n\nWho's this for? Anyone working on joint audio-video generation or identity-customized video. The task definition is useful even if the system results are unverified. I'd bring it to a reading group, and I'd cite it for the task alone.\n\nRecommendation: this deserves a serious referee. The method is clear, the task is new, and the weaknesses are fixable with more rigorous evaluation and a sensitivity analysis of the contrastive loss. Don't desk-reject it.","headline":"A new task and a plausible method, but the SOTA claim rests on a 100-example benchmark with no error bars.","tokens_in":21133,"tokens_out":4143,"would_cite":true,"duration_ms":41113,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OmniCustom claims a single zero-shot model can generate a talking video that keeps a reference face while cloning a reference voice, with the spoken words supplied as text.","keywords":["sync audio-video customization","identity preservation","timbre cloning","zero-shot generation","LoRA","flow matching","contrastive learning","joint audio-video generation"],"falsifier":"Run a controlled swap: use reference audio from speaker A and a prompt with content spoken by speaker B. If the generated audio's speaker embedding is no closer to A than to a random held-out speaker—while word error rate stays low—then the timbre-cloning claim fails. Similarly, if the contrastive loss's gains vanish when the no-reference baseline is replaced by a well-trained text-only flow, the regularization is not doing what the paper says.","tokens_in":20333,"feed_emoji":"🎬","tokens_out":4658,"duration_ms":45170,"temperature":0.7,"pith_summary":"This paper introduces a new task, sync audio-video customization: from one reference image and one reference audio clip, generate a video whose face matches the image, whose voice matches the timbre of the audio, and whose spoken content is whatever the user types. It then presents OmniCustom, a diffusion-transformer audio-video model fine-tuned to do this zero-shot. The core mechanism is to feed the reference image and audio as static tokens through separate low-rank adapters in the model's self-attention layers, and to add a contrastive objective that pushes reference-conditioned predictions away from no-reference predictions. Trained on a constructed one-million-clip talking-head dataset, the model is reported to beat existing video-customization systems on identity and quality metrics while matching dedicated TTS systems on timbre cloning. If true, this collapses the usual split between video customization and voice cloning into one flexible pipeline.","feed_headline":"Face from one photo, voice from one clip, words from your text","feed_subtitle":"OmniCustom claims a zero-shot model that clones identity and timbre in one pass, with free text control of the speech.","key_machinery":"Two independent LoRA adapters (one for reference-image tokens, one for reference-audio tokens) inserted into the QKV projections of self-attention in the video and audio branches of a joint audio-video diffusion transformer. The reference tokens are given a static time step of 0, so they act as time-invariant conditioning. Around this, the paper adds a contrastive regularization term: the velocity field conditioned on references (positive) is pushed away from the velocity field with references masked out and embeddings zeroed (negative), with a stop gradient on the negative. The face/timbre embeddings enter as one-global-token conditions. These pieces together are what the paper credits for","core_discovery":"The central claim is that identity and timbre can be synchronized in a single generative pass rather than stitched from separate video-customization and TTS models. The paper argues that a joint audio-video diffusion transformer can absorb both conditions at once: the reference image is injected into the video branch's self-attention and the reference audio into the audio branch's self-attention, each through low-rank adapters, with face and timbre embeddings added as global tokens. A contrastive flow-matching loss then regularizes the velocity field by repelling the reference-conditioned flow from the text-only, no-reference flow. The authors report that this combination yields the best ide","pith_inferences":["The paper leaves open whether the same recipe extends beyond 5-second English clips; testing the LoRA-plus-contrastive approach on longer or multilingual generations is a direct next step.","A sharper test of the timbre claim would be cross-gender and cross-accent references: the paper shows one cross-gender example and notes a slight timbre drop, suggesting the model carries implicit gender priors.","If the contrastive term is doing real work, it should also improve generalization to unseen identities; a held-out identity study with error bars on the 30 unseen persons would make that visible.","The same-video pairing of reference and target could be probed by swapping reference audio from a different speaker while keeping content; if speaker similarity stays high, the model may be copying prosody rather than timbre."],"forward_implications":["Video customization stops being a silent-film technology: the same model supplies synchronized speech with freely editable wording.","Voice cloning no longer needs a dedicated TTS stack; a joint audio-video model can carry timbre alongside identity, and even produce text-relevant background sound.","The contrastive flow-matching trick may transfer to other reference-conditioned generation tasks that need strong separation from no-reference baselines.","A single benchmark with 100 held-out identities and a 1M-clip dataset is proposed as a foundation for comparing sync audio-video customization systems.","The reported results suggest that modeling the joint distribution of video and audio is better for lip-sync than modeling audio-driven conditional video generation."],"fun_headline_variants":["One pass: clone face, timbre, and speech from references","Sync identity and timbre in a single zero-shot pass","Joint model matches face and voice from one image and clip","Customize video and audio together with free text control","Zero-shot: generate video that mimics identity and timbre"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a reference clip and a training clip drawn from the same video teach timbre rather than incidental recording conditions, and that the no-reference flow is a meaningful negative rather than a degenerate baseline.","fun_headline_variants_meta":{"raw":{"variants":["One pass: clone face, timbre, and speech from references","Sync identity and timbre in a single zero-shot pass","Joint model matches face and voice from one image and clip","Customize video and audio together with free text control","Zero-shot: generate video that mimics identity and timbre"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1515,"prompt_tokens":820,"completion_tokens":695,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":623}},"tokens_in":564,"tokens_out":695,"duration_ms":7325,"temperature":1.0,"reasoning_tokens":623,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:05:52.844661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled swap: use reference audio from speaker A and a prompt with content spoken by speaker B. If the generated audio's speaker embedding is no closer to A than to a random held-out speaker—while word error rate stays low—then the timbre-cloning claim fails. Similarly, if the contrastive loss's gains vanish when the no-reference baseline is replaced by a well-trained text-only flow, the regularization is not doing what the paper says.","supporting_citations":[],"review_version":1}