{"id":"078c9cda-9ad2-4056-879c-b85c69df8ef3","arxiv_id":"2506.00830","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An audio-conditioned video diffusion transformer that animates portraits from image, video, text, and audio inputs with a sliding-window fusion for long videos.","lead":"SkyReels-Audio is a system that generates talking-portrait videos from a still image, video, text, and audio, aligning lip movement and expressions to the sound using a video diffusion transformer. It targets long and controllable portrait videos, which matters for dubbing, virtual avatars, and digital storytelling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 contradicts the Abstract's 'superior performance' claim: OmniHuman-1 beats SkyReels-Audio on all four metrics on the internal benchmark.","rationale":"The reader's weakest assumption concerned the reliability and impartiality of the evaluation pipeline, including SyncNet metrics and the internal benchmark. My stress-test found a stronger and more specific problem in the same evidence: the paper's own Table 3 shows that on the internal benchmark, described as 'complex and challenging conditions,' OmniHuman-1 outperforms SkyReels-Audio on all four reported automatic metrics. This is not a question of external standards or missing error bars; the reported numbers themselves contradict the Abstract's blanket claim of 'superior performance.' The load-bearing condition for the central claim is that SkyReels-Audio is at least competitive with the strongest baselines on its own evaluation; that condition fails. I therefore recommend rejecting the current version, since the headline result is directly refuted by the manuscript's data, though the authors could resubmit with corrected or appropriately hedged claims. The reader's conditional verdict was based on missing artifacts and significance testing; my concern moves it to reject because the internal contradiction is more decisive. I still credit the paper for clear architectural descriptions, the data pipeline, and the BLF idea, but these do not rescue the stated central claim.","tokens_in":14339,"tokens_out":7288,"duration_ms":66145,"concrete_test":"Audit Table 3 against the Abstract's claim: for 'superior performance' to hold, SkyReels-Audio must not be dominated by a baseline on the internal benchmark. Ask the authors to release per-sample metric values for Table 3 and run a paired bootstrap or Wilcoxon signed-rank test comparing SkyReels-Audio versus OmniHuman-1 on Sync-C, Sync-D, IQA, and ASE. If the reported means are accurate and the differences are significant, the Abstract's 'superior performance' claim is contradicted by the paper's own data and must be softened to 'competitive' or restricted to specific settings. If the table contains a typo, correcting it would resolve the concern.","verdict_should_be":"REJECT","load_bearing_attack":"The Abstract and Section 3.2 claim SkyReels-Audio achieves 'superior performance' and 'consistently outperforms' baselines, with the internal benchmark described as representing 'complex and challenging conditions.' Table 3, which is the primary evidence for this claim, shows SkyReels-Audio with Sync-C 6.75, Sync-D 8.32, IQA 4.42, and ASE 2.91, while OmniHuman-1 achieves 7.50, 7.47, 4.66, and 3.19 respectively. OmniHuman-1 is better on every metric (higher is better for Sync-C, IQA, ASE; lower is better for Sync-D). Even MagicInfinite beats SkyReels-Audio on IQA and ASE (4.56 vs 4.42 and 3.00 vs 2.91). The paper reports no error bars or significance tests, so the small HDTF advantages in Table 1 (Sync-C 6.06 vs 5.75) are also uninterpretable, but the Table 3 reversal is decisive because it directly contradicts the headline claim on the paper's own benchmark. At best the results are 'comparable' (as Section 3.2 concedes for close-source models), not 'superior.' This is an internal inconsistency, not a disagreement with external consensus, and it undermines the central claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SkyReels-Audio, a unified audio-conditioned talking-portrait generation and editing framework built on the SkyReels-V2 video diffusion transformer. Audio is encoded by Whisper and injected via cross-attention layers with 1D RoPE; a hybrid curriculum trains image animation and video editing jointly with a facial-region-weighted loss; inference uses audio/text classifier-free guidance, a Bidirectional Latent Fusion (BLF) sliding-window mechanism for infinite-length generation, and TeaCache/USP acceleration. The authors describe a multi-stage data pipeline that filters 10K hours of video down to 1K hours of audio-video-text triplets. The evaluation reports quantitative metrics on HDTF and an internal benchmark, a 20-participant user study, and ablations of Audio CFG and Audio RoPE.","tokens_in":14625,"tokens_out":4209,"duration_ms":41080,"significance":"If the central claim of superior lip-sync accuracy, identity consistency, and facial dynamics were supported, the paper would be a useful systems contribution to audio-conditioned portrait animation, particularly for its multimodal conditioning and long-video inference strategy. The manuscript has genuine strengths: it provides a detailed data curation pipeline, an algorithmic specification of BLF, and ablations that isolate the effects of Audio CFG and RoPE. It also gives concrete inference-acceleration measurements. However, the empirical evidence as presented is not sufficient to support the headline claims: the internal benchmark is not released, Tables 1-3 report single numbers without variance or significance testing, and the paper's own Table 3 contradicts the abstract's 'superior performance' claim. The central contribution is plausible and the framework is well-motivated, but the claims need substantial revision and the evaluation needs strengthening.","major_comments":[{"comment":"The claim of 'superior performance' is contradicted by Table 3 on the paper's own internal benchmark. OmniHuman-1 beats SkyReels-Audio on all four metrics (Sync-C 7.50 vs. 6.75, Sync-D 7.47 vs. 8.32, IQA 4.66 vs. 4.42, ASE 3.19 vs. 2.91), and MagicInfinite also exceeds SkyReels-Audio on IQA (4.56 vs. 4.42) and ASE (3.00 vs. 2.91). The Section 3.2 phrasing 'comparable with close-source models' is accurate, but the Abstract and parts of the text claim consistent superiority. This is an internal inconsistency that directly affects the paper's central claim and must be resolved by either revising the claims or providing additional evidence that supports superiority.","section":"Abstract and Section 3.2, Table 3"},{"comment":"The BLF update in Algorithm 1, line 10, is written as z_{t-1}^{[s,s+o]} = w * z_{t-1}^{[s,s+o]} + (w-1) * z_{t-1}^{[e_prev-o,e_prev]}. Since w is defined as a vector from 0 to 1, (w-1) is negative or zero, so this is not a convex combination and at w=0 the previous-window latent is subtracted rather than fused. This appears to be a sign error; the intended formula is likely (1-w) * z_prev. As written, the algorithm does not implement the 'weighted fusion' described in the text, so the correctness of the BLF mechanism is not established.","section":"Algorithm 1, line 10"},{"comment":"All quantitative results are reported as single numbers without error bars, standard deviations, multiple seeds, or significance tests. For differences like HDTF Sync-C 6.06 vs. 5.75 in Table 1, it is impossible to determine whether the improvement is meaningful. This is especially problematic because the internal benchmark is neither released nor described in sufficient detail to be reproduced, and the Table 3 baseline numbers for close-source models are not independently verifiable. The evaluation should include variance estimates or, at minimum, clearly specify the evaluation protocol and release the benchmark materials.","section":"Section 3.1 and Tables 1-3"},{"comment":"The user study has only 20 participants and the table reports no inter-rater agreement or significance analysis. The heading 'HDTF User Study(Internal)' is also confusing because HDTF is a public dataset while the study is described as being on the Internal dataset. Since the user study is used as evidence for perceptual superiority, the small sample size and unclear protocol substantially weaken this evidence. Please provide the study protocol, participant instructions, and statistical treatment.","section":"Section 3.2 and Table 2"}],"minor_comments":[{"comment":"The manuscript contains multiple typographical errors that should be corrected, including 'denosing network' (Section 2.2), 'iamge' (Section 2.4), 'seto' (Section 3.1), 'seuqences' (Section 2.4), 'out contributions' (Section 1), and 'claim naturally consistent video results' (Figure 8 caption).","section":"Throughout"},{"comment":"The loss notation is unclear: L_msk appears in Equation (2) but is not formally defined; presumably it is the flow-matching loss restricted to non-masked areas, but the distinction between L_mse and L_msk should be stated explicitly.","section":"Equation (2)"},{"comment":"The post-processing step for color unification is mentioned but never described; since the authors identify a color-darkening problem during long-video inference, they should specify the color-unification procedure for reproducibility.","section":"Section 2.4"},{"comment":"The 'Audio CFG=1' row is effectively a no-audio-CFG ablation but is not labeled as such; also, the row order in the table should be clarified so that the default configuration is identified unambiguously.","section":"Table 3"},{"comment":"The description of the internal benchmark says it comprises '50+ audio-driven scenarios' and also '50+ portrait images, 30 audio segments, 20 textual prompts'; the exact composition of the benchmark should be stated more precisely to allow interpretation of the results.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system report from an industrial team, and it relies substantially on the authors' own SkyReels-V2 backbone and internal benchmark. That is not itself a problem, but the manuscript should disclose more precisely how baseline numbers in Table 3 were obtained and should avoid overclaiming in the abstract. The Algorithm 1 sign error is a concrete technical flaw that must be fixed. If the authors cannot revise the abstract and Section 3.2 to remove the 'superior performance' claim, the paper may need to be rejected even after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent systems paper with a real piece of engineering, but the paper's own Table 3 contradicts the abstract's claim of 'superior performance.' That's the first thing your readers should know.\n\nWhat is genuinely new is the integration: audio, image, video, and text conditioning in one video DiT, with a hybrid image/video training loss, audio-guided CFG, and a bidirectional sliding-window fusion for long videos. The ablations are informative: audio CFG and audio RoPE both produce measurable gains, and BLF improves transition quality. The data pipeline is described specifically enough to be useful, and the TeaCache/USP speedup numbers are concrete.\n\nThe soft spots are in the evaluation. On the internal benchmark in Table 3, OmniHuman-1 beats SkyReels-Audio on all four metrics, and MagicInfinite beats it on two. The abstract says 'consistently outperforms'; the body says 'comparable with close-source models.' Those two sentences cannot both be the headline. There are no error bars or significance tests anywhere, so the HDTF advantages in Table 1 are not interpretable. The internal benchmark isn't released, the code isn't released, and the user study has 20 participants. Minor but telling: the conclusion says 'conditioned exclusively on audio inputs,' which contradicts the multimodal framing used everywhere else. That's the kind of inconsistency a referee would want cleaned up.\n\nNone of this is fatal to the underlying work. The system is plausible, the ablations support the design choices, and the approach is well-situated in the literature. But the paper as submitted overstates its own results. I'd send it to peer review with a required revision on the evaluation claims—fixed numbers or honest language, plus error bars if available. The audience is people building or benchmarking talking-head systems; they'll get useful engineering detail even if the headline doesn't hold.\n\nMy bottom line: give it a serious referee, but make clear the central claim needs to be rewritten before acceptance.","headline":"Solid system paper whose own Table 3 undercuts the 'superior performance' claim in the abstract.","tokens_in":15201,"tokens_out":2317,"would_cite":false,"duration_ms":22252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkyReels-Audio generates and edits talking portraits from audio plus text, image, or video inputs, claiming better lip-sync, identity consistency, and natural motion than current open baselines.","keywords":["audio-conditioned talking portraits","video diffusion transformers","lip-sync","infinite-length video generation","bidirectional latent fusion","multimodal conditioning","portrait animation and editing","flow matching"],"falsifier":"Run SkyReels-Audio and the open baselines on a held-out set of speakers and languages not seen in its training data, compute Sync-C and Sync-D with bootstrapped confidence intervals, and add a forced-choice human listening study; if the HDTF Sync-C edge (6.06 versus 5.75) collapses within noise or reverses on fast speech or non-verbal vocalizations, the superiority claim fails. Alternatively, a long-video continuity test in which BLF is replaced by no-overlap or unidirectional fusion should show visible seam jumps if BLF is doing the claimed work.","tokens_in":14143,"feed_emoji":"🎤","tokens_out":8880,"duration_ms":77998,"temperature":0.7,"pith_summary":"SkyReels-Audio is a unified system for generating and editing talking portraits, built on a pretrained video diffusion transformer and driven by audio together with text, image, or video controls. The paper claims it produces better lip-sync accuracy, identity consistency, and natural facial dynamics than open-source baselines such as Hallo3 and FantacyTalking, on both the public HDTF dataset and the authors' internal benchmark. The model treats audio as the main controller of articulation, uses a hybrid curriculum to jointly learn image animation and video editing, and adds a facial-mask loss plus audio-guided classifier-free guidance to keep lips aligned without losing global consistency. A tuning-free sliding-window scheme, Bidirectional Latent Fusion, stitches overlapping denoised latents so output can be extended indefinitely. If the claims hold, one model could replace separate pipelines for animating a single photo, editing an existing video, and producing long-form portraits from speech, singing, or rap in multiple languages.","feed_headline":"SkyReels-Audio animates portraits with lips synced to any audio","feed_subtitle":"Audio, text, and video cues drive one model, with sliding-window fusion for arbitrarily long clips.","key_machinery":"The load-bearing mechanism is the audio-conditioned DiT: a pretrained 3D video VAE compresses frames, a pretrained video diffusion transformer backbone supplies the spatio-temporal prior, and a speech encoder turns the waveform into audio tokens that are fused in dedicated cross-attention layers with 1D rotary position embeddings. Training uses a hybrid curriculum where image-animation and video-editing tasks are distinguished by a binary temporal mask, and the flow-matching loss is reweighted by facial masks: $L_{\\mathrm{joint}} = w_1 \\, V_m^{\\mathrm{downsample}} \\odot L_{\\mathrm{mse}} + w_2 \\, (1 - V_m^{\\mathrm{downsample}}) \\odot L_{\\mathrm{msk}}$, with a probabilistic gate that focuses on lip regions. At inference, audio-guided classifier-free guidance with separate, time-scheduled audio and text scales sharpens synchronization, and Bidirectional Latent Fusion overlaps adjacent windows during denoising and linearly interpolates the overlapping latents to remove stitching artifacts in long videos.","core_discovery":"The paper's central claim is that a single audio-conditioned video diffusion transformer can outperform specialized talking-head systems on lip-sync accuracy, identity preservation, and motion realism, while also handling arbitrary-length generation. The reported evidence: on HDTF, SkyReels-Audio reaches Sync-C 6.06 and Sync-D 9.12, ahead of Hallo3 (5.75/10.12) and FantacyTalking (5.36/11.68), with lower FID and FVD; on the internal benchmark it beats open baselines though it trails the closed OmniHuman-1 on some metrics; and a user study on the internal set rates it above LatentSync on audio-visual consistency and visual quality. The design that carries the claim is audio injection through cross-attention with 1D rotary position embeddings, joint image and video training under a mask-weighted flow-matching loss, inference-time audio classifier-free guidance, and Bidirectional Latent Fusion for stitching long videos.","pith_inferences":["Beyond the paper: the Sync-C deltas over open baselines are small (for instance 6.06 versus 5.75 on HDTF), so the practical win may lie more in long-form stability and multimodal control than in raw lip-sync; that split is testable by comparing single-shot clips separately from multi-window videos.","Beyond the paper: BLF is stated as tuning-free and window-agnostic, which suggests it could be lifted into other windowed DiT video models; a drop-in study would establish whether the fusion weights transfer.","Beyond the paper: the data funnel from 10K to 1K hours, with manual annotation and a sub-5% bad-case target, positions data curation as a first-class contribution; an ablation that trains the same architecture on unfiltered versus filtered data would quantify how much of the gain comes from the pipeline rather than the model.","Beyond the paper: the benchmark mixes anime, sculpture, and photorealistic portraits but trains only on real videos, so the generalization claim could be stress-tested by adding out-of-domain portrait types the pipeline never saw."],"forward_implications":["A single pretrained DiT can serve both image-animation and video-editing use cases, with audio as the primary driver of lip motion and auxiliary modalities controlling expression, gesture, and background.","With Audio CFG at 4.5 the model trades a small amount of visual quality for noticeably better sync metrics; lowering CFG to 1 or 3 degrades Sync-C and Sync-D on the internal benchmark.","Removing audio RoPE drops Sync-C from 6.75 to 5.58 on the internal benchmark, indicating that positional alignment of audio tokens within cross-attention is a necessary component.","TeaCache and USP together cut the time for 80 frames from about 23.6 minutes to under 1 minute on 8 A800 GPUs, making the approach practical for long-form and interactive use.","Bidirectional Latent Fusion produces smooth window transitions without extra training, so infinite-length portrait videos can be stitched online during denoising."],"supporting_citations":[{"why":"Supplies the pretrained video diffusion transformer and 3D VAE that SkyReels-Audio adapts for audio conditioning.","marker":"[6]"},{"why":"Provides the HDTF test clips used for the main public-dataset quantitative comparison.","marker":"[79]"},{"why":"Defines the Sync-C and Sync-D audio-visual synchronization metrics used throughout the evaluation.","marker":"[11]"},{"why":"Supplies the speech encoder that converts the waveform into audio tokens for cross-attention.","marker":"[53]"},{"why":"Is the open-source baseline in the HDTF and internal tables that SkyReels-Audio is measured against.","marker":"[12]"},{"why":"Is the other open-source baseline compared on HDTF and the internal benchmark.","marker":"[65]"},{"why":"Is the closed-source baseline in the internal benchmark at the top of the table.","marker":"[38]"},{"why":"Provides the denoising-step caching used to accelerate inference.","marker":"[40]"},{"why":"Provides multi-GPU sequence parallelism for long-video inference acceleration.","marker":"[16]"},{"why":"Is the infinite-length talking-video baseline compared in the internal benchmark.","marker":"[77]"}],"fun_headline_variants":["SkyReels-Audio syncs lips to any audio, for infinite portraits","Endless talking portraits with audio-driven lip sync","Multimodal audio cues for infinite talking-head video","Sliding-window denoising for long audio-synced portraits","Outperforms specialized models on lip-sync accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that SkyReels-Audio is superior rests on the assumption that SyncNet-based Sync-C and Sync-D scores on the HDTF clips and the authors' 50-image, 30-audio internal benchmark are faithful, unbiased measures of real lip-sync quality, and that the 1K-hour filtered training set does not silently favor the model's biases.","fun_headline_variants_meta":{"raw":{"variants":["SkyReels-Audio syncs lips to any audio, for infinite portraits","Endless talking portraits with audio-driven lip sync","Multimodal audio cues for infinite talking-head video","Sliding-window denoising for long audio-synced portraits","Outperforms specialized models on lip-sync accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000549,"raw_usage":{"total_tokens":2623,"prompt_tokens":946,"completion_tokens":1677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":1593}},"tokens_in":562,"tokens_out":1677,"duration_ms":14486,"temperature":1.0,"reasoning_tokens":1593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:55:58.300432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SkyReels-Audio and the open baselines on a held-out set of speakers and languages not seen in its training data, compute Sync-C and Sync-D with bootstrapped confidence intervals, and add a forced-choice human listening study; if the HDTF Sync-C edge (6.06 versus 5.75) collapses within noise or reverses on fast speech or non-verbal vocalizations, the superiority claim fails. Alternatively, a long-video continuity test in which BLF is replaced by no-overlap or unidirectional fusion should show visible seam jumps if BLF is doing the claimed work.","supporting_citations":[{"cited_title":"Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"Provides the HDTF test clips used for the main public-dataset quantitative comparison."},{"cited_title":"Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models, 2025","cited_arxiv_id":null,"evidence_quote":"Is the closed-source baseline in the internal benchmark at the top of the table."}],"review_version":1}