{"id":"2abe3320-050b-4707-876d-c43d5096ffe3","arxiv_id":"2608.11752","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"UniSwap is a streaming audio-video diffusion model that transfers a reference face and voice onto a source talking video while keeping motion, scene, and speech content.","lead":"UniSwap swaps both a person's appearance and voice in a talking video using one streaming model. It aims to make dubbing and character replacement faster, with lip movements matching the new voice.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed joint-generation sync advantage may be an artifact of evaluating cascades without lip-sync re-anchoring to the converted audio; a re-anchored cascade should be tested before accepting the headline.","rationale":"I read the paper's central claim as an empirical one: a single audio-visual diffusion transformer outperforms cascaded single-modality replacement on audio-visual synchronization, with Table 1 as the primary support. The reader's weakest assumption about swap-and-reconstruct fidelity is legitimate, but it is partly self-checking: if the pose proxy or Seed-VC conversion badly corrupted the source, the downstream identity and content preservation metrics (DINO-S, SSIM) would degrade, and the reported numbers do not show catastrophic failure. The evaluation-protocol concern is more directly decisive because it attacks the specific comparison used to justify joint modeling. The paper itself states that each visual output is paired with Seed-VC audio, but the cascade is never allowed to re-synchronize the lips to that audio. A fair cascade should be closed-loop, with the video conditioned on or post-processed by the converted speech; only then can the Sync-C gap be attributed to joint modeling rather than to the absence of feedback in the baseline. This concern is falsifiable: re-running Table 1 with a lip-sync re-anchored cascade would settle it. Because the reader already identified this needed control and returned CONDITIONAL, my stress-test sharpens the condition but does not change the verdict. I also note the paper is internally honest in Sec. 4.3 that 13.6 FPS is below the 25 FPS playback rate, so the Fig. 1 'Real Time' label is an overstatement rather than a correctness flaw in the method.","tokens_in":15847,"tokens_out":7826,"duration_ms":87828,"concrete_test":"Re-run the Table 1 comparison after inserting a lip-sync re-anchoring step into each video-replacement baseline: generate the baseline video, convert the source audio with Seed-VC, then post-process the video with a lip-sync model conditioned on the converted Seed-VC audio before computing Sync-C and Sync-D. If the best re-anchored cascade reaches or exceeds UniSwap's Sync-C of 3.633, the joint-modeling advantage claimed in the paper is not demonstrated; if it does not, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is not the swap-and-reconstruct fidelity but the evaluation protocol behind the central claim. In Table 1, every video-replacement baseline is paired with Seed-VC audio, and Sync-C/Sync-D are then measured on the resulting cascade. Nothing re-anchors the baseline's lip and face motion to the converted speech; the video models generate their output from the source clip and reference identity without conditioning on the Seed-VC output, so any timing or articulation mismatch between the original lip motion and the converted waveform is charged against the cascade. Voice conversion preserves prosody only approximately, and the paper provides no evidence that the baseline videos are even lip-synced to the original source audio after identity replacement. The claimed advantage of joint generation (Sync-C 3.633 vs. 3.289 for SCAIL-2 + Seed-VC) is therefore a comparison against an off-the-shelf pipeline with no feedback loop between the converted audio and the generated video, not against the strongest cascade the paper's own formulation allows. If a lip-sync re-anchoring module is inserted after Seed-VC and before scoring, the cascade can correct exactly the inconsistency that UniSwap claims only joint modeling can fix. Until that experiment is run, the paper's headline empirical assertion is not established. Secondary issues such as the missing OmniForcing baseline, the unreleased code and weights, and the 'Real Time' label contradicted by the 13.6 FPS reported in Sec. 4.3 are real but do not bear on the central comparison as directly as this protocol gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UniSwap, a streaming framework for joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers appearance and vocal timbre in a single audio-visual diffusion transformer while preserving source motion, scene, and content. The authors address the lack of aligned cross-identity training pairs with a swap-and-reconstruct pipeline that creates identity-altered sources from real clips and uses the originals as targets. Starting from the LTX-2.3 backbone, they progressively adapt it through in-context pretraining, conditional streaming adaptation with a block-causal mask, and self-forcing DMD with three-step sampling. Feature-RoPE Decomposition and adaptive sink blocks are introduced for long-form streaming inference. Experiments report a Sync-C advantage over cascaded baselines, competitive identity preservation, and 13.6 FPS on one H100, while acknowledging that this is below the 25 FPS playback rate and therefore streaming rather than real-time.","tokens_in":16052,"tokens_out":4542,"duration_ms":47519,"significance":"If the empirical claims are established, UniSwap would be a notable engineering contribution: it demonstrates a scalable way to construct aligned supervision for joint identity replacement and shows that a bidirectional audio-visual diffusion transformer can be converted into an efficient streaming generator with bounded cache and few-step sampling. The systematic ablations of the three training stages and of Feature-RoPE components are valuable. The central claim, however, is that joint generation yields superior audio-visual synchronization compared with cascades; this claim is not yet convincingly supported because the cascade comparison is not a like-for-like cascade, and the long-form claim is extrapolated from one-minute evaluations.","major_comments":[{"comment":"The comparison protocol undermines the paper's central claim. Every video-replacement baseline is paired with the same Seed-VC audio backend, but the video model never conditions on the Seed-VC output; Sync-C and Sync-D are then measured between the baseline video, whose lip movements were generated for the original source speech, and the converted audio track. This charges the cascade for a mismatch that a natural cascade with a lip-sync re-anchoring or dubbing stage would correct, and it may account for a substantial portion of the reported advantage (Sync-C 3.633 vs. 3.289). The authors should compare against a re-anchored cascade, for example a lip-sync module conditioned on the converted waveform, before attributing the synchronization gain to joint modeling.","section":"Sec. 4.1, Table 1"},{"comment":"The abstract and Fig. 1 claim stable hour-scale long-form generation, yet the long-form benchmark consists only of one-minute videos, with metrics computed independently on three 20-second segments. No experiment measures behavior beyond one minute, and the training clips are only about 9.6 seconds. The hour-scale claim is therefore unsupported. Please either extend the benchmark to a much longer duration (ideally hour-scale, or at least an order of magnitude beyond the training length) or remove the hour-scale claim from the abstract and Fig. 1.","section":"Sec. 4.2, Fig. 1, Table 2"},{"comment":"The swap-and-reconstruct pipeline is the sole source of training supervision, but its two components are not quantitatively validated. The pose-proxy video is assumed to preserve source motion, timing, background, and scene exactly, and Seed-VC is assumed to preserve speech content and temporal alignment; no metric is reported for either assumption. If the visual proxy or the voice conversion introduces misalignment or artifacts, the model is trained to reconstruct a target from corrupted sources, which directly weakens the identity-transfer and synchronization claims. The authors should report quantitative fidelity checks for the synthesized sources, such as pose alignment error, temporal sync between the converted audio and the source video, and content preservation of the converted speech.","section":"Sec. 3.1"}],"minor_comments":[{"comment":"The 'Real Time' label in Fig. 1 is misleading given that 13.6 FPS is below the 25 FPS playback rate; the text correctly states that the implementation supports streaming generation but not real-time playback. The figure label should be qualified accordingly.","section":"Fig. 1, Sec. 4.3"},{"comment":"Given that OmniForcing is described as a streaming audio-visual diffusion model with block-causal alignment and self-forcing distillation, the abstract's 'first framework for streaming joint audio-visual identity replacement' should either explicitly distinguish UniSwap from OmniForcing in terms of identity replacement or justify why OmniForcing is not a valid baseline. As written, the novelty claim is not fully contextualized.","section":"Sec. 2.3 and Sec. 4.1"},{"comment":"The efficiency table mixes per-block timings for UniSwap with whole-clip timings for baselines. While the footnote explains this, the FPS row is the only directly comparable measure; presenting per-step times in the same table invites apples-to-oranges comparisons. Consider reporting only wall-clock FPS and end-to-end latency in the main table.","section":"Table 3"},{"comment":"The user study inherits the same cascade-protocol issue as Table 1, since the baselines are again paired with Seed-VC without re-anchoring. In addition, no significance testing is reported for the difference between UniSwap and SCAIL-2+Seed-VC on lip synchronization (4.11 vs. 3.67).","section":"Supplementary Sec. 8"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a serious system paper, but the headline empirical claim is not yet established. UniSwap is the first streaming model I've seen that swaps both visual and vocal identity in a single audio-visual diffusion transformer, and the engineering is plausible: swap-and-reconstruct synthetic pairs, three-stage adaptation from a bidirectional backbone, and Feature-RoPE Decomposition for bounded-cache long-form inference. The ablations actually support the components. Removing the condition PE offset collapses Sync-C from 4.620 to 1.738, and removing any Feature-RoPE piece causes clear drift in the one-minute runs. The 3-step distillation holds up reasonably well, and the efficiency gain is real.\n\nThe soft spot is the comparison protocol behind the central sync claim. Every video-replacement baseline is paired with Seed-VC audio, but the video models never condition on that converted speech—they generate lip motion from the original source audio or whatever driving signal they use. Then SyncNet scores sync against the converted waveform. That charges the cascade for a mismatch it was never given a chance to fix. If you insert a lip-sync re-anchoring step after Seed-VC, the cascade gets exactly the feedback loop UniSwap claims only joint modeling provides. The paper even notes that cascades generate lip motion without access to the converted speech, which confirms the gap. Until that experiment is run, the 3.633 vs 3.289 Sync-C difference is not a fair comparison.\n\nOther issues are real but less central. OmniForcing, the closest streaming audio-visual work, is missing as a baseline. 'Hour-scale generation' and 'Real Time' in Figure 1 overstate what is shown: the actual evaluations run one minute, and 13.6 FPS is below the 25 FPS playback rate. The text admits this, but the figure does not. No code or weights are released, which hurts reproducibility. The swap-and-reconstruct pipeline assumes Seed-VC and the pose proxy preserve timing and content, and that assumption is never quantified—though it is a training-time trick rather than a claim about final outputs, so I weight it less.\n\nNone of this sinks the paper. The method is coherent, the ablations are convincing, and the qualitative examples show genuine joint replacement. But the central improvement over cascades needs a fairer test. This paper deserves a serious referee, and with a re-anchored cascade experiment plus a few honesty fixes to the claims, it could be a strong contribution. I would send it to review with a request for that experiment.","headline":"UniSwap is a well-engineered streaming audio-visual identity-swap system, but its headline sync advantage over cascades is not proven until a lip-sync re-anchored cascade baseline is tested.","tokens_in":16752,"tokens_out":2849,"would_cite":true,"duration_ms":31547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single audio-video diffusion transformer can stream both face and voice replacement for talking videos, with better lip-sync than cascaded systems.","keywords":["audio-visual identity swapping","talking video generation","streaming diffusion transformer","voice conversion","swap-and-reconstruct training","distribution matching distillation","positional encoding for streaming","long-form video generation"],"falsifier":"A concrete timing-shift probe would settle the synchronization claim: train or fine-tune on swap-and-reconstruct pairs in which the converted source audio is delayed by roughly 100 ms relative to the video, then evaluate on normally timed test clips; if Sync-C does not degrade, the model is reproducing reference appearance and timbre while ignoring cross-modal timing, contradicting the claim that joint generation is what drives the lip-sync gain.","tokens_in":15520,"feed_emoji":"🎭","tokens_out":11900,"duration_ms":117927,"temperature":0.7,"pith_summary":"This paper seeks to establish that a single streaming model, rather than a cascade of separately optimized modules, can replace both the face and the voice of a person in a talking video while preserving what the person says and how they move. If true, it matters because dubbing, localization, and interactive avatars need converted speech and lip motion to stay synchronized, and cascading visual replacement with voice conversion leaves the two modalities uncoordinated. To get training data for a task with no natural paired examples, the authors turn each ordinary video into its own aligned pair by stripping out the original identity and training the model to put it back. The reported payoff is better lip-sync than any evaluated cascade, comparable identity preservation, roughly ten times faster generation than the fastest evaluated baseline, and stable identity over minute-long outputs.","feed_headline":"UniSwap swaps face and voice in talking videos at 13.6 FPS","feed_subtitle":"One audio-video transformer does both identity transfers and beats cascades on lip synchronization.","key_machinery":"The load-bearing object is a single audio-video diffusion transformer whose video and audio tokens share a physical-time axis and interact through cross-modal attention, so neither modality is generated without seeing the other. Three training adaptations carry the argument. In-context pretraining concatenates reference, source, and noisy target latents into one attention context with condition positional offsets. Conditional streaming adaptation then applies a Decoupled Streaming Conditioning Mask that gives each token region its inference-time receptive field, turning the bidirectional model into a block-causal KV-cached generator. Efficient self-forcing distribution-matching distillation rolls out the student's own predictions and distills 30 denoising steps down to 3 per block, using switchable low-rank adapters (LoRA) so teacher, generator, and critic share one frozen backbone. For long-form inference, Feature-RoPE Decomposition stores unrotated cached keys and re-applies bounded rotary positions, with a fixed sink block and re-anchored reference, keeping every position inside the training range.","core_discovery":"UniSwap's central claim is that talking-video character replacement is best posed as one conditional audio-video generation problem rather than a cascade of a visual replacer and a voice converter. Given a source video with audio, a reference image, and a reference voice clip, a single audio-video diffusion transformer outputs synchronized video and audio carrying the reference appearance and vocal timbre while preserving source motion, background, and linguistic content. Because aligned cross-identity pairs do not exist at scale, every real clip becomes its own training pair: the visual identity is removed by replacing the person with a pose proxy on the masked background, the vocal timbre is randomized by voice conversion, and the model is trained to reconstruct the original clip from those identity-swapped conditions plus the original identity as references. The pipeline then converts a bidirectional backbone into a block-causal three-step generator through in-context pretraining, streaming adaptation with a decoupled conditioning mask, and self-forcing distribution-matching distillation. On the paper's benchmark, the joint model achieves the best audio-visual synchronization (Sync-C 3.633 versus 3.289 for the strongest cascade) with nearly identical identity preservation (DINO-S 0.629 versus 0.630), at 13.6 FPS on one H100, while holding identity roughly flat across one-minute videos.","pith_inferences":["Because swap-and-reconstruct turns any talking-video corpus into training data, the bottleneck shifts to the fidelity of the pose proxy and the voice converter; a testable extension is that raising proxy fidelity should improve identity-transfer accuracy more than adding model capacity.","The joint model's sync advantage over cascades suggests cross-modal information is doing real work; ablating audio conditioning from the video stream during inference would isolate how much of the Sync-C gain comes from seeing the converted speech.","Window-bounded positional remapping is a general cure for streaming diffusion drift; the same Feature-RoPE idea could be reused in other autoregressive multimodal diffusion models trained with absolute positions.","A deployment consequence the paper leaves implicit: at 13.6 FPS on one H100, real-time playback still needs roughly a 2x speedup, but the per-block cost profile suggests this gap is engineering rather than conceptual."],"forward_implications":["A cascade-free identity-swap pipeline becomes viable: one transformer produces converted speech and matching lip motion from a single reference image and voice clip, so no separate visual replacer and voice converter need to be reconciled.","Blockwise KV-cached inference makes per-block cost independent of total generated duration, so the same architecture can stream a short clip or an hour-long clip without reprocessing the full sequence at each step.","Three-step distillation lowers the per-block cost to 1.76 seconds for 24 frames (13.6 FPS on one H100), about 10 times faster than the fastest cascade the paper evaluates, though still below 25 FPS playback.","Long-form identity stability improves: DINO-S stays at 0.590–0.596 across three 20-second segments of one-minute videos, while a strong cascade baseline declines from 0.566 to 0.517.","Joint generation preserves speech quality close to the dedicated voice converter (SIG 3.486 versus 3.489), showing the synchronization gain is not purchased by degrading the converted voice."],"supporting_citations":[{"why":"Supplies the frozen bidirectional audio-video backbone with native cross-modal attention that all three training stages adapt.","marker":"[11]"},{"why":"Supplies the off-the-shelf voice converter that creates the identity-swapped source audio in every training tuple and serves as the common audio backend for the cascaded baselines.","marker":"[22]"},{"why":"Supplies the distribution-matching distillation objective that Stage 3 turns into efficient self-forcing, cutting denoising from 30 to 3 steps.","marker":"[42]"},{"why":"Supplies the in-context conditioning paradigm that Stage 1 extends to joint audio-video replacement.","marker":"[9]"},{"why":"Supplies the rotary position embeddings whose absolute coordinates Stage 1 offsets and Feature-RoPE Decomposition later bound.","marker":"[29]"},{"why":"Supplies the pose estimator whose keypoints drive the visual identity removal in swap-and-reconstruct data synthesis.","marker":"[37]"},{"why":"Supplies the video segmentation model that produces the person mask used to build the background plate and pose proxy.","marker":"[25]"},{"why":"Supplies the self-forcing idea of conditioning generation on the model's own previous outputs to reduce exposure bias.","marker":"[16]"},{"why":"Supplies the attention-sink mechanism that the adaptive sink block uses to anchor identity in long-form generation.","marker":"[36]"},{"why":"Supplies the lip-sync confidence and distance metrics used in the short-video evaluation.","marker":"[5]"}],"fun_headline_variants":["One transformer swaps face and voice in talking videos","UniSwap unifies audio-visual identity transfer in one model","Joint audio-video transformer beats cascades on lip sync","13.6 FPS talking-video identity swap with single diffusion model","Streaming identity swap with one audio-video diffusion transformer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the swap-and-reconstruct pipeline's identity-removed source—a pose proxy composited onto the background plus machine-converted voice—preserves the original clip's motion, scene, speech content, and timing faithfully enough that reconstructing the original clip teaches real identity transfer; if either proxy is unfaithful, the model is trained on corrupted targets and the measured transfer is partly artifact.","fun_headline_variants_meta":{"raw":{"variants":["One transformer swaps face and voice in talking videos","UniSwap unifies audio-visual identity transfer in one model","Joint audio-video transformer beats cascades on lip sync","13.6 FPS talking-video identity swap with single diffusion model","Streaming identity swap with one audio-video diffusion transformer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2504,"prompt_tokens":1054,"completion_tokens":1450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":1366}},"tokens_in":670,"tokens_out":1450,"duration_ms":11065,"temperature":1.0,"reasoning_tokens":1366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:28:43.082557+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete timing-shift probe would settle the synchronization claim: train or fine-tune on swap-and-reconstruct pairs in which the converted source audio is delayed by roughly 100 ms relative to the video, then evaluate on normally timed test clips; if Sync-C does not degrade, the model is reproducing reference appearance and timbre while ignoring cross-modal timing, contradicting the claim that joint generation is what drives the lip-sync gain.","supporting_citations":[{"cited_title":"Freeman, and Taesung Park","cited_arxiv_id":null,"evidence_quote":"Supplies the distribution-matching distillation objective that Stage 3 turns into efficient self-forcing, cutting denoising from 30 to 3 steps."},{"cited_title":"Roformer: Enhanced transformer with rotary position embedding","cited_arxiv_id":null,"evidence_quote":"Supplies the rotary position embeddings whose absolute coordinates Stage 1 offsets and Feature-RoPE Decomposition later bound."},{"cited_title":"Vit- pose: Simple vision transformer baselines for human pose estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the pose estimator whose keypoints drive the visual identity removal in swap-and-reconstruct data synthesis."},{"cited_title":"Sam 2: Segment anything in images and videos","cited_arxiv_id":null,"evidence_quote":"Supplies the video segmentation model that produces the person mask used to build the background plate and pose proxy."},{"cited_title":"Efficient streaming language models with attention sinks","cited_arxiv_id":null,"evidence_quote":"Supplies the attention-sink mechanism that the adaptive sink block uses to anchor identity in long-form generation."},{"cited_title":"Out of time: Auto- mated lip sync in the wild","cited_arxiv_id":null,"evidence_quote":"Supplies the lip-sync confidence and distance metrics used in the short-video evaluation."}],"review_version":1}