{"id":"65184bf2-1053-43f9-9451-59700da342d9","arxiv_id":"2412.10494","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SnapGen-V prunes, searches, and adversarially distills a video diffusion model down to 0.6B parameters that generates a five-second, 512x512 video on an iPhone 16 Pro Max in under five seconds.","lead":"A Snap team built a compact text-to-video model, named SnapGen-V, that runs fully on an iPhone 16 Pro Max and produces a five-second clip in about five seconds. This suggests video diffusion models can move from cloud servers onto personal devices, which matters for cost, privacy, and offline content creation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quality evaluation and mobile deployment use different operating points: VBench is reported at 120-frame 432x768 24fps while the iPhone demo is a 51-frame 512x512 10fps checkpoint that was 'further finetuned' at 10fps; the deployed model's quality is never measured.","rationale":"Read in good faith, the engineering story is coherent, the speed measurement is detailed, and the ablations are informative. However, the claim that matters is the conjunction 'within 5 seconds on iPhone while delivering on-par quality.' The paper provides a careful latency budget for one deployment configuration (51 frames, 512x512, 10 fps, and a model 'further finetuned' at 10 fps), while the quality evaluation uses a different configuration (120 frames, 432x768, 24 fps) and is not tied to that 10 fps fine-tune. The appendix sentence about further fine-tuning at 10 fps is easy to miss, but it is exactly where the checkpoint identity is at stake. If the 10 fps fine-tune occurred after the VBench evaluation, the reported scores cannot certify the deployed model. If it occurred before, the paper should say so and should report VBench at the mobile setting, because frame count and fps are not free parameters in a video-quality benchmark: motion smoothness, temporal flicker, and dynamic degree all depend on them. The reader's weakest assumption (VBench as both search objective and evaluation metric) is also valid, but it is a separate issue: it questions whether the metric is trustworthy, whereas this concern questions whether the metric was even applied to the deployed model. Both are resolved by releasing the exact mobile checkpoint and re-running evaluation at the mobile operating point. This does not invalidate the speed contribution; it means the headline combined claim remains conditional on evaluation of the actual deployed model.","tokens_in":19709,"tokens_out":6584,"duration_ms":58762,"concrete_test":"Obtain or reconstruct the exact 10 fps fine-tuned mobile checkpoint described in Appendix F (e.g., by requesting weights or re-running the stated fine-tuning), then run the VBench evaluation at the mobile operating point (51 frames, 512x512, 10 fps) and compare with Table 2. If the total score drops by more than roughly 0.13 (the gap to T2V-Turbo) or a human study against T2V-Turbo/AnimateLCM at the mobile setting does not show parity, the 'on-par quality while within 5 seconds' claim for the deployed model is unsupported. The authors should also state explicitly whether Table 2 uses the pre-10fps or post-10fps checkpoint.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim bundles speed and quality, but the evidence for the two halves comes from different settings. The VBench evaluation in Sec. 4 reports 120-frame horizontal videos at 432x768, saved at 5 seconds 24 FPS. Appendix F describes the mobile demo as 51 frames at 512x512, 10 FPS, and states that 'the model is further finetuned with video datasets with a framerate of 10 fps.' If that fine-tuning is part of the deployed checkpoint, then the 81.14 VBench total score in Table 2 and the user-study win rates in Table 3 were obtained before this final fine-tuning and at a different frame count/fps. Therefore the paper does not demonstrate 'on-par quality' for the model actually running on the iPhone within 5 seconds. The reader's concern about VBench being used both for architecture search and final evaluation is real, but it is secondary: even an unbiased VBench score would not resolve this checkpoint mismatch. The speed number (4.12 s) and the quality number (81.14) are not shown to describe the same model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SnapGen-V, a framework for mobile text-to-video generation. Starting from a pruned Stable Diffusion v1.5 spatial backbone, the method performs an evolutionary architecture search over temporal layer types and placements using a hardware latency/memory lookup table and VBench-based quality scores, trains the final model with image-video joint training, and then applies latent adversarial fine-tuning with unified spatial-temporal discriminator heads to reduce sampling to 4 denoising steps. It also compresses the OpenSora spatial-temporal VAE decoder. The headline result is a 0.6B-parameter model that generates a 5-second 512x512 video on an iPhone 16 Pro Max in 4.12 seconds (Appendix F) and achieves a VBench total score of 81.14 (Table 2), claimed to be on par with or better than server-side models.","tokens_in":20107,"tokens_out":6160,"duration_ms":57642,"significance":"If the claims hold, this is a substantial engineering contribution: it is the first demonstration of text-to-video diffusion on a phone, with a measured latency under five seconds, a systematic temporal-layer search, and a plausible joint image-video adversarial distillation recipe. The speed claim rests on an independent hardware benchmark rather than on a fitted parameter, and the VAE decoder compression is quantified with reconstruction metrics. The architecture ablations and the comparison against SF-V-style discriminator heads are informative. However, the quality evidence is weaker: the VBench-based architecture selection creates a selection-artifact risk for the final VBench comparison, and the evaluated checkpoint differs from the deployed mobile checkpoint. Releasing the code and model, evaluating the exact deployed artifact, and reporting end-to-end latency would materially strengthen the paper.","major_comments":[{"comment":"The central quality claim and the central speed claim are not shown to describe the same model. The VBench evaluation in Section 4 generates 120-frame horizontal videos at 432x768 and saves them at 5 seconds and 24 fps, while Appendix F reports a mobile demo at 51 frames, 512x512, 10 fps, and states that 'the model is further finetuned with video datasets with a framerate of 10 fps.' Consequently, the 81.14 VBench total in Table 2 and the user-study win rates in Table 3 were not measured on the checkpoint that runs on the iPhone in 4.12 seconds. Please either provide VBench and user-study numbers for the exact deployed 51-frame/10-fps checkpoint, or explicitly restrict the on-par-quality claim to the non-deployed evaluation setting.","section":"Section 4 (Evaluation) and Appendix F"},{"comment":"The architecture search in Section 3.2 selects temporal layers by training candidates for 20K iterations and ranking them with VBench scores (Appendix A specifies a VBench subset over 268 prompts), and Table 2 then reports the final model's VBench total. The quality comparison is therefore partly a selection artifact, because the search objective and the final evaluation metric are the same. The paper should report the variance of VBench scores across evaluation seeds or prompt sets, evaluate held-out architectures that were not selected by VBench, or support the final architecture choice with a human study at the deployed operating point.","section":"Section 3.2 and Appendix A"},{"comment":"The sub-5-second speed is assembled from component measurements (four denoising steps at 1.02 seconds each, plus 0.5 seconds for the VAE decoder and 6 milliseconds for the text encoder) rather than from an end-to-end wall-clock measurement of a complete run. On-device pipelines can have non-compositional costs from model loading, the two-part split of the exported CoreML model, memory pressure, and scheduling. Please report the measured end-to-end latency of the full pipeline on the iPhone 16 Pro Max, ideally with a per-prompt distribution, in addition to the component benchmarks.","section":"Appendix F"},{"comment":"Many of the VBench differences in Table 2 are small (e.g., 81.14 vs. 81.01 for T2V-Turbo and 81.14 vs. 80.91 for CogVideoX-2B), and no error bars, number of seeds, or significance tests are provided. Without such information, the claim that a 0.6B 4-step mobile model 'achieves higher total score compared to recent arts' is not robustly supported. Please include confidence intervals or per-prompt score distributions for at least the main comparisons.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"The text 'OponSora-v1.2' should read 'OpenSora-v1.2'.","section":"Appendix D"},{"comment":"The header line 'color spatial appearance temporal overall' is missing separators; please format it so the VBench sub-metric names are clearly distinguishable.","section":"Table 2"},{"comment":"The user study does not state the number of prompts, annotators, or pairwise comparisons; please include these details so the win rates can be interpreted.","section":"Table 3"},{"comment":"The row label 'Scaling' and the column abbreviations '∆T', '∆AQ', '∆IQ', '∆DD', and '∆MS' are hard to parse; please define every symbol in the caption and state the baseline latency explicitly.","section":"Table 4"},{"comment":"The description of the temporal self-attention in the discriminator heads should specify whether it operates over the full frame sequence or over chunks, since memory constraints are central to the mobile design.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The main unresolved risk is the mismatch between the evaluated checkpoint and the deployed mobile checkpoint, not any apparent intent to mislead. I would ask the authors to evaluate the exact deployed artifact and to provide an end-to-end latency measurement before acceptance. The paper would also benefit from releasing the model or at least providing a reproducible evaluation protocol, given that the quality claims rest on a metric used during architecture search."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the speed claim is real and the engineering is serious, but the paper never evaluates the checkpoint that actually runs on the phone. The 81.14 VBench score is for a 120-frame 432x768 24fps model; the iPhone demo is a 51-frame 512x512 10fps model that was 'further fine-tuned' at 10fps after that evaluation (Appendix F). So the 'on-par quality' half of the central claim is unsupported for the deployed artifact.\n\nWhat is genuinely new: first text-to-video diffusion model that runs on a consumer phone in under five seconds, with a latency-memory joint architecture search over temporal layer types/placements and a tailored adversarial fine-tuning to four steps. The ablations are thorough: they test discriminator head design, image-video joint fine-tuning, noise distribution, step count, and temporal layer placement effects. The hardware benchmark is detailed (1.02s per denoising step, 0.5s VAE decoder, text encoder 6ms), which makes the speed number credible and independently checkable.\n\nSoft spots, in order of severity. First, the checkpoint mismatch above—this is the load-bearing issue for quality. Second, VBench is used both as the search objective (even if a subset, with 25 steps and CFG) and as the final evaluation metric, so part of the score is a selection artifact. Third, no code or weights are released, and the user study only pits the model against OpenSora-1.2 and CogVideoX-2B, omitting the closest few-step baselines like AnimateLCM and T2V-Turbo. None of these undercut the speed measurement; they just mean the quality-parity conclusion is not yet demonstrated.\n\nThe citation and related work are fair, with no obvious attribution problems. The paper is clearly written and honest about its limitations.\n\nBottom line: this deserves a serious referee. I'd send it to review but with the explicit requirement that the authors either release the deployed checkpoint and its evaluation, or add a human study on the actual 10fps model at 512x512. The reader's circularity concern is valid but secondary; the checkpoint mismatch is the one that needs to be fixed.","headline":"The speed claim is solid; the quality claim is orphaned from the deployed model.","tokens_in":20553,"tokens_out":2406,"would_cite":true,"duration_ms":20179,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"A 0.6B-parameter text-to-video model generates a five-second clip on an iPhone 16 Pro Max in under five seconds, with quality scores near the top of the open-source comparison table.","keywords":["mobile video generation","diffusion models","architecture search","adversarial distillation","text-to-video","latency optimization","model pruning","on-device inference"],"falsifier":"Run the 4-step model and a strong server baseline such as CogVideoX-2B on the same set of prompts and collect pairwise human preference judgments on motion, prompt alignment, and aesthetics; if human judges clearly prefer the baseline despite close VBench scores, the paper's quality claim is undercut. A second decisive test is timing the full pipeline on an iPhone 16 Pro Max from text input to displayed video; if end-to-end time exceeds five seconds on the released build, the headline speed claim fails.","tokens_in":19525,"feed_emoji":"🎬","tokens_out":3922,"duration_ms":34764,"temperature":0.7,"pith_summary":"This paper claims that a 0.6B-parameter text-to-video diffusion model can generate a five-second, 512x512 video on an iPhone 16 Pro Max in under five seconds, while scoring on par with much larger server-side video models. The route has three stages: prune a Stable Diffusion 1.5 image UNet into a compact spatial backbone; search among temporal layer types, placements, and counts under measured mobile latency and memory limits; then adversarially fine-tune the resulting model so it needs only four denoising steps and no classifier-free guidance. The authors also compress the video VAE decoder, cutting its 17-frame decode time from 27.2 seconds to 0.5 seconds. If the claim holds, on-device text-to-video generation moves from impractical to near-realtime, changing where and how such models can be deployed.","feed_headline":"A five-second video in five seconds on a phone","feed_subtitle":"A 0.6B-parameter model matches server-side video models and runs on an iPhone 16 Pro Max in under five seconds.","key_machinery":"The engine of the result is a three-part combination. First, a latency-memory joint evolutionary architecture search picks which temporal layers (1D or 3D self-attention, cross-attention, or convolution) to place at which UNet blocks, ranking candidates by the ratio of VBench score gain to measured latency and memory. Second, a latent adversarial fine-tuning recipe with a spatial-temporal discriminator head and joint image-video training distills the 25-step diffusion process into 4 steps without classifier-free guidance. Third, a decoupled and pruned spatial-temporal VAE decoder reduces the decoding bottleneck from 27.2 seconds to 0.5 seconds for a 17-frame clip. Together these components fit a video generation pipeline inside the memory and latency budget of an Apple A18 Pro chip.","core_discovery":"The paper argues that a video diffusion model small and fast enough for a phone can be obtained without training from scratch. Starting from a pruned image-generation UNet, the authors search for the best way to add temporal modeling under strict hardware budgets, then distill the model from 25 denoising steps to 4 using an adversarial objective with a unified spatial-temporal discriminator head. The resulting model, at 0.6B parameters, generates 51 frames at 512x512 resolution in about 4.1 seconds on an iPhone 16 Pro Max, according to the paper's benchmark, and reaches a VBench total score of 81.14, which sits above several open-source server-side models in the reported table. The paper frames this as the first demonstration of real-time text-to-video generation on a mobile device.","pith_inferences":["The paper leaves implicit that the same pipeline could be pushed toward true real-time streaming video generation, where frames are produced continuously rather than as one fixed clip; the authors explicitly note that their current system is conventional one-shot generation, so this is a natural next step rather than a demonstrated capability.","Applying the same latency-memory search and adversarial distillation to a DiT-based video backbone would be a telling test of whether the conclusions generalize beyond UNets, since the paper restricts itself to a UNet and cites quadratic token complexity as the reason.","A direct implication of the search metric being VBench is that the architecture is optimized for what VBench measures; if future benchmarks better capture motion semantics or long-range consistency, the searched architecture may need to be revisited.","The reported per-step latency of about one second suggests that modest further gains in neural engine efficiency, weight quantization, or step reduction could push generation time to the two-second range on the same hardware."],"forward_implications":["If the reported speed holds on the iPhone 16 Pro Max, text-to-video generation no longer requires a cloud round-trip for short clips, which makes private, offline video creation feasible on consumer hardware.","The 4-step adversarial distillation recipe, combined with the compact backbone, suggests that step-count reduction techniques developed for large image models can transfer to much smaller video models when the discriminator is redesigned for joint image-video training.","The latency-memory search criterion is general enough that the same methodology could be applied to other hardware targets, such as Android flagships, laptops, or automotive chips, by rebuilding the latency lookup table for those devices.","Compressing the VAE decoder by more than fifty times shows that decoding, not just denoising, is a major bottleneck for edge video generation, and that it can be addressed separately from the denoiser."],"supporting_citations":[{"why":"Stable Diffusion v1.5 is the pre-trained image model that the spatial backbone is pruned from, supplying the initial generative capability.","marker":"[45]"},{"why":"Structural pruning method used to compress the image UNet to 327M parameters with 2.5x size reduction.","marker":"[11]"},{"why":"Mobile-oriented image diffusion pruning and deployment approach whose speedup and design priors are borrowed for the spatial backbone.","marker":"[29]"},{"why":"AnimateDiff is a temporal inflation baseline that the paper compares against and whose temporal self-attention design is one of the searched options.","marker":"[13]"},{"why":"Stable Video Diffusion provides a temporal convolution inflation baseline and serves as a comparison point for model size and speed.","marker":"[3]"},{"why":"OpenSora contributes the spatial-temporal VAE, the Rectified-Flow-style training recipe, and a comparison baseline in the experiments.","marker":"[77]"},{"why":"VBench is the benchmark used both to score architecture candidates during search and to evaluate the final model, making it the central quality metric.","marker":"[19]"},{"why":"SF-V supplies the adversarial step-distillation framework that the paper adapts, including the discriminator initialization and reconstruction loss.","marker":"[74]"},{"why":"CogVideoX is a server-side DiT baseline used for VBench comparison and as an example of a model that cannot run on a phone.","marker":"[67]"}],"fun_headline_variants":["Five-second videos in five seconds on a phone","SnapGen-V: 5-sec video on an iPhone in under 5 sec","0.6B model creates 5-sec videos on iPhone in 5 sec","Phone generates 5-sec clip in 5 sec, server-like quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality comparison rests on VBench, and VBench is also the metric used during the architecture search to choose the final model; if VBench scores diverge from what human viewers actually prefer, the on-par quality conclusion is not supported.","fun_headline_variants_meta":{"raw":{"variants":["Five-second videos in five seconds on a phone","SnapGen-V: 5-sec video on an iPhone in under 5 sec","0.6B model creates 5-sec videos on iPhone in 5 sec","Phone generates 5-sec clip in 5 sec, server-like quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":2036,"prompt_tokens":904,"completion_tokens":1132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1050}},"tokens_in":520,"tokens_out":1132,"duration_ms":10284,"temperature":1.0,"reasoning_tokens":1050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:53:58.733714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 4-step model and a strong server baseline such as CogVideoX-2B on the same set of prompts and collect pairwise human preference judgments on motion, prompt alignment, and aesthetics; if human judges clearly prefer the baseline despite close VBench scores, the paper's quality claim is undercut. A second decisive test is timing the full pipeline on an iPhone 16 Pro Max from text input to displayed video; if end-to-end time exceeds five seconds on the released build, the headline speed claim fails.","supporting_citations":[{"cited_title":"Structural pruning for diffusion models","cited_arxiv_id":null,"evidence_quote":"Structural pruning method used to compress the image UNet to 327M parameters with 2.5x size reduction."},{"cited_title":"Snap- fusion: Text-to-image diffusion model on mobile devices within two seconds","cited_arxiv_id":null,"evidence_quote":"Mobile-oriented image diffusion pruning and deployment approach whose speedup and design priors are borrowed for the spatial backbone."},{"cited_title":"Open-sora: Democratizing efficient video production for all, 2024","cited_arxiv_id":null,"evidence_quote":"OpenSora contributes the spatial-temporal VAE, the Rectified-Flow-style training recipe, and a comparison baseline in the experiments."},{"cited_title":"VBench: Com- prehensive benchmark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"VBench is the benchmark used both to score architecture candidates during search and to evaluate the final model, making it the central quality metric."}],"review_version":1}