{"id":"dd1e681e-f81e-4732-a881-b6d55fdb28ae","arxiv_id":"2412.04432","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Diffusion-conditioned denoising trains a video tokenizer whose features support both video question answering and text-to-video generation in a single LLM.","lead":"Divot trains a video tokenizer by asking a video diffusion model to denoise clips using only the tokenizer's features as a guide, then reuses that diffusion model as a video decoder. The result, Divot-LLM, is a single 7B model that can answer questions about videos and generate short video clips from text.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The denoising objective cannot certify temporal encoding: with DynamiCrafter's U-Net as the denoiser, the tokenizer could supply appearance only while the U-Net's motion prior supplies dynamics, and no ablation varies temporal order or frame rate.","rationale":"The paper's proposal is coherent and the experiments are extensive. The diffusion-conditioned tokenizer is a reasonable self-supervised objective, and the comparison against caption-loss tokenizer is a useful baseline. However, the load-bearing premise is an attribution claim: denoising success is taken as proof that the tokenizer encodes temporal dynamics. This is exactly the reader's weakest assumption. The mechanism makes the attribution fragile because the denoiser is a pre-trained video diffusion U-Net (DynamiCrafter) with strong motion priors; the tokenizer sees only 5 low-fps frames, so the U-Net could plausibly hallucinate dynamics. The paper provides no ablation that isolates temporal information: Table 6 compares diffusion vs caption loss, but both conditions use the same sparse frame input, and neither varies frame rate, token count, or temporal order. A shuffled-frame or single-frame training run would settle whether the representation is order-sensitive. If the tokenizer is order-agnostic, the central claim is false, though the system might still work as a spatial encoder coupled to a video generator. I do not see this as grounds for rejection: the released code and the positive ablation provide partial support, and the condition is testable. Hence the reader's CONDITIONAL verdict stands unchanged.","tokens_in":18610,"tokens_out":8473,"duration_ms":94604,"concrete_test":"Using the released code, retrain the Divot tokenizer under the exact protocol of Sec. 3.1.2/3.1.3 but with the 5 input frames presented in random temporal order (re-initializing the temporal transformer to prevent order leakage via pre-trained positional embeddings), keeping the same U-Net initialization and data. Then compare reconstruction FVD on held-out videos and, after the standard LLM integration, MVBench and Perception-Test accuracy. If shuffled-order training matches the original metrics within a small margin, the tokenizer encodes no temporal order, contradicting the paper's central premise. If it degrades substantially, temporal order is being captured and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's load-bearing claim (Sec. 1, Sec. 3.1.2) is that successful denoising of dense 8 fps VAE latents conditioned on Divot features proves the tokenizer has captured 'robust spatial and temporal information.' This implication is not secure. The conditioning tokenizer sees only 5 frames at 2 fps (Sec. 3.1.3) and compresses them to 64 tokens, while the denoiser is the DynamiCrafter U-Net (Sec. 3.1.3), a pre-trained video diffusion model with a strong prior for coherent motion. During end-to-end training, the U-Net can minimize the denoising loss by using its own temporal convolutions and attention to generate plausible motion between the sparsely sampled frames, with the tokenizer supplying only appearance. Nothing in the loss forces the tokenizer to encode temporal order or true dynamics; the gradient signal only requires that the combined system reconstruct the video. The ablation in Table 6 compares diffusion loss against caption loss but does not vary the amount or order of temporal information available to the tokenizer. Consequently, the observed comprehension and generation gains could stem from higher-fidelity spatial features learned under reconstruction pressure, rather than from temporal dynamics. If the tokenizer is order-agnostic, the central premise fails: the method would still work, but the contribution would be a spatial encoder plus a pre-trained video generator, not a spatiotemporal tokenizer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Divot, a video tokenizer trained by conditioning a pre-trained video diffusion U-Net (DynamiCrafter) on sparse-frame features and asking it to denoise dense VAE latents of the same video; after training, the same U-Net is used as a de-tokenizer to decode videos from the tokenizer's features. The authors then build Divot-LLM by combining this tokenizer with Mistral-7B, using video-to-text autoregression for comprehension and GMM-based sampling for text-to-video generation. The model is evaluated on several video QA benchmarks, zero-shot MSR-VTT generation, and a video storytelling demo. The paper's central claim is that successful denoising of dense 8 fps video latents conditioned on Divot features certifies that the tokenizer has captured robust spatial and temporal information.","tokens_in":18903,"tokens_out":8585,"duration_ms":80342,"significance":"If the central attribution is established, the paper offers a useful step toward unified video comprehension and generation: a self-supervised tokenizer trained without caption supervision, a diffusion model that doubles as a de-tokenizer, and a GMM framework for fitting continuous video features with an LLM. The release of models and code would aid reproducibility, and the ablations in Tables 6 and 8 address several design choices that the community is actively exploring. The main risk is that the paper's core conceptual claim, that denoising success proves temporal encoding, is not supported by the current experiments; the ablations do not isolate temporal information from spatial appearance or from the U-Net's own motion prior.","major_comments":[{"comment":"The central claim that successful denoising of dense 8 fps latents from 5 sparse 2 fps frames certifies temporal encoding is not established. Because the conditioning tokenizer sees appearance-rich sparse frames and the denoising U-Net is a pre-trained DynamiCrafter with temporal convolutional and attention priors, the loss could in principle be minimized with appearance-only conditioning plus the U-Net's own motion prior. The ablation in Table 6 compares diffusion loss against caption loss but does not vary frame rate, token count, or temporal order, so it does not rule out this alternative. I would ask for a permutation/shuffle ablation (or a single-frame condition) and, if temporal encoding cannot be demonstrated, the paper should reframe the contribution as a spatial encoder coupled with a pre-trained motion prior rather than as a spatiotemporal tokenizer.","section":"Sec. 1, Sec. 3.1.2, Sec. 3.1.3"},{"comment":"The diffusion-vs-caption tokenizer ablation is confounded with training data. The text says both models are pre-trained on Valley and instruction-tuned on Video-ChatGPT, but the caption-loss tokenizer is pre-trained with captioning loss on Valley, while the Divot tokenizer is pre-trained on 10M pure videos (WebVid-10M and Panda-70M, per Sec. 3.1.2). The improvement in Table 6 could therefore reflect additional training data or data domain rather than the diffusion objective. Please match data scale/domain between the two tokenizer training regimes, or explicitly account for this mismatch in the conclusions.","section":"Sec. 4.3, Table 6"},{"comment":"The 'zero-shot' label on the MSR-VTT generation evaluation is not yet supported. Appendix A.2 describes a de-tokenizer adaptation technique that fine-tunes the de-tokenizer on features sampled from the predicted GMM distribution, but does not state which videos are used. If this adaptation uses the MSR-VTT training split, Table 4 is not a zero-shot comparison and must either be re-labeled or the adaptation must be removed/disclosed; the authors should also specify whether the CLIPSIM/FVD evaluation uses one sampled set of 16-frame clips per caption or multiple samples.","section":"Appendix A.2, Table 4"},{"comment":"The quantitative comparisons are reported as single point estimates with no error bars or repeated runs for stochastic procedures (GPT-3.5 scoring, FVD, GMM sampling). As a result, small differences such as the CLIPSIM difference between Divot-LLM (0.2938) and VILA-U (0.2937) in Table 4 are not meaningful, and the claim of 'comparable' performance should be supported by variance estimates or multiple seeds. In addition, the comparison rows in Tables 3 and 4 differ in training data composition and evaluation protocols (e.g., 4.8M video-text pairs for pretraining versus 10M pure videos for tokenizer training), so the table should explicitly note which number is being used for each row.","section":"Tables 3 and 4"}],"minor_comments":[{"comment":"The abstract uses 'Divot-Vicuna' while the rest of the paper consistently uses 'Divot-LLM'; please unify the model name.","section":"Abstract"},{"comment":"The text says 'adds Gaussian noise θ' to the latent; this should be ϵ_t or z_t, consistent with Eq. (1).","section":"Sec. 3.1.2"},{"comment":"The text refers to 'Tab. 5' and 'Tab. 7', but the displayed tables are numbered Table 6 and Table 8; fix the cross-references and table numbering.","section":"Sec. 3.2 and Sec. 4.3"},{"comment":"The appendix states that training uses 'v prediction' for denoising, but Eq. (1) is written for ϵ-prediction; please define the v-prediction parametrization and its relationship to Eq. (1).","section":"Appendix A.1"},{"comment":"The GMM mixture size is fixed to k=16 with no sensitivity analysis; please report at least one alternative value of k to justify the choice.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"Dear Editor, the main risk is the temporal-attribution claim: the paper's central conceptual contribution is that denoising success certifies spatiotemporal features, and the current ablations do not isolate that claim. I would not reject because the system is well-engineered and the GMM modeling comparison is useful, but the revision must either add the missing ablations or weaken the claim. Please also ensure the zero-shot claim is clarified before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: Divot is a genuinely new combination — a continuous video tokenizer trained by making a pre-trained video diffusion U-Net de-noise dense VAE latents conditioned on sparse tokenizer features, with that same U-Net later acting as a de-tokenizer. It works well enough to put a 7B LLM near the top of both video QA and text-to-video with only 4.8M video-text pairs, and the code is out. That is a real contribution to the unified-MLLM subfield.\n\nWhat the paper does well: the design is simple and motivated, the caption-loss tokenizer ablation (Table 6) gives direct evidence that the diffusion objective helps comprehension, and the comparison of MSE vs diffusion vs GMM for fitting continuous features (Tables 7-8) is the kind of ablation that moves the field forward. Releasing the code makes the empirical claims checkable.\n\nThe soft spots are mostly about what the central claim does not prove. The paper's core premise is that successful denoising certifies that the tokenizer captured temporal dynamics. That implication is not secure. The tokenizer sees 5 frames at 2 fps; the denoiser is DynamiCrafter, a video diffusion model with a strong learned motion prior. Nothing in the loss forces the tokenizer to encode motion, and no ablation varies frame rate, token count, or temporal order for the comprehension tasks. The observed gains could come from better spatial features under reconstruction pressure. This is a load-bearing gap, not a fatal flaw — the method still works as a spatial encoder plus a strong generator — but the paper should be honest about it and test it.\n\nSecond, the de-tokenizer adaptation in Appendix A.2 is performed after instruction tuning, and the paper does not state whether MSR-VTT data was used there. That matters for the 'zero-shot' generation numbers. If it was, the comparison is no longer zero-shot for the de-tokenizer. It is a transparency issue the authors can fix in a revision. Third, no error bars on FVD/CLIPSIM or comprehension accuracies; with stochastic metrics across seeds this is worth reporting. The mixed training-data-size comparisons are acknowledged in the tables, so that is minor.\n\nThe stress-test note is right that the temporal certification is weak; I checked the paper and the missing ablation is real. But it is a missing experiment, not an internal contradiction.\n\nBottom line: this paper deserves a serious referee. I would send it out, with the expectation that the authors add a temporal-sensitivity ablation and clarify the de-tokenizer adaptation. For anyone working on unified video LLMs, it is worth reading now.","headline":"A genuinely new continuous video tokenizer trained by diffusion denoising, with real empirical promise, but the temporal-encoding claim is under-supported and the generation eval has a transparency gap.","tokens_in":19452,"tokens_out":2959,"would_cite":true,"duration_ms":40264,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Divot claims that a video tokenizer has learned adequate spatial and temporal representations exactly when a video diffusion model can denoise video from those representations, and that the same diffusion model can then decode them back…","keywords":["video tokenizer","diffusion models","self-supervised video representation learning","unified video understanding and generation","Gaussian mixture model","continuous visual tokens","video question answering","text-to-video generation"],"falsifier":"Train an otherwise identical Divot tokenizer whose features come from a single frame instead of five, and compare video-question accuracy plus the fidelity of reconstructed motion (for example FVD on action clips); if the single-frame version matches the five-frame version, the claim that the tokenizer captures temporal dynamics fails.","tokens_in":18391,"feed_emoji":"🎬","tokens_out":7371,"duration_ms":63920,"temperature":0.7,"pith_summary":"The paper proposes Divot, a video tokenizer trained by a self-supervised diffusion task: a video diffusion model must denoise densely sampled video latents using only the tokenizer's features from five sparsely sampled frames. The authors' claim is that the tokenizer has captured both appearance and motion if the denoiser can succeed with those features, and that the denoiser then becomes a natural de-tokenizer for generation. Divot-LLM attaches this tokenizer to a 7-billion-parameter language model and trains the model to read video via next-word prediction and to write video by predicting Gaussian-mixture distributions over the continuous token space. If right, the work shows that one model can both understand and generate video without discrete visual codes, and that video representation learning can be driven by pure video rather than captions.","feed_headline":"One diffusion U-Net both tokenizes and generates video","feed_subtitle":"Sparse frames become 64 continuous tokens that let a single 7B LLM answer video questions and create clips.","key_machinery":"The central mechanism is the paired tokenizer and de-tokenizer. The tokenizer is a pretrained ViT encoder followed by a spatial-temporal transformer and a Perceiver Resampler, compressing five frames at 2 fps into 64 continuous tokens. The de-tokenizer is a pretrained video diffusion U-Net (DynamiCrafter) whose denoising of 8 fps latent video is conditioned on those tokens via cross-attention; the denoising loss is the tokenizer's training signal. For generation, the LLM predicts per-token Gaussian Mixture Model parameters (with 16 mixture components) and samples from that distribution to feed the de-tokenizer.","core_discovery":"On the paper's own terms, the discovery is that a pretrained video diffusion U-Net, stripped of its image-conditioning branch, can serve as both the training objective and the decoder for a continuous video tokenizer. Sparse frames are converted by a ViT encoder, a spatial-temporal transformer, and a Perceiver Resampler into 64 tokens; the U-Net then denoises the compressed latent representation of densely sampled frames conditioned on those tokens through cross-attention, and the noise-prediction error is backpropagated into the tokenizer. After training, the same U-Net decodes the tokens into video clips. For LLM-driven generation, the paper reports that minimizing mean-squared error between LLM outputs and video features yields averaged, repeating video, while predicting the parameters of a 16-component Gaussian Mixture Model and sampling from it gives better generation quality and semantic alignment. The authors also report that the diffusion-trained tokenizer improves video-question benchmarks over the same architecture trained with captioning loss, and that Divot-LLM reaches competitive comprehension results at 7B scale.","pith_inferences":["Editorial inference: if the denoising-conditioning principle transfers, the same recipe could be used to learn tokenizers for other continuous modalities such as audio or sensor streams, wherever a pretrained diffusion decoder exists.","Editorial inference: the paper does not isolate how much temporal signal comes from the tokenizer versus the U-Net's prior; a natural next experiment varies the number of input frames and the frame rate while holding the decoder fixed.","Editorial inference: because generation samples from a learned GMM, the spread of that distribution could serve as a per-clip uncertainty signal, which the paper does not analyze.","Editorial inference: the reported strength of continuous over discrete tokens suggests re-examining hybrid tokenizers where discrete semantic tokens and continuous motion tokens are combined."],"forward_implications":["A unified 7B model can answer video questions and generate video from text without converting video into discrete codes.","Video representation learning can proceed from large unlabeled video corpora, since the diffusion loss requires no captions.","The diffusion decoder doubles as a de-tokenizer, so any representation the tokenizer produces is immediately decodable into a video clip.","Probabilistic generation, rather than deterministic regression, is what makes LLM prediction of high-level continuous video features viable.","The same tokenizer transfers to new visual domains such as animation by fine-tuning only the de-tokenizer."],"supporting_citations":[{"why":"Supplies the latent diffusion denoising objective used to train the tokenizer.","marker":"[50]"},{"why":"Provides the pretrained video denoising U-Net that becomes the de-tokenizer after removing its image-conditioning branch.","marker":"[78]"},{"why":"Supplies the Perceiver Resampler that compresses frame features into 64 fixed tokens, the representation the LLM reads and predicts.","marker":"[1]"},{"why":"Introduces probabilistic modeling of continuous visual features, which Divot adapts for video generation.","marker":"[33]"},{"why":"The discrete-token unified video model Divot is compared against and seeks to improve on for comprehension and generation.","marker":"[26]"},{"why":"Documents the comprehension degradation caused by discrete visual tokens, motivating continuous representations.","marker":"[77]"},{"why":"Supplies the GMM parameter-prediction idea used to model Divot feature distributions.","marker":"[61]"},{"why":"WebVid-10M is the main video-text source for training the tokenizer and Divot-LLM.","marker":"[2]"}],"fun_headline_variants":["Diffusion U-Net tokenizes and generates video in one model","Video tokenizer from diffusion U-Net powers LLM video tasks","Divot uses diffusion to tokenize video for LLM comprehension and generation","Diffusion powers a video tokenizer that also generates clips","One diffusion U-Net serves as video tokenizer and decoder for LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a denoiser's success at reconstructing full video from the tokenizer's five sparse frames is caused by temporal information in those tokens, not by the denoiser's own motion habits or by static appearance cues.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion U-Net tokenizes and generates video in one model","Video tokenizer from diffusion U-Net powers LLM video tasks","Divot uses diffusion to tokenize video for LLM comprehension and generation","Diffusion powers a video tokenizer that also generates clips","One diffusion U-Net serves as video tokenizer and decoder for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2896,"prompt_tokens":1009,"completion_tokens":1887,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":1795}},"tokens_in":625,"tokens_out":1887,"duration_ms":11958,"temperature":1.0,"reasoning_tokens":1795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:24:06.761122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an otherwise identical Divot tokenizer whose features come from a single frame instead of five, and compare video-question accuracy plus the fidelity of reconstructed motion (for example FVD on action clips); if the single-frame version matches the five-frame version, the claim that the tokenizer captures temporal dynamics fails.","supporting_citations":[{"cited_title":"Givt: Generative infinite-vocabulary transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the GMM parameter-prediction idea used to model Divot feature distributions."}],"review_version":1}