{"id":"3cb9fd4d-7d9f-4271-a844-72ccca2b73aa","arxiv_id":"2507.02862","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"By conditioning video tokenization on an unquantized reference frame, RefTok reports substantially better reconstruction and generation quality than existing tokenizers at claimed equal compression.","lead":"RefTok is a video tokenizer that keeps one frame unquantized and uses it as a reference when encoding and decoding the other frames, improving reconstruction of text, faces, and small details. It reports large gains over Cosmos and MAGVIT tokenizers, though the comparisons may not account for the extra bits the reference frame costs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RefTok's 'same compression ratio' claim is not established: the unquantized reference frame is a free side channel unavailable to baselines, so the 36.7% reconstruction gain and 27.9% generation gain may largely reflect uncharged reference information rather than tokenizer quality.","rationale":"I acknowledge the paper's independent support: the method is simple and plausible, the qualitative figures show genuine conditioning-based reconstruction of text, faces, and fine patterns, the speed and memory measurements are concrete, and the ablations honestly report that more reference frames do not help. No formal verification or code release is claimed, so the quantitative results rest on the paper's own pipeline. The central quantitative claim is the 36.7% average improvement at 'same or higher' compression ratios, and the load-bearing condition is that the comparison is made at equal information cost. That condition fails as reported because the unquantized reference frame is outside the bit budget. The issue is not that conditioning exists—conditional generation legitimately has a given reference—but that the paper presents the result as a tokenizer improvement at equal compression and as a Pareto-front shift. The reader's weakest_assumption identified the same issue, and I agree. A rate-distortion test that charges reference bits would settle whether the advantage survives. Because the issue is addressable and does not necessarily invalidate the method's value for its intended conditional setting, I would keep the reader's CONDITIONAL verdict rather than escalate to REJECT.","tokens_in":11612,"tokens_out":4767,"duration_ms":62319,"concrete_test":"Recompute the rate-distortion comparison of Figure 4 by charging reference bits: for each clip in K600 and BAIR, encode the reference frame at several bitrates (e.g., JPEG quality 90/70/50 or a neural image codec), add those bits to RefTok's quantized token stream, and plot PSNR versus total bits-per-pixel against Cosmos and MAGVIT at their reported bitrates. Also run a matched-conditioning ablation where the reference-less baselines are given the same first frame as an extra input, or where RefTok's reference is quantized to the same codebook bitrate as target tokens. If RefTok no longer dominates at equal total bitrate, the headline 36.7% gain is an accounting artifact; if it still dominates, the compression claim survives. Separately, report whether MAGVIT in Table 2 received the same first-frame conditioning; if not, Table 2's comparison is confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 and Figure 4 report a 1024:1 compression ratio while Equation (3) and Figure 2 send h_r = E(x_r) around the quantizer, giving the decoder an unquantized, near-continuous embedding of the ground-truth first frame. None of the reference-less baselines (Cosmos, MAGVIT, etc.) receive this information, and Section 4.4 states 'we use the first frame as the reference frame' without adding its bit cost to any reported number. One full-resolution reference per 16-frame clip is a side channel whose cost must be charged if 'same compression ratio' is claimed; with reference bits included, RefTok operates at a materially lower compression ratio. The Pareto-front plot therefore conflates a free conditioning signal with architectural gains. The generation result in Section 4.5 and Table 2 is exposed to the same issue: RefTok's BAIR model is trained on tokens produced with ground-truth reference conditioning, whereas the MAGVIT baselines are compared without any stated reference-conditioning protocol, so the 27.9% improvement may partly reflect extra information supplied at inference. The paper's own limitation section acknowledges that a reference frame is required, but never quantifies its bitrate. This is not an internal inconsistency: for genuinely conditional image-to-video tasks, a given reference frame is a legitimate input. But the abstract and Figure 4 generalize to 'same compression ratio' in a way that the current evaluation does not support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RefTok proposes a reference-based video tokenizer in which an unquantized reference frame embedding bypasses the quantization bottleneck and conditions the decoder, while all other frames are quantized as usual. The paper claims that this design improves reconstruction PSNR/SSIM/LPIPS by an average of 36.7% over Cosmos and MAGVIT at the same or higher compression ratios across K600, UCF-101, BAIR, and DAVIS, and that a generation model trained on RefTok tokens outperforms both MAGVIT-B and MAGVIT-L on BAIR by an average of 27.9%. The method is evaluated with a fixed first-frame reference, along with ablations, speed/memory measurements, and a zero-shot video-editing application.","tokens_in":11930,"tokens_out":5010,"duration_ms":60187,"significance":"If the quantitative claims were fully supported, RefTok would be a meaningful step for conditional video generation and editing: the idea of letting reference frames skip quantization is simple, plausible, and well-motivated by temporal redundancy, and the qualitative examples of text, face, and fine-pattern preservation are compelling. The paper also evaluates on four datasets, reports speed and memory efficiency, and demonstrates a practical editing use case. However, the central significance depends on the compression-ratio accounting and on whether baselines receive matching conditioning; as presented, the quantitative gains may largely reflect uncharged reference information rather than tokenizer quality. The contribution is best viewed as a conditional tokenization design whose unconditional compression claims are not yet established.","major_comments":[{"comment":"The headline 1024:1 compression ratio does not account for the reference frame. In Eq. (3), the reference embedding hr = E(xr) bypasses the quantizer and is passed to the decoder, and §4.4 states that the first frame is used as the reference. This is an unquantized, effectively high-rate representation of a ground-truth frame, and no bits are charged for it in the reported ratio. During reconstruction of a 16-frame clip, the reference frame is a substantial side channel that reference-less baselines (Cosmos, MAGVIT, etc.) do not receive. Consequently, the Pareto plot in Figure 4 and the average 36.7% improvement in Table 1 conflate free conditioning information with architectural or tokenizer gains. Please provide rate-distortion comparisons that include the reference-frame bit cost (e.g., by quantizing or otherwise coding the reference and adding its bitrate), or explicitly restrict all claims to conditional reconstruction where the reference is a free input and remove the unconditional 'same compression ratio' wording.","section":"§4.4, Eq. (3), Fig. 4"},{"comment":"The generation comparison does not establish that baselines receive the same conditioning. RefTok's tokens are produced with ground-truth reference conditioning, and the generation model is trained on these tokens; the MAGVIT-B/L numbers in Table 2 are reported without any stated reference-conditioning protocol. If the MAGVIT generators are not conditioned on the first frame at inference, the reported 27.9% average improvement may reflect extra information supplied to RefTok rather than a fair tokenizer comparison. Please specify the exact conditioning protocol for every method, including whether the reference (first) frame is provided to the baseline generators, and report results under matched conditioning wherever possible.","section":"§4.5, Table 2"},{"comment":"All quantitative results appear to be single runs with no error bars, confidence intervals, or significance tests. The abstract's claim of 'significantly outperforms' and the averages of 36.7% and 27.9% are therefore not statistically supported, particularly for FVD where run-to-run variation can be large. Please state the number of seeds, report means with variance (or confidence intervals), and indicate which differences are statistically significant.","section":"Tables 1–3"}],"minor_comments":[{"comment":"There is a typo in 'MAGVIG-B-sized'; it should read 'MAGVIT-B-sized'.","section":"§1"},{"comment":"The phrase 'latent-space probabilitics models' contains a typo; 'probabilistic' is intended.","section":"§2"},{"comment":"The compression-ratio axis appears to be logarithmic but is not labeled as such, and several legend labels are difficult to read; please improve the figure clarity.","section":"Fig. 4"},{"comment":"The paper refers to an appendix for training details, but no appendix is included in the submitted manuscript; please include the appendix or provide sufficient details for reproduction (e.g., exact ViTMAE initialization, 2D-to-3D position-embedding inflation, and fine-tuning schedule).","section":"§3.3 / Training details"},{"comment":"The ablation on the number of reference frames reports only L1 error with no variance; adding more reference frames slightly degrades performance, which is not discussed. A brief explanation or additional qualitative comparison would help.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the compression-ratio accounting and the unmatched conditioning in the generation comparison. These are fixable within the manuscript's scope if the authors either supply rate-distortion results that charge reference bits or reframe the contribution as conditional video tokenization with a freely given reference frame. The qualitative evidence is strong enough that I would not recommend rejection, but the current headline quantitative claims should not appear in their present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know before you read it: the core architecture is genuinely interesting, but the headline numbers are built on a free side channel. RefTok passes an unquantized embedding of the reference frame around the quantizer, so the decoder sees near-continuous information about the ground-truth frame. That's the whole trick—and it works, in that the qualitative reconstructions of text, faces, and fine patterns are visibly better than reference-less baselines. But the 1024:1 compression ratio is calculated only on the quantized tokens; the reference frame's bit cost is never counted. Section 4.4 says 'we use the first frame as the reference frame' without adding its bits, and the baselines (Cosmos, MAGVIT) don't receive any equivalent conditioning. So the 36.7% average improvement is not at the same compression ratio.\n\nThat said, the novel combination of an unquantized reference bypass, a one-way attention barrier, and decoder-only conditioning is not in the cited tokenizer literature, and the mechanism is plausible. The downstream generation result on BAIR is suggestive, though the same caveat applies: the MAGVIT baselines aren't given a reference input, so part of the 27.9% improvement may be extra information rather than better tokens.\n\nThe paper's own limitations section admits a reference frame is required but never quantifies its bitrate—that's a missing support that should have been flagged. The evaluation also shows single runs with no error bars, which is a minor issue for a tokenizer paper but worth noting. No code is released, so the reconstruction numbers are hard to verify independently.\n\nThis is a systems paper, not a derivation, so there's no fitted-constant circularity. The reconstruction gain is structural—the decoder is handed the ground-truth reference—but that's the method, not a cheat, as long as the problem is framed as conditional video generation where the reference is a given input. The overreach is in claiming 'same compression ratio' in the abstract and Figure 4. If the authors reframe the contribution as a conditional tokenizer and add reference-conditioned baselines, the paper would be solid.\n\nIs it worth a serious referee? Yes. The idea is novel and the qualitative evidence is strong enough to demand careful scrutiny. Send it to review, but tell the reviewers to focus on the bitrate accounting and baseline conditioning.\n\nWould I cite it? Probably yes, as related work, once I can point to a version that charges reference bits or explicitly frames the conditional setting.","headline":"RefTok's reference-bypass idea is novel and the qualitative gains are real, but the headline compression ratio ignores the cost of the unquantized reference side channel.","tokens_in":12429,"tokens_out":2737,"would_cite":true,"duration_ms":30251,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By keeping one reference frame unquantized, RefTok reconstructs text, faces, and fine patterns that reference-less tokenizers lose, improving reconstruction metrics by 36.7% at equal compression.","keywords":["video tokenization","reference-based compression","quantization bypass","video generation","temporal redundancy","conditional video generation","video reconstruction","discrete latent space"],"falsifier":"Recompute rate-distortion with the reference frame's compressed bit cost included, for example by compressing the first frame with an intra-frame image codec and adding its bytes to the target tokens' budget, or evaluate with reference frames that carry little useful information such as static or heavily corrupted frames; if the PSNR, SSIM, and LPIPS gains over the baselines vanish or reverse, the compression-equivalence premise fails.","tokens_in":11448,"feed_emoji":"🎞️","tokens_out":6917,"duration_ms":75289,"temperature":0.7,"pith_summary":"RefTok is a video tokenizer that keeps one reference frame unquantized and encodes every other frame relative to it. The paper's claim is that this single design change lets a neural tokenizer reconstruct a larger range of RGB content, such as legible text, stable faces, and small patterns, than reference-less tokenizers at the same or higher compression. Across four video datasets, RefTok reports average gains of 36.7% across PSNR, SSIM, and LPIPS at a 1024:1 compression ratio, and a generation model trained on its latents outperforms a baseline with four times more parameters by an average of 27.9%. The point is to reopen a design dimension, reference frames, that learned tokenizers had dropped, bringing neural video tokenization closer to classical codecs. If this holds, discrete video tokenization can be made more accurate without sacrificing compression.","feed_headline":"Keeping one frame unquantized lifts video token quality 36.7%","feed_subtitle":"RefTok cuts the encoder's job to encoding what changed, preserving text, faces, and fine detail at 1024:1 compression.","key_machinery":"The central mechanism is the quantization bypass: an unquantized reference frame's encoder representation $h_r$ is fed directly to the decoder as a conditioning signal, while target frames' tokens are quantized to $z_t$ and decoded as $\\hat{x}_t = D(z_t | h_r)$. A one-way attention mask in a masked-autoencoder-style transformer keeps target information from leaking into the reference representation, preserving causality. Techniques such as training at higher frame intervals and codebook splitting guard against reference posterior collapse, where the decoder would simply copy the reference frame and ignore the latent tokens.","core_discovery":"On its own terms, RefTok's central discovery is that the bottleneck layer need not quantize everything. By passing continuous reference-frame features straight to the decoder and quantizing only the target tokens under a one-way attention barrier, the encoder has to represent only the temporal difference between reference and target, and the decoder can consult an information-rich continuous reference when reconstructing. This shifts the hard part of video compression from inventing all pixels from a small codebook to encoding what changed, which is why RefTok preserves text, faces, and fine patterns that reference-less tokenizers distort. The paper positions this as a Pareto improvement: better reconstruction at the same or higher compression ratio, with faster and more memory-efficient inference.","pith_inferences":["If the reference frame's bits were counted in the compression ratio, for instance by compressing the first frame with a strong image codec and adding those bytes to the token budget, the effective rate would be higher than 1024:1, so the honest rate-distortion gain remains an open question.","The method is most natural when a high-quality reference frame already exists, such as keyframe-based coding or image-to-video generation; for unconditional generation, the model must first produce an anchor frame, so the gains may shift downstream.","The reference-posterior-collapse fixes suggest reference frames should be informative but not too similar to the targets, making adaptive reference-frame selection among candidate frames a direct and testable extension.","If skipping quantization is the source of the gains, then codebook size and quality matter less for video than previously thought; the bottleneck's role becomes representing residuals rather than full frames."],"forward_implications":["At the same 1024:1 compression, RefTok reconstructs text, faces, and repetitive patterns from the reference frame, where reference-less tokenizers blur or distort them.","The reported 36.7% average improvement across PSNR, SSIM, and LPIPS means reference-based conditioning can push the Pareto front of discrete video tokenization instead of trading quality against compression.","Downstream video-generation models trained on RefTok latents can match or beat a model with roughly 4 times more parameters, so better tokenization directly improves generation quality without a larger generator.","Because RefTok keeps the encoder and decoder architecture and only changes how frames are routed around the bottleneck, existing tokenizers could adopt the reference-bypass design at a low compute cost.","Conditional applications such as zero-shot video editing become single-forward-pass operations by swapping in an edited reference frame at decoding time."],"supporting_citations":[{"why":"Supplies the masked-autoencoder encoder and decoder that RefTok initializes, and whose pass-through behavior enables the quantization bypass.","marker":"[16]"},{"why":"Provides the VQGAN-style tokenizer architecture that RefTok modifies with reference conditioning.","marker":"[13]"},{"why":"Serves as the state-of-the-art reference-less video tokenizer baseline in the reconstruction comparisons.","marker":"[29]"},{"why":"Serves as the discrete video tokenizer baseline and supplies the generation-model training recipe and hyperparameters used in the downstream BAIR experiment.","marker":"[48]"},{"why":"Represents the causal video-tokenizer and finite-scalar-quantization line that RefTok contrasts with, motivating decoder access to the unquantized reference.","marker":"[49]"},{"why":"Establishes the vector-quantization bottleneck whose information deficiency RefTok's reference bypass is designed to address.","marker":"[37]"}],"fun_headline_variants":["Unquantized reference frame boosts video token quality by 36.7%","With an unquantized reference, video tokens improve 36.7%","Reference-based tokenization lifts video quality 36.7%","Video tokenizer with raw reference beats SOTA by 36.7%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 1024:1 compression ratio does not include the bits needed to store or transmit the unquantized reference frame, and if those bits are counted, or if the reference is not a clean, informative first frame, the reported gains shrink.","fun_headline_variants_meta":{"raw":{"variants":["Unquantized reference frame boosts video token quality by 36.7%","With an unquantized reference, video tokens improve 36.7%","Reference-based tokenization lifts video quality 36.7%","Video tokenizer with raw reference beats SOTA by 36.7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00109,"raw_usage":{"total_tokens":4545,"prompt_tokens":930,"completion_tokens":3615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":3535}},"tokens_in":546,"tokens_out":3615,"duration_ms":25983,"temperature":1.0,"reasoning_tokens":3535,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:19:08.978269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute rate-distortion with the reference frame's compressed bit cost included, for example by compressing the first frame with an intra-frame image codec and adding its bytes to the target tokens' budget, or evaluate with reference frames that carry little useful information such as static or heavily corrupted frames; if the PSNR, SSIM, and LPIPS gains over the baselines vanish or reverse, the compression-equivalence premise fails.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the masked-autoencoder encoder and decoder that RefTok initializes, and whose pass-through behavior enables the quantization bypass."},{"cited_title":"Cosmos tok- enizer: A suite of image and video neural tokenizers, 2024","cited_arxiv_id":null,"evidence_quote":"Serves as the state-of-the-art reference-less video tokenizer baseline in the reconstruction comparisons."},{"cited_title":"Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang","cited_arxiv_id":null,"evidence_quote":"Serves as the discrete video tokenizer baseline and supplies the generation-model training recipe and hyperparameters used in the downstream BAIR experiment."},{"cited_title":"Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vigh- nesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G","cited_arxiv_id":null,"evidence_quote":"Represents the causal video-tokenizer and finite-scalar-quantization line that RefTok contrasts with, motivating decoder access to the unquantized reference."}],"review_version":1}