{"id":"bac9eb85-c776-4394-8ac2-f68ca24b2088","arxiv_id":"2412.10958","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A soft-categorical image tokenizer compresses images to 32-64 tokens and reaches competitive ImageNet generation FID with large speedups.","lead":"SoftVQ-VAE is an image tokenizer that replaces VQ-VAE's hard pick-one codebook step with a soft blend of many codewords, packing a 256x256 image into as few as 32 or 64 tokens. If the results hold, it could make diffusion and autoregressive image generators several times cheaper to train and run, at a small FID cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline few-token FIDs are not yet attribution-tested against a scale-matched VQ/AE baseline; DINOv2 alignment may be doing most of the work.","rationale":"The paper is a solid systems contribution: it reports reproducible-looking experiments, releases code, runs a genuine high-compression regime (32/64 tokens), and includes controlled ablations (Tables 3, 4, 6) that many similar papers omit. The efficiency gains (10-55x throughput, lower GFLOPs) are well supported by the token-count reduction. The strongest defensible reading is 'a DINOv2-aligned continuous tokenizer with soft codebook aggregation gives near-SOTA generation at extreme compression.' The reader's weakest-assumption diagnosis is right that DINOv2 alignment is heavily load-bearing; Table 4 makes that explicit. However, Table 3 already provides a partial control (VQ-S/AE-S with the same recipe, both much worse than SoftVQ-S), so the reader's requested control is partly present. The residual gap is scale: the S-scale control does not prove that a B-scale AE with DINOv2 alignment would not match SoftVQ-B. Because the headline FID uses SoftVQ-B/BL/L, the attribution of the headline result to the soft codebook specifically is not yet settled. That is a concrete, addressable concern rather than a refutation. Other issues (the KL loss in Eq. (4)/Appendix B.1 is not the stated KL, and 'state-of-the-art' is stronger than the FID 1.78 vs REPA 1.42 comparison supports) are real but secondary: they affect presentation and reproducibility framing, not the core efficiency result. Since the reader already conditioned acceptance, my concern reinforces that condition and does not change the verdict.","tokens_in":28588,"tokens_out":17060,"duration_ms":154594,"concrete_test":"Train VQ-B and AE-B (~173M parameters, matching SoftVQ-B) with the exact SoftVQ-B recipe, including DINOv2 encoder initialization and the Eq. (6) alignment loss, using 64 latent tokens on ImageNet 256. Then train SiT-XL on each tokenizer and measure gFID without CFG; compare with Table 4's SoftVQ-B value of 10.13 and with the CFG FID 1.78 pipeline. If either baseline matches SoftVQ-B within FID noise, the softness is not the load-bearing ingredient; if they remain near 17-25 gFID, the attribution is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central generation claim (FID 1.78/2.21 at 32/64 tokens) is a system-level result that includes DINOv2 encoder initialization and the Eq. (6) alignment loss. Table 4 shows this matters enormously: for SoftVQ-B with 64 tokens, adding DINOv2 init+alignment moves gFID (SiT-L, no CFG) from 20.33 to 10.13, and alignment alone moves it to 10.96. This makes the external teacher load-bearing, not a minor ingredient. Table 3 does train VQ-S and AE-S 'with the same training recipe of SoftVQ' and shows they are far worse than SoftVQ-S, which supports the softness hypothesis at the 46M S scale. But the headline numbers are reported for SoftVQ-B, BL, and L (173M-608M) with DINOv2 initialization, and no VQ-B or AE-B control with the same initialization and alignment is reported. The claim that the soft categorical codebook, rather than the pretrained-feature alignment and initialization, is what enables high-compression generation is therefore not yet established at the model scale where the central claim is set. This is an attribution gap, not an internal inconsistency; it is testable with one experiment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SoftVQ-VAE proposes replacing VQ-VAE's hard argmin assignment with a soft categorical posterior over a learnable codebook, making the tokenizer fully differentiable and allowing each 1D latent token to aggregate multiple codewords. On ImageNet 256x256 and 512x512, the authors train ViT-based tokenizers with 32 or 64 latent tokens, align the latent space with DINOv2 (or other) features via Eq. (6), and then train DiT, SiT, and MAR generative models on these latents. They report reconstruction rFID around 0.6-0.9 at 32-64 tokens, generation gFID as low as 1.78 (256x256, SiT-XL, CFG) and 2.21 (512x512), and large inference throughput gains (up to 55x), claiming state-of-the-art few-token generation while reducing training iterations. The paper includes extensive system-level tables, tokenizer ablations (codebook size, temperature, PQ/RQ/GMM variants), linear-probing analyses, and visualizations.","tokens_in":28788,"tokens_out":7257,"duration_ms":64491,"significance":"If the results are reproducible and correctly attributed, this is a practically important contribution: it shows that a soft, continuous tokenizer can compress images to 32-64 tokens without collapsing generation quality, directly attacking the token-count bottleneck of transformer-based generative models. The study is unusually thorough on the empirical side, covering two resolutions, three generative backbones, multiple tokenizer scales, and a wide ablation set including codebook size, softmax temperature, product/residual quantization, and GMM variants. The paper also ships code and model releases, and its linear-probing analysis adds evidence about latent-space quality. However, the central attribution of the headline gains to the soft posterior is not yet clean: the few-token FID results are obtained with a DINOv2-initialized encoder and the Eq. (6) alignment loss, and the factorial design in Table 4 shows that this external teacher is load-bearing (gFID moves from 20.33 to 10.13). The scale-matched VQ/AE controls in Table 3 are at the small (46M) scale only. The theoretical framing also needs correction: the quantity called a KL divergence in Eq.","major_comments":[{"comment":"The term Lkl in Eq. (4), presented as the 'KL divergence term in the ELBO', is not the KL divergence between the posterior and the uniform codebook prior. For a uniform prior over K codewords, KL(q||p) = log K - H(q). The implemented expression H(q) - H(E[q]) is, after batch averaging, approximately the negative of the Jensen gap between the average posterior entropy and the conditional entropy; it is not equal to log K - H(q) and is not even a valid KL divergence as written. The derivation in Appendix B.1 substitutes E[q] into the cross-entropy term without justification, and the first entropy term is not averaged over x. Since the method's core formulation is built on this term, the equation and derivation should be corrected or the term should be explicitly reframed as a diversity regularizer rather than a KL divergence. The empirical results are unlikely to be invalidated because lambda4 is small (0.01), but the description is mathematically incorrect.","section":"Section 3.2, Eq. (4); Appendix B.1"},{"comment":"The headline few-token generation numbers (e.g., FID 1.78 on 256x256 and 2.21 on 512x512) are reported for SoftVQ-B, SoftVQ-BL, and SoftVQ-L with DINOv2 encoder initialization and the Eq. (6) alignment loss, yet the only scale-matched VQ and AE controls are the 46M-parameter SoftVQ-S variants in Table 3. Table 4 shows that DINOv2 initialization plus alignment changes SoftVQ-B 64-token gFID from 20.33 to 10.13, with alignment alone giving 10.96 and initialization alone 17.20. This makes the external teacher load-bearing, and without a VQ-B or AE-B baseline trained with the same DINOv2 initialization and alignment objective, the paper cannot yet attribute the central generation gains to the softness of the posterior. Please add such controlled baselines at the scale of the headline results (or clearly state if the Table 3 controls already include DINOv2 alignment and initialization, in which case that should be stated explicitly in the text).","section":"Section 4.3, Table 3; Section 4.4, Table 4"},{"comment":"The throughput speedup claims (up to 18x for 256x256, 55x for 512x512) are presented as system-level comparisons, but the measurement protocol is not fully specified for baseline rows. The text says throughput is measured 'on a single AMD MI250', yet Table 1 includes throughput values for many baseline systems (LDM-4, U-ViT, DiT-XL/2, SiT-XL/2, REPA) that are typically quoted from their original papers, which used different hardware and software stacks. If those baseline numbers are not re-measured on the same MI250 setup, the speedup ratios are not controlled comparisons. Please clarify in the tables or text which throughput values were measured in this work and which are taken from other sources, and if possible report same-hardware measurements for the main baselines.","section":"Section 4.1, Tables 1-2; efficiency claims"}],"minor_comments":[{"comment":"The loss weights are written as 'lambda1 = 1.0, lambda1 = 0.2'; the second should presumably be lambda2 = 0.2, matching Eq. (7). Please fix this typo in both places.","section":"Section 4.1 and Appendix C.1"},{"comment":"The unqualified phrase 'state-of-the-art generation results' overclaims: in Table 1, SiT-XL/2 + REPA achieves gFID 1.42 with CFG, which is better than the reported SoftVQ best of 1.78. The claim is defensible if restricted to 'with only 32 or 64 tokens' or 'among high-compression tokenizers', and the bullet already includes that qualifier, so please make the abstract and introduction equally precise.","section":"Abstract and Introduction, bullet 2"},{"comment":"The similarity function 'sim' in Eq. (6) is not defined; later text and Table 4 suggest cosine similarity, but the reader should not have to infer this. Also, the notation z_r[n] is garbled in Eq. (5).","section":"Section 3.3, Eq. (6); Section 4.1"},{"comment":"There are several typographical errors: 'Gflops' vs 'GLOPs' (the latter is not a standard unit), 'Similarly to RPEA' should read 'Similarly to REPA' (Appendix C.2), and '1.41 rFID of of' and 'an 1.34 rFID of of' appear in Appendix A.1/Table 5 text.","section":"Throughout"},{"comment":"The sentence 'The encoder initialization and alignment with DINOv2-B achieve superior performance with rFID 0.88 and IS 103.4, compared to using either component alone' is accurate, but it could be clearer that CLIP-B and EVA-02-B achieve even better reconstruction and IS; the current wording may mislead readers into thinking DINOv2 is best on all metrics.","section":"Section 4.4, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically strong on the empirical side and likely to be of interest to the community. The two main conditions for acceptance are (1) correcting the KL-derivation error or relabeling it as a regularizer, and (2) adding scale-matched VQ/AE baselines with the same DINOv2 initialization and alignment loss at the model scales used for the headline FID numbers. A third condition is to clarify the hardware/measurement provenance of the throughput numbers in Tables 1-2. The 'state-of-the-art' phrasing also needs tightening relative to REPA's 1.42 FID. These are all addressable within a revision; I do not see a fatal internal inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a serious systems contribution, not a repackaging of an old trick. The soft categorical posterior is indeed known (the authors admit it), but the combination of 1D learnable latent tokens, soft aggregation, and DINOv2 initialization/alignment actually delivers the goods: 32 and 64 tokens on 256x256 and 512x512 with rFID around 0.6-0.9 and SiT-XL gFIDs of 1.78 at 64 tokens and 2.21 at 512x512. The throughput numbers (18x, 55x) are large and, given the token count reduction, credible rather than surprising. The ablation suite is extensive: codebook size, temperature, latent size, PQ/RQ/GMM variants, and a careful comparison against TiTok. That is real experimental work.\n\nThe main soft spot is exactly what the stress-test note flags. The headline numbers come from SoftVQ-B/BL/L with DINOv2 init and the Eq. (6) alignment loss, but there is no scale-matched VQ-B or AE-B baseline trained with the same init and alignment. Table 4 shows the alignment alone moves SoftVQ-B gFID from 20.33 to 10.96; with DINOv2 init it goes to 10.13. That makes the teacher load-bearing, and until you run the same recipe on a hard VQ baseline, you cannot claim the soft mechanism is what enables high-compression generation at the model sizes where the central claim lives. Table 3 does support the softness hypothesis at the 46M scale (VQ-S and AE-S are much worse, and they use the same recipe), so this is an attribution gap, not a fatal flaw. One experiment would close it.\n\nThere is also a real but likely minor technical error in the KL derivation. Eq. (4) writes Lkl as H(q) - H(E[q]), and Appendix B.1 replaces the cross-entropy with the uniform prior by the entropy of the average posterior. That is not the KL divergence to the uniform prior; it is closer to a negative mutual information or a different regularizer. The training seems to work regardless, but the paper should present the objective accurately or fix the derivation. Also, the \"state-of-the-art\" wording is too strong when REPA gets 1.42 and SoftVQ gets 1.78; the efficiency and training-iteration story is the real win. Minor issues: no variance or multiple seeds for headline FIDs, no commit hash for the released code, and a leftover \"λ1 = 0.2\" typo in Sec 4.1.\n\nOverall: this deserves a serious referee. The experimental scope is large, the code is released, and the attribution question is testable. A conditional acceptance asking for the VQ-B/AE-B control, a corrected KL section, and tempered claims would be the right outcome. I would bring this to reading group and would cite it if I worked on tokenizers or efficient generation.","headline":"Solid systems paper on 1D continuous tokenizers with real efficiency wins; the DINOv2 alignment is doing much of the work behind the headline few-token FIDs, and the KL framing needs a fix.","tokens_in":29462,"tokens_out":1856,"would_cite":true,"duration_ms":17373,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A soft codebook lets 32 tokens match 1024-token image generation","keywords":["soft categorical posterior","continuous tokenizer","image generation","representation alignment","vector quantization","high compression ratio","latent diffusion","ImageNet generation"],"falsifier":"Train the identical SoftVQ-B/64 pipeline on ImageNet with a frozen randomly-initialized or non-semantic alignment target (or with the alignment loss removed and no DINOv2 init), and check whether the final SiT-XL gFID stays near 1.78; a rise above about 5 would confirm that the DINOv2 teacher, not the soft posterior alone, carries the few-token generation quality. A second check: the throughput figures count generative-model FLOPs only, so recompute end-to-end latency including tokenizer decoding; if the 55× advantage shrinks below 10×, the practical speedup claim needs qualification.","tokens_in":28277,"feed_emoji":"⚡","tokens_out":5780,"duration_ms":45932,"temperature":0.7,"pith_summary":"SoftVQ-VAE claims that replacing a vector-quantized tokenizer's hard codeword assignment with a soft categorical posterior — a weighted blend of many codewords per latent token — yields a fully differentiable, continuous tokenizer that compresses 256×256 and 512×512 images to just 32 or 64 one-dimensional tokens. On ImageNet, this lets diffusion (DiT), flow (SiT), and autoregressive (MAR) generators match or beat the quality of systems using 256–1024 tokens, with SiT-XL reaching FID 1.78 at 256×256 and 2.21 at 512×512 while raising inference throughput up to 18× and 55×. The paper argues that the soft posterior removes the broken gradients of VQ, letting the latent space be directly aligned to pre-trained semantic features, and that this semantic richness — not reconstruction quality alone — is what makes few-token generation work. If correct, token count stops being the main cost driver for transformer-based generation, and the same recipe could carry over to video, audio, and multimodal generation.","feed_headline":"Soft tokenizer: 32 tokens match 1024-token image generation","feed_subtitle":"A weighted codebook blend lets diffusion and flow models hit FID 1.78 with up to 55x faster inference.","key_machinery":"The central object is the soft categorical posterior, $q_{\\phi}(z|x) = \\operatorname{Softmax}(-\\|\\hat{z} - C\\|^2/\\tau)$, which maps the encoder output $\\hat{z}$ to a convex combination of codebook vectors $C$. This turns the discrete VQ lookup into soft K-means, makes the whole tokenizer end-to-end differentiable, and — together with the representation-alignment loss of Eq. (6), which replicates each latent token $N/L$ times and matches a pre-trained encoder's patch features through an MLP projector — creates a latent space whose semantics carry the generation task at very few tokens. The KL term is the entropy of the soft posterior minus the entropy of its batch average, regularizing codebook usage without the VQ codebook or commitment losses.","core_discovery":"The central claim is that a VQ-VAE can be turned continuous by replacing the arg-min codeword selection with a softmax over negative squared distances to the codebook (temperature τ=0.07), then taking the latent as the probability-weighted sum of codewords. Because every step is differentiable, the codebook and encoder learn directly from reconstruction, perceptual, adversarial, and KL losses without codebook or commit losses, and — crucially — a cosine-similarity loss can align each latent token to the corresponding patch features of a pre-trained self-supervised ViT (DINOv2), with a projector MLP and replicated tokens to match resolution. The paper shows that with this alignment and DINOv2-initialized encoder, SoftVQ variants hold rFID at 0.61–0.89 with only 32–64 tokens, and trained downstream generators (SiT-XL with 64 tokens) obtain gFID 1.78 on ImageNet 256 and 2.21 on 512, using 2.3× fewer training steps than the vanilla baselines.","pith_inferences":["If the alignment premise transfers across domains, the same soft-posterior recipe could push video and audio tokenizers to similarly tiny token budgets, where the quadratic attention cost of transformers is even more prohibitive.","The paper's Table 4 shows that choosing a different teacher (CLIP or EVA) keeps quality high, suggesting the method may be teacher-agnostic; a testable extension is to see whether a small, task-specific teacher can substitute DINOv2 and reduce tokenizer training cost further.","Since better reconstruction (rFID) does not track better generation (gFID) here, an implicit design rule for tokenizers is to optimize semantic alignment rather than pixel fidelity at high compression — a principle that, if true, could reshape how tokenizers are evaluated.","The CFG gains with SoftVQ are smaller than for the REPA baseline, an observation the paper leaves open; investigating why classifier-free guidance saturates at very low token counts could yield methods that further close the gap."],"forward_implications":["With 32 and 64 tokens, DiT-XL, SiT-XL, and MAR-H reach generation quality comparable to or better than 256–1024-token baselines on ImageNet 256×256 and 512×512, at a fraction of the GFLOPs.","Inference throughput for generating a 512×512 image rises up to 55× with 64 tokens, and training iterations drop by roughly 2.3× for comparable FID.","The soft posterior removes the need for codebook and commitment losses, making the tokenizer a drop-in continuous alternative compatible with existing VQ techniques such as product and residual quantization.","Reconstruction quality stays high at high compression (rFID 0.61–0.89 at 256×256), unlike hard VQ and plain AE baselines whose rFID degrades sharply when tokens drop below 128.","The tokenizer's linear-probing accuracy transfers to the trained generative model, indicating that the latent space is semantically organized for downstream generation."],"supporting_citations":[{"why":"Supplies the base VQ-VAE formulation and categorical posterior that SoftVQ modifies into a soft, differentiable posterior.","marker":"[104]"},{"why":"Supplies the reconstruction, perceptual, and adversarial training recipe and the codebook/commit losses that SoftVQ removes.","marker":"[23]"},{"why":"Provides the 1D-token transformer autoencoder architecture and the 128/32-token compression baseline this work builds on and compares against.","marker":"[115]"},{"why":"Supplies the representation-alignment idea (aligning latents to pretrained features) that SoftVQ transfers from generative-model layers to the tokenizer latent space; also the key CFG and FID comparison baseline.","marker":"[116]"},{"why":"Provides the pre-trained self-supervised ViT whose patch features serve as the alignment target and whose weights initialize the encoder.","marker":"[79]"},{"why":"One of the three downstream transformer backbones; SoftVQ-tokenized DiT-XL reports gFID 2.93 at 64 tokens with CFG.","marker":"[80]"},{"why":"The flow-based interpolant backbone that achieves the headline FID 1.78 (256×256) and 2.21 (512×512) results with 64 tokens.","marker":"[72]"},{"why":"The autoregressive-with-diffusion backbone (MAR-H) showing 32/64-token generation against its 256-token KL baseline.","marker":"[60]"},{"why":"The concurrent high-compression AE-based tokenizer (DC-AE) compared directly in reconstruction and generation tables at 64 and 256 tokens.","marker":"[11]"}],"fun_headline_variants":["SoftVQ: 64 tokens, 55x faster image generation","SoftVQ: 64 tokens, FID 1.78, 55x faster","Soft tokenizer: 2.3x fewer steps, 55x faster inference","All-differentiable VQ: 32 tokens match 1024 tokens","Weighted codebook: 18x faster on 256x256"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline generation numbers rest on the assumption that aligning the tokenizer's latents to a pre-trained self-supervised vision teacher (DINOv2), and initializing the encoder from it, is a sufficient and transferable inductive bias for downstream generative modeling; Table 4 shows the same 64-token SoftVQ-B drops from gFID 10.13 to 17.20 when the alignment is removed.","fun_headline_variants_meta":{"raw":{"variants":["SoftVQ: 64 tokens, 55x faster image generation","SoftVQ: 64 tokens, FID 1.78, 55x faster","Soft tokenizer: 2.3x fewer steps, 55x faster inference","All-differentiable VQ: 32 tokens match 1024 tokens","Weighted codebook: 18x faster on 256x256"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000963,"raw_usage":{"total_tokens":4134,"prompt_tokens":1010,"completion_tokens":3124,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":3019}},"tokens_in":626,"tokens_out":3124,"duration_ms":21019,"temperature":1.0,"reasoning_tokens":3019,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:27:00.549725+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical SoftVQ-B/64 pipeline on ImageNet with a frozen randomly-initialized or non-semantic alignment target (or with the alignment loss removed and no DINOv2 init), and check whether the final SiT-XL gFID stays near 1.78; a rise above about 5 would confirm that the DINOv2 teacher, not the soft posterior alone, carries the few-token generation quality. A second check: the throughput figures count generative-model FLOPs only, so recompute end-to-end latency including tokenizer decoding; if the 55× advantage shrinks below 10×, the practical speedup claim needs qualification.","supporting_citations":[{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the base VQ-VAE formulation and categorical posterior that SoftVQ modifies into a soft, differentiable posterior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained self-supervised ViT whose patch features serve as the alignment target and whose weights initialize the encoder."},{"cited_title":"Scalable diffusion models with transformers, 2023","cited_arxiv_id":null,"evidence_quote":"One of the three downstream transformer backbones; SoftVQ-tokenized DiT-XL reports gFID 2.93 at 64 tokens with CFG."}],"review_version":1}