{"id":"fd7fa8cd-6261-4ef3-978b-8d0a80c41400","arxiv_id":"2412.10208","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ResGen predicts cumulative vector embeddings of masked RVQ tokens, decoupling generative sampling cost from token depth and improving FID and TTS metrics over autoregressive baselines.","lead":"ResGen is a new way to train generative models on compressed image and audio tokens, predicting summed token embeddings so the sampling steps no longer grow with the token hierarchy depth. It reports faster, higher-quality generation than autoregressive baselines on ImageNet and text-to-speech, but the theoretical framing needs tightening and code is not yet public.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.2's variational derivation assumes the training target z decodes back to the masked tokens via greedy RVQ; this is not guaranteed, so the claimed ELBO grounding may not hold.","rationale":"The reader's conditional verdict is appropriate: the paper contributes a useful empirical design and controlled ablations, but the theoretical framing has a real soft spot. The reader identified the Section 3.2 variational simplification as the weakest assumption; I agree with that choice, though the mechanism is slightly different. The issue is not only that terms are 'negligible or cancel'; it is that the fixed-decoder premise may be false. The training target z_i is a partial cumulative sum of codebook embeddings, and greedy RVQ applied to that sum is not guaranteed to recover the masked tokens, because the sum is not the residual input the first quantizer saw during RVQ training. If the decoder does not invert the training target, then -log p_theta(z|x^(t)) is not a variational bound on -log p_theta(x^(0)|x^(t)), and the paper's claim of a principled probabilistic foundation is unsupported. The empirical central claim---cumulative embedding prediction improves efficiency over RQ-Transformer in the controlled Table 1 ablation---still appears valid, and the FID differences versus MAR/VAR/MAGVIT-v2 do not by themselves invalidate the method. The proposed decoder-consistency test would settle whether the theoretical concern lands; if recovery is high, the simplification is benign for training, while if recovery is low, the paper should be revised to present L_mask explicitly as a surrogate latent objective rather than as an ELBO.","tokens_in":20567,"tokens_out":16994,"duration_ms":174396,"concrete_test":"Measure the decoder consistency of the training target on a random sample of validation RVQ tokens: for each position i and each training mask m, compute z_i = sum_j e(x_{i,j};j) * (1-m_{i,j}), then run the same greedy RVQ used in Algorithm 2 and check whether the quantized token sequence equals the masked token subsequence of x_i. Report the per-depth recovery rate, e.g., for depth-16 RVQ on ImageNet. If the recovery rate is materially below 100% (or below the reconstruction rate of the full encoder output h_{i,0}), the 'p(x0|z)=RVQ quantization' premise in Section 3.2 is false, and the variational derivation must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is the Section 3.2 claim that training loss (1) is the variational objective for masked RVQ generation. The derivation drops the term -log p(x^(0)|z,x^(t)) and the q-denominator, saying p is 'RVQ quantization' and q is 'RVQ dequantization'. Even if both are treated as fixed, this only legitimizes optimizing -log p_theta(z|x^(t)) if the deterministic decoder actually maps the training target z_i = sum_j e(x_{i,j};j) * (1-m_{i,j}) back to the masked tokens x_{i,j}. That is not guaranteed: greedy RVQ on a partial cumulative sum is not the same as the original sequential residual quantization of the encoder output, because the sum excludes unmasked depths and includes quantization error. If RVQ(z_i) disagrees with the target tokens, the dropped reconstruction term is not a constant and the objective is not an ELBO for p_theta(x^(0)|x^(t)); it is a surrogate latent regression. The empirical ablations in Table 1 support the design, but the theoretical grounding stated in Section 3.2 is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ResGen, a masked generative model over Residual Vector Quantization (RVQ) tokens. Instead of predicting individual token indices, the model predicts the sum of the embeddings of the masked tokens at each sequence position, modeled with a mixture of Gaussians. Masking is applied from the highest RVQ depth downward, and sampling iteratively unmasks tokens using confidence scores. The paper claims that this design decouples the number of inference steps from both RVQ depth and sequence length. It reports a controlled ablation on ImageNet 256x256 with matched RVQ tokens, parameter counts, and training budget against RQ-Transformer and a MaskGIT variant, alongside scaling ablations and benchmark comparisons for image generation and zero-shot text-to-speech synthesis.","tokens_in":20811,"tokens_out":12628,"duration_ms":132658,"significance":"If the empirical results hold, the cumulative-embedding prediction idea is a simple and practically useful contribution: it avoids autoregressive decoding over RVQ depth and shortens the sampling trajectory. The controlled ablation in Table 1 with matched tokens, parameters, and training budget is a strong piece of evidence that the design, rather than model scale or tokenizer choice, drives the gains. The paper also provides useful ablations of sampling hyperparameters and an honest limitations section. However, the theoretical framing in Section 3.2 currently overclaims, and the selective reporting in the AR-ResGen ablation needs to be addressed.","major_comments":[{"comment":"The variational derivation drops the reconstruction term -log p(x^(0)|z,x^(t)) and the normalization term -log q(z|x^(0),x^(t)) by asserting that p is RVQ quantization and q is RVQ dequantization. This is only valid if the deterministic mapping z_i = sum_j e(x_{i,j};j)(1-m_{i,j}) is inverted exactly by the RVQ quantization used in Algorithm 2. For partial masks, z_i contains only the masked (higher-depth) embeddings, so greedy residual quantization of z_i cannot recover unmasked lower-depth tokens from the sum alone; even for fully masked sequences, residual quantization is not guaranteed to be the inverse of embedding summation. If RVQ(z_i) disagrees with the training target x_{i,·}, the dropped term is not a constant and Eq. (2) is not an ELBO for p_theta(x^(0)|x^(t)). Please either prove the inverse property, verify it empirically on the actual RVQ tokenizer, and specify how unmasked depths are handled, or explicitly present Eq. (1) as a surrogate latent-regression objective rather than as a variational bound.","section":"Section 3.2"},{"comment":"The reduction of the ELBO to L_simple(x^(0);theta) = -log p_theta(x^(0)|x^(t)) is not a derivation: the ELBO contains a sum over diffusion steps and a reconstruction term, and replacing them with a single cross-entropy at an unspecified time t under 'equal emphasis' is a heuristic simplification. The connection from the ELBO to Eq. (1) therefore involves two unstated approximations: ELBO to L_simple, and L_simple to Lmask. Please state explicitly which parts are exact bounds and which are design choices, and give the distribution of the training time t used in Eq. (2).","section":"Section 3.2, Eq. (2)"},{"comment":"The right-hand side of Table 1 shows that without classifier-free guidance, AR-ResGen is substantially worse than RQ-Transformer at every step budget (e.g., FID 23.48 versus 15.71 at 8 steps), while with CFG it is comparable or better. The text's claim that the depth-prediction strategy 'delivers substantial improvements in both the generation speed and the sample quality' is therefore only supported in the CFG evaluation. Please report and discuss the w/o-CFG numbers in the main text and scope the 'comparable quality' claim accordingly.","section":"Table 1 (right panel)"}],"minor_comments":[{"comment":"The formulas for the marginal q(x^(t)|x^(0)) and the conditional q(x^(t)|x^(t+1),x^(0)) are hard to parse because the binomial coefficients appear to contain the random variables k_i^(tau) on both sides; please rewrite them with standard multivariate hypergeometric notation and explicitly define n^(tau).","section":"Section 3.2"},{"comment":"The text says the counts k_i are 'sampled without replacement from a multinomial distribution'; this should be a multivariate hypergeometric distribution, and the sampling procedure in Algorithm 2 should be specified more precisely.","section":"Section 3.1 and Algorithm 1"},{"comment":"The confidence score formula log p(x_{i,j}|z_i) proportional to a Gaussian log-density omits the proportionality constant and is a heuristic posterior over tokens; please state this explicitly so readers do not confuse it with the model's actual predictive distribution.","section":"Appendix B.1"},{"comment":"In the right panel, 'FID of 5.33s' should read 'FID of 5.33' (the unit 's' appears to be a typo).","section":"Table 1"},{"comment":"The benchmark comparisons rely largely on numbers reported in the original papers, which may use different training data, tokenizers, and evaluation protocols; the paper should state more prominently that the controlled ablation in Table 1 is the only directly comparable evidence for the claims of superiority.","section":"Tables 2 and 3"},{"comment":"The sentence 'deeper RVQ depth enhances leads to improved reconstruction quality' contains a grammatical error and should be reworded.","section":"Section 5.2.3"}],"recommendation":"major_revision","confidential_remarks":"The core empirical contribution is solid and the controlled ablation in Table 1 is convincing evidence for the method's effectiveness. My recommendation is driven by the overclaimed variational derivation in Section 3.2 and the selective reporting in the AR-ResGen ablation. I would not reject: the method likely works, and the derivation can be reframed as a heuristic or motivated objective rather than a rigorous ELBO."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ResGen's core idea is worth taking seriously: instead of predicting RVQ tokens depth-by-depth, predict the cumulative vector embedding of the masked tokens, then quantize that embedding back into tokens during sampling. That decouples inference steps from RVQ depth, and the controlled ablation in Table 1 backs it up—same RVQ tokens, comparable model sizes, and the cumulative-embedding variant beats both direct token prediction and the MaskGIT-style depth-wise baseline. The AR-ResGen ablation is a nice touch: it isolates the depth-prediction gain from the masked-generation framework. I buy the central empirical claim.\n\nThe soft spot is Section 3.2. The variational derivation drops the reconstruction term by assuming p(x^(0)|z,x^(t)) is RVQ quantization and q(z|x^(0),x^(t)) is RVQ dequantization. That only works if applying greedy RVQ to the training target z_i = sum of masked-codebook embeddings recovers the original masked tokens. It generally won't: z is a partial sum with gaps at unmasked depths, and it already contains quantization error from the forward encoder. So the objective is better described as a surrogate latent regression, not an ELBO for the token sequence. The paper should either prove the inverse condition or soften the 'theoretical foundation' claim. This doesn't sink the method—empirical results stand on their own—but it's an overreach.\n\nAlso, no code is released, and the headline FID (1.93) trails MAGVIT-v2 (1.78) and MaskBit (1.62), which the authors acknowledge. The TTS numbers are competitive but not uniformly SOTA. These are minor. The comparison against faster MAR variants is handled more carefully in Appendix C.2 than in the main text, which is fine.\n\nWho should read this: anyone building token-based generative models, especially with RVQ or other hierarchical quantizers. The ablation alone is worth a look. It deserves a serious referee; I'd accept it with a request to fix or caveat the theoretical section.","headline":"ResGen's core idea—predicting cumulative RVQ embeddings to decouple sampling from depth—is genuinely useful and backed by controlled ablations, but the ELBO derivation in Section 3.2 overreaches.","tokens_in":21342,"tokens_out":4372,"would_cite":true,"duration_ms":41309,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ResGen predicts the cumulative embedding of masked RVQ tokens rather than individual tokens, so inference steps no longer scale with quantization depth, reaching FID 1.93 on ImageNet and beating autoregressive TTS baselines.","keywords":["residual vector quantization","masked token prediction","discrete diffusion","cumulative embedding prediction","generative modeling","image generation","text-to-speech synthesis","mixture of Gaussians"],"falsifier":"Compute the dropped ELBO terms on held-out latents: estimate $-\\log p(x^{(0)}|z,x^{(t)})$ for the trained RVQ quantizer and the KL divergence between the quantization posterior and the dequantization distribution, and check whether their sum is small relative to $-\\log p_\\theta(z|x^{(t)})$; if it is large, the Section 3.2 simplification fails. A cheaper decisive test is to train the same architecture with the cumulative-embedding loss but without the hypergeometric diffusion masking, using a fixed uniform mask ratio instead; identical FID would show the probabilistic framing is not what drives the result.","tokens_in":20346,"feed_emoji":"⚡","tokens_out":14077,"duration_ms":119640,"temperature":0.7,"pith_summary":"ResGen is a generative model for residual vector quantization (RVQ) tokens that sets out to make sampling cost independent of both sequence length and quantization depth. RVQ packs fidelity into a token grid by quantizing residuals at several depths, but autoregressive models pay a sampling step for every depth at every position. ResGen instead trains the model to predict the cumulative vector embedding — the sum of the masked embeddings across depths — and then quantizes that vector back into tokens. On ImageNet 256x256 it reports FID 1.93 with classifier-free guidance in 63 steps, and in zero-shot text-to-speech it beats autoregressive baselines on word and character error rates with far fewer inference steps. The point is that adding RVQ depth for fidelity no longer has to cost sampling speed.","feed_headline":"Token-sum prediction breaks the RVQ depth–speed trade-off","feed_subtitle":"Both image and speech models keep sampling steps fixed as RVQ depth grows, beating autoregressive baselines.","key_machinery":"The central object is the cumulative embedding prediction: instead of classifying each masked RVQ token separately, the model outputs a mixture of Gaussians over the sum of the masked depth embeddings at each position, $z_i = \\sum_j e(x_{i,j}; j) \\odot (1 - m_{i,j})$, with a modified training objective that forces every mixture component to be used. At sampling time the predicted vector is quantized back into tokens with the RVQ codebook, and the most confident tokens are unmasked first, following a schedule in which deeper (finer) tokens are masked first and filled in last. This machinery carries the argument by collapsing the depth axis into a single regression target, so the number of generation steps depends only on the masking schedule and not on sequence length or RVQ depth.","core_discovery":"The paper's central claim is that predicting the summed vector embedding of the masked tokens at a position, $z_i = \\sum_j e(x_{i,j}; j) \\odot (1 - m_{i,j})$, is both sufficient and better than predicting individual tokens, because the RVQ decoder consumes vector embeddings, not token indices. This design removes the dependence of inference steps on RVQ depth: one forward pass predicts all depths at once, and re-quantizing the predicted embedding into tokens adds negligible overhead. The authors cast masking and unmasking as a discrete diffusion process with a multivariate hypergeometric forward step, deriving the training objective $-\\log p_\\theta(z|x^{(t)})$ as the surviving term of a variational bound after assuming the reconstruction and entropy terms cancel. Empirically the claim is supported by a controlled ablation in which the cumulative-embedding variant beats direct token prediction at equal capacity (FID 2.43 versus 2.91 with CFG on ImageNet under the same RVQ tokens), by an autoregressive-depth ablation that isolates the depth-prediction gain (FID 5.22 in 3.05 seconds versus 5.50 in 5.35 seconds for RQ-Transformer), and by scaling results showing that deeper RVQ improves generation quality at the same step count.","pith_inferences":["The 'predict the cumulative embedding, then re-quantize' loop should transfer to any hierarchical codec whose decoder consumes additive vectors — multi-group bit tokens in LFQ-style models, audio codec language models, or video tokenizers — none of which the paper tests.","Because step count and representation depth are now independent, an adaptive-compute regime becomes possible that the paper does not pursue: draft with few steps and refine with more, choosing the budget per application.","A non-variational version of the same loss — cumulative-embedding regression without the diffusion-masking story — is a testable extension that would reveal whether the probabilistic framing is load-bearing; the paper itself notes there is no theory for why so few steps suffice.","Combining progressive unmasking with key-value caching, which the paper lists as future work, would make per-step cost shrink as generation proceeds — an efficiency gain orthogonal to the depth decoupling."],"forward_implications":["RVQ depth can be scaled for higher fidelity without increasing sampling steps: 16-depth beats 8-depth at the same step count, and the audio model runs 25 steps with 72 quantization levels.","In the controlled comparison under identical RVQ tokens, ResGen reaches FID 2.43 with CFG in 63 steps where RQ-Transformer needs 1024 steps for FID 5.50; the depth-only ablation is 1.8x faster at better FID.","For zero-shot text-to-speech, ResGen posts the lowest word and character error rates on the cross-sentence task among the compared models (WER 1.70, CER 0.46) with 25 inference steps.","Standard diffusion tooling transfers to the framework: classifier-free guidance, temperature, and top-p each tune the quality-diversity balance of the unmasking process, as the paper demonstrates in its sampling ablations."],"supporting_citations":[{"why":"Introduces RVQ and RQ-Transformer, the autoregressive length-times-depth token model whose sampling cost ResGen is designed to remove, and supplies the main image baseline.","marker":"(Lee et al., 2022)"},{"why":"MaskGIT: provides the masked token modeling paradigm, the masking schedule, and the confidence-based iterative unmasking procedure that ResGen adapts to multi-depth RVQ tokens.","marker":"(Chang et al., 2022)"},{"why":"CLaM-TTS: contributes the multi-token vector prediction idea, the mixture-of-Gaussians training objective with low-rank projection, and the MelVAE module used in the audio experiments.","marker":"(Kim et al., 2024)"},{"why":"Establishes VQ-VAE, the vector-quantized representation framework that RVQ extends by quantizing residuals.","marker":"(Van Den Oord et al., 2017)"},{"why":"GIVT: shows mixture-of-Gaussians vector prediction can replace softmax token prediction in masked generation, the direct precursor of the cumulative-embedding target.","marker":"(Tschannen et al., 2024)"},{"why":"VALL-E: the autoregressive RVQ text-to-speech baseline and the source of the WER/CER/SIM evaluation metrics used for the audio task.","marker":"(Wang et al., 2023)"},{"why":"VQ-Diffusion: formulates masked generation on quantized tokens as a discrete diffusion process, the framing ResGen extends with multivariate hypergeometric masking.","marker":"(Gu et al., 2022)"},{"why":"MAGVIT-v2: a recent masked generative model with Lookup-Free Quantization whose lower FID (1.78) is the comparison point for ResGen's efficiency argument.","marker":"(Yu et al., 2024)"}],"fun_headline_variants":["Token-sum prediction keeps RVQ sampling steps fixed","Sum embeddings, not tokens: RVQ depth becomes free","Deeper RVQ no longer means slower sampling","ResGen predicts collective tokens to bypass RVQ depth cost","One set of steps for all RVQ depths: ResGen"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the terms dropped from the evidence lower bound in Section 3.2 — the reconstruction likelihood of the original tokens and the entropy of the quantization-versus-dequantization distributions — are negligible or cancel; if they do not, the training loss is not the claimed variational bound, even though it may still work empirically.","fun_headline_variants_meta":{"raw":{"variants":["Token-sum prediction keeps RVQ sampling steps fixed","Sum embeddings, not tokens: RVQ depth becomes free","Deeper RVQ no longer means slower sampling","ResGen predicts collective tokens to bypass RVQ depth cost","One set of steps for all RVQ depths: ResGen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2062,"prompt_tokens":979,"completion_tokens":1083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1019}},"tokens_in":595,"tokens_out":1083,"duration_ms":11549,"temperature":1.0,"reasoning_tokens":1019,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:12:59.129858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the dropped ELBO terms on held-out latents: estimate $-\\log p(x^{(0)}|z,x^{(t)})$ for the trained RVQ quantizer and the KL divergence between the quantization posterior and the dequantization distribution, and check whether their sum is small relative to $-\\log p_\\theta(z|x^{(t)})$; if it is large, the Section 3.2 simplification fails. A cheaper decisive test is to train the same architecture with the cumulative-embedding loss but without the hypergeometric diffusion masking, using a fixed uniform mask ratio instead; identical FID would show the probabilistic framing is not what drives the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLaM-TTS: contributes the multi-token vector prediction idea, the mixture-of-Gaussians training objective with low-rank projection, and the MelVAE module used in the audio experiments."},{"cited_title":"These log probabilities are derived from the squared distance between token embeddings and the sampled latent zi at each position i","cited_arxiv_id":null,"evidence_quote":"GIVT: shows mixture-of-Gaussians vector prediction can replace softmax token prediction in masked generation, the direct precursor of the cumulative-embedding target."}],"review_version":1}