{"id":"408946ba-4253-4f44-b4f9-5dd8564fb0ea","arxiv_id":"2501.06218","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Token-based language-style vision models (VAR, LlamaGen) tolerate quantization better than diffusion models, and a custom TopKLD distillation loss pushes their low-bit scaling roughly one precision level higher.","lead":"This paper compares how two families of image generation models, diffusion-style and token-based language-style, degrade when compressed to low bit precision, and finds the token-based family loses much less quality. This comparison matters because low-bit compression is the main lever for deploying large generative models on limited hardware, so knowing which family compresses better changes real deployment choices.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claims depend on an unreported bivariate power-law fit; without it, '3-bit beats 4-bit' and 'W4A8 surpasses W8A8' are not quantitatively established.","rationale":"I agree with the reader's weakest_assumption: the central claims are only meaningful through a fitted bivariate power law that is never shown. The paper says FID 'followed a distinct bivariate power function' with 'nearly parallel scaling trends' (Section 3, first paragraph), but reporting neither the fitted function nor its quality makes the headline comparisons unfalsifiable. The Appendix C.3 tables are the only numeric evidence, and they are actually consistent with the claims when interpreted at equal total bits: for example, W3A16 at d30 (2B parameters, ~6G weight bits) has FID 2.25, which is better than an interpolated W4A16 model at ~6G weight bits, and W4A8 at d30 (~64G compute-bit units) has FID 2.15 versus W8A8 at d24 (~64G units) at 2.18. So the issue is not that the tables contradict the claims; it is that the interpolation needed to make them comparable is provided only as unshown curve fits. This is a genuine correctness risk because a different fitted model (e.g., non-parallel slopes, different curvature) could reverse the crossovers. I would keep the reader's CONDITIONAL verdict: the paper has useful empirical content, including the LlamaGen/MAR comparison in Appendix C.2 that supports the discrete-space mechanism, and TopKLD consistently improves over Forward/Reverse KLD in the low-bit rows of Tables 2, 4, and 5. But the central scaling-law claims require the authors to publish the fits and data; until then, the conclusions are not verifiable. The embedded 'rebuttal' appendix is structurally unusual but not itself an argumentative flaw; it contains the only numeric tables and should be promoted to the main text.","tokens_in":18412,"tokens_out":9737,"duration_ms":87890,"concrete_test":"Ask the authors to release the complete FID table (model size x precision x method) and the fitted parameters of the Section 3 power law, then recompute the equal-total-bits comparisons: for each precision pair (W3A16 vs W4A16, W4A8 vs W8A8), use the fitted FID(N, b) to find the model-size ratio at which the lower-bit setting matches or beats the higher-bit setting at the same total bits, and report residual plots and confidence intervals. If the lower-bit fitted curve never reaches the higher-bit curve within a reasonable extrapolation (say, up to 10x the largest tested size), the abstract's 'outperform/surpass' claims should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative object is the 'bit-level scaling law': a bivariate power function of model size and bit precision with 'nearly parallel' trends, asserted in the first paragraph of Section 3. None of the fitted parameters, residuals, confidence intervals, or per-configuration FID values behind Figures 1, 5, 6, and 7 are reported, so the comparisons that carry the headline (language-style is superior to diffusion-style; TopKLD 'elevates the bit-level scaling laws by one level') cannot be checked. The Appendix C.3 tables do give FID for VAR at four sizes, but pointwise comparisons are not decisive: W3A16 is worse than W4A16 at every listed size, and W4A8 is worse than W8A8 at every listed size, yet the claims may still hold if 'better' is evaluated at equal total bits, e.g., a 3-bit 2B model versus a 4-bit 1B model, using interpolated scaling curves. Because the fitted curves are absent, the crossover claims are unfalsifiable from the submitted evidence. A secondary load-bearing assumption is the Gaussian-SNR error simulation in Section 3.2: real low-bit quantization error is structured and input-dependent, so the tolerance experiments support the discrete-space mechanism only if Gaussian noise at matched SNR is a faithful proxy, which is not argued.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper empirically compares how quantization affects the scaling behavior of two families of vision generative models: diffusion-style DiT (458M–7B) and language-style VAR (310M–2B), under weight-only and weight-activation PTQ/QAT settings. It claims that language-style models have better bit-level scaling laws because their discrete token/codebook representation space tolerates quantization noise better than the continuous latent space of DiT, and it proposes a distillation objective, TopKLD, that splits the teacher distribution into top-K mode-seeking and complement mode-covering terms. The authors report that TopKLD improves low-bit scaling, making 3-bit weight-only and W4A8 weight-activation models competitive with 4-bit and W8A8 counterparts, and that the benefit transfers to floating-point quantization. The appendix provides pointwise FID tables for VAR, comparisons with SmoothQuant/GPTQ/GPTVQ/OmniQuant, additional model validation with MAR and LlamaGen, and ablations of the K hyperparameter.","tokens_in":18590,"tokens_out":7337,"duration_ms":140424,"significance":"If the central claims are correct, the paper would provide a practically relevant design guideline: under bit budgets, language-style discrete-token generators are more robust to aggressive quantization, and TopKLD distillation is a simple recipe to shift their bit-level scaling curves. The study is broad in scope—multiple model families, sizes, bit-widths, PTQ/QAT, integer/FP, and additional validation on MAR/LlamaGen—and reports full pointwise FID tables in the appendix. I do not see a circularity problem: TopKLD is evaluated against external baselines and ablations. However, the quantitative evidence for the headline scaling-law conclusion is incomplete: the asserted bivariate power-law fits and their uncertainties are never reported, and the mechanism analysis rests on a Gaussian-noise proxy that is not validated against real quantization error. These issues are fixable with additional analysis and reporting.","major_comments":[{"comment":"The paper states that FID follows \"a distinct bivariate power function\" of parameter count and bit precision with \"nearly parallel\" trends, but no fitted equation, coefficients, R², residuals, or confidence intervals are given anywhere in the main text or appendix. As a result, the headline comparisons—language-style models having superior bit-level scaling laws, and TopKLD \"elevating the scaling laws by one level\"—cannot be checked. Appendix C.3 Tables 2 and 3 show W3A16 FID higher than W4A16, and W4A8 higher than W8A8, at every listed model size; if the authors' \"3-bit outperforms 4-bit\" and \"W4A8 surpasses W8A8\" statements are meant at equal total bits through interpolated curves, those curves and their uncertainty intervals must be reported. Without them, the central scaling-law claims are not quantitatively grounded.","section":"Section 3, first paragraph; Sections 3.1 and 3.3"},{"comment":"The tolerance experiments substitute Gaussian noise at controlled SNR for quantization error. Low-bit quantization error is structured, layer-dependent, and input-dependent (as the activation-distribution analysis in the same section shows), so matched-SNR Gaussian noise is not obviously a faithful surrogate. The reported correlations (0.99 vs. 0.86) lack sample sizes and confidence intervals. The mechanistic conclusion that the discrete representation space is the primary reason for superior bit-level scaling therefore needs either actual quantization-error injection experiments or an explicit empirical validation that Gaussian noise matches the effect of quantization error on the reconstruction process.","section":"Section 3.2, Figure 3"},{"comment":"All FID values are single runs with no seeds, sample counts, or error bars. Many differences used to support conclusions are small (e.g., W3A16 TopKLD 3.85 vs. Reverse-KLD 4.02 at d16; W8A8 TopKLD 2.75 vs. SmoothQ 3.81 at d16). Without repeated seeds and uncertainty quantification, pointwise comparisons and fitted scaling exponents cannot be distinguished from sampling noise. Please report seeds and, for any fitted power-law parameters, bootstrap or confidence intervals.","section":"Appendix C.3, Tables 2–5; Figures 1, 5–7"}],"minor_comments":[{"comment":"Equations (4)–(6) contain typographical corruption: the quantizer formula is missing the division by the step size and the zero-point definition is self-referential; please rewrite them cleanly.","section":"Equations (4)–(6)"},{"comment":"The sentence defining total model bits and compute bits is incomplete: \"MT ∝8\" and \"CT ∝82\" should be written as MT ∝ 8 × N and CT ∝ 8^2 × N with N the parameter count, and the example text should be completed.","section":"Section 2.2"},{"comment":"The column headers d16, d20, d24, d30 are not defined; clarify that they denote VAR model sizes and map them to parameter counts.","section":"Tables 2–5 in Appendix C.3"},{"comment":"The caption \"A lower SNR indicates a higher noise component\" is insufficient; define the x-axis and state whether the plotted \"loss\" is FID or another metric, and for which model size the data are shown.","section":"Figure 3 caption"},{"comment":"Section 3.3 refers to \"W4A6\" when the surrounding experiments concern W4A8; correct the typo.","section":"Section 3.3"},{"comment":"Equation (7) should define the decomposition P = [Ms, Mc], specify that Ms/Mc are the top-K and complement token sets, and explain how the chosen K relates to the inference-time top-k of VAR; Table 4 in the appendix should state whether K is selected on a validation split or by matching the teacher's decoding setting.","section":"Equation (7) and Table 4"},{"comment":"Minor language and formatting issues: inconsistent \"DiT\"/\"DIT\" and \"V AR\"/\"VAR\" spacing, \"wight W\" in Eq. (13), \"Institue\" in the affiliation, and the use of \"loss\" versus \"FID\" in Figure 3.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a well-motivated empirical study with a potentially useful method, and the appendix contains substantial data. My main concern is that the central scaling-law claim is asserted rather than demonstrated with reported fits and uncertainty. The deficiencies are addressable in revision: supply the fitted bivariate power laws, R², confidence intervals, seed variance, and validate the noise proxy. The pointwise tables in Appendix C.3 should be reconciled with the \"3-bit versus 4-bit\" claims. I do not see grounds for rejection, but I would not accept the current version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about, but read the appendix before trusting the abstract.\n\nWhat's genuinely new: a head-to-head comparison of bit-level scaling behavior between diffusion-style (DiT) and discrete-token language-style (VAR) vision generators, across a wide range of sizes and precisions, plus the TopKLD distillation objective. The discrete-representation-space explanation is plausible, and the appendix check with LlamaGen (discrete) and MAR (continuous) is the right kind of generalizability test. TopKLD is simple, well-motivated, and the ablation tables in Appendix C.3 show consistent gains across bit settings. This is a useful empirical contribution.\n\nThe problem is that the headline quantitative claims are not backed by anything the reader can check. Section 3 asserts that FID follows a bivariate power function with nearly parallel trends across precisions, but the fit is never reported — no parameters, residuals, confidence intervals. This matters because the pointwise tables in the appendix cut against the abstract. In Table 2, W3A16 is worse than W4A16 at every model size, and in Table 3, W4A8 is worse than W8A8 at every size. The \"3-bit beats 4-bit\" claim can only survive if it means equal total bits — a 3-bit 2B model versus a 4-bit 1B model — and that comparison requires the interpolated curves that aren't in the paper. As submitted, the central scaling-law claim is unfalsifiable.\n\nOther soft spots, in order of severity: FID values are single runs with no seeds, error bars, or code release; the Gaussian-SNR noise injection in Section 3.2 is a coarse proxy for quantization error, since real low-bit error is structured and input-dependent, though the step-like tolerance behavior of VAR is suggestive and the activation-variance analysis adds independent support. Minor issues: a stray \"W4A6\" in the distillation section, a wrong GLIDE citation (Halgren et al. is a docking paper, not the diffusion model), and the odd embedded \"Supplement Materials for Rebuttal\" appendix structure.\n\nMy read: the qualitative finding is probably right — discrete-token models quantize more gracefully and the mechanism story holds up — but the quantitative framing overreaches what is reported. The paper deserves a serious referee, not a desk reject. The revision conditions are concrete: report the fitted scaling-law parameters with residuals and error bars, release code and seeds, and reframe crossover claims explicitly as equal-total-bit comparisons.\n\nWho gets value: anyone working on quantization of generative models or model selection under bit budgets. I'd bring it to a reading group and cite the qualitative finding, but not the scaling-law numbers until they're checkable.\n\nRecommendation: send to peer review, with fitted curves and code as conditions of acceptance.","headline":"The qualitative finding — discrete-token vision models quantize more gracefully — is credible and useful, but the headline scaling-law curves are never reported, so the crossover claims can't be checked.","tokens_in":19194,"tokens_out":5213,"would_cite":true,"duration_ms":38899,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantized language-style image models beat diffusion models at equal bit budgets.","keywords":["bit-level scaling laws","quantization","vision generative models","language-style models","diffusion models","knowledge distillation","discrete representation space","TopKLD"],"falsifier":"Fit the published FID values for V AR at W16A16, W8A16, W4A16, and W3A16 in log-log coordinates and test whether the slopes are equal; if the W3A16 line is not parallel to the others or its confidence interval overlaps W4A16, the claim that TopKLD elevates scaling by one level would be refuted without needing new experiments. Alternatively, repeat the SNR simulation replacing Gaussian noise with correlated noise matched to actual 3-bit quantization error; if the step-like tolerance disappears, the discrete-space explanation would be an artifact of the noise model.","tokens_in":18123,"feed_emoji":"🧩","tokens_out":5729,"duration_ms":50376,"temperature":0.7,"pith_summary":"The paper sets out to show that when vision generative models are quantized, the choice of generative paradigm matters as much as the bit width: language-style models that emit discrete tokens from a codebook lose less quality per bit than diffusion-style models that operate on continuous latents. The authors measure Fréchet Inception Distance across model families from roughly 300M to 7B parameters and bit widths from 3 to 16, and find that discrete-token models keep a consistent quality advantage under the same total model bits and compute bits. They trace the advantage to the codebook's discrete reconstruction space, which absorbs small quantization errors instead of letting them accumulate over generation steps. To push the advantage further, they introduce TopKLD, a distillation loss that fits the top-K token probabilities with a mode-seeking term and the tail with a mode-covering term; this restores low-bit scaling so that W3A16 behaves like W4A16 and W4A8 behaves like W8A8 in both integer and floating-point quantization.","feed_headline":"Quantized token models beat diffusion at same bit budget","feed_subtitle":"Paper shows codebook-based generation absorbs quantization noise, and TopKLD distillation lifts 3-bit quality to 4-bit levels.","key_machinery":"The central objects are the discrete codebook representation space of language-style models—images are tokenized into indices in $\\{1,\\ldots,K\\}$ and reconstructed by table lookup—and the TopKLD distillation loss, $\\mathrm{TopKLD}(P_T \\| P_S) = \\sum_{t,\\, y' \\in M_s} P_S(y'|x,y_{<t}) \\log\\frac{P_S(y'|x,y_{<t})}{P_T(y'|x,y_{<t})} + \\sum_{t,\\, y' \\in M_c} P_T(y'|x,y_{<t}) \\log\\frac{P_T(y'|x,y_{<t})}{P_S(y'|x,y_{<t})}$, where $M_s$ holds the top-K token probabilities and $M_c$ the remaining tail. The codebook space is what makes the model tolerant to quantization noise: reconstruction in discrete space absorbs errors that accumulate in continuous diffusion latents, and the codebook's consistent input features keep activation variance low across inference steps. TopKLD carries the improvement: fitting the top-K tokens with a mode-seeking reverse KL preserves the teacher's confident choices, while fitting the tail with a mode-covering forward KL preserves the less confident 'implicit knowledge' that low-bit students otherwise discard.","core_discovery":"Under quantization, language-style visual generative models have superior bit-level scaling laws to diffusion-style models despite comparable full-precision FID. The paper argues the cause is the discrete codebook representation space: when a quantized model maps features back to codebook indices, small perturbations from low-bit weights and activations are corrected rather than compounded, so FID degrades more gracefully as bits are removed. The paper's proposed TopKLD distillation decomposes the teacher's next-token distribution into top-K and tail parts, applying reverse-KL mode-seeking to the former and forward-KL mode-covering to the latter, and shows this balances explicit and implicit knowledge well enough to lift the scaling curve by one level: 3-bit weights match 4-bit behavior under weight-only quantization, and W4A8 surpasses W8A8 under weight-activation quantization, in both integer and floating-point settings.","pith_inferences":["If the discrete-representation mechanism is the true cause, then any model with discrete tokens—including tokenizer-free bit-token or lookup-free discrete schemes—should show similar quantization tolerance, which is testable without retraining a full model series.","The TopKLD split of the probability distribution is not specific to vision; it could be applied to quantized large language models that use top-k sampling, where the tail of the distribution is known to carry calibration-relevant knowledge.","The paper's scaling comparisons are relative to FID; an extension to other quality metrics or to latency-constrained serving could change which model is optimal, since bit-level scaling ignores hardware efficiency differences beyond bit count."],"forward_implications":["Under a fixed total bit budget, a discrete-token autoregressive generator should beat a diffusion transformer of similar full-precision quality, so bit-constrained deployment should prefer the language-style family.","Reducing a language-style model's weights from 16 to 4 bits and activations from 16 to 8 bits can improve FID at equal memory and compute, because the discrete space tolerates the loss.","Existing post-training quantization methods and plain quantization-aware training do not materially improve bit-level scaling; knowledge distillation is the effective lever.","With TopKLD, W3A16 reaches the scaling behavior of W4A16 and W4A8 surpasses W8A8, and the same one-level gain holds for floating-point quantization."],"supporting_citations":[{"why":"Supplies the V AR language-style model series and its scaling behavior, the primary object of the quantization experiments.","marker":"(Tian et al., 2024)"},{"why":"Supplies the DiT diffusion model series used as the comparison baseline.","marker":"(Peebles & Xie, 2023)"},{"why":"Establishes bit-level (k-bit) scaling laws for quantized models, the framework the paper transfers to vision generation.","marker":"(Dettmers & Zettlemoyer, 2023)"},{"why":"Defines the PTQ and QAT quantization settings that structure the experiments.","marker":"(Nagel et al., 2021)"},{"why":"Provides the finding that reverse-KL mode-seeking fits explicit knowledge better, motivating TopKLD's top-K term.","marker":"(Agarwal et al., 2023)"},{"why":"Shows that decoupled knowledge distillation preserves implicit knowledge in non-target logits, motivating the tail term.","marker":"(Zhao et al., 2022)"},{"why":"Provides SmoothQuant as a baseline for weight-activation quantization comparisons.","marker":"(Xiao et al., 2023)"},{"why":"Provides OmniQuant as a first-order gradient PTQ baseline for scaling comparisons.","marker":"(Shao et al., 2023)"},{"why":"Provides GPTVQ as a vector-quantization PTQ baseline and comparison point.","marker":"(van Baalen et al., 2024)"}],"fun_headline_variants":["Discrete codebooks make quantized vision models more robust","Language-style generators hold up better under low-bit quantization","TopKLD distillation lifts quantized vision model scaling","Discrete token representation tolerates quantization loss","Codebook-based vision models scale better under bit budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on the assumption that FID falls on a single bivariate power law in model size and bit precision with parallel slopes across precisions, so that one number can rank bit-level scaling; the paper does not report the fitted curve, residuals, or confidence intervals, and if the lines are not parallel the 3-bit-versus-4-bit claims lose their quantitative basis.","fun_headline_variants_meta":{"raw":{"variants":["Discrete codebooks make quantized vision models more robust","Language-style generators hold up better under low-bit quantization","TopKLD distillation lifts quantized vision model scaling","Discrete token representation tolerates quantization loss","Codebook-based vision models scale better under bit budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2922,"prompt_tokens":924,"completion_tokens":1998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1933}},"tokens_in":540,"tokens_out":1998,"duration_ms":13359,"temperature":1.0,"reasoning_tokens":1933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:58:24.002232+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit the published FID values for V AR at W16A16, W8A16, W4A16, and W3A16 in log-log coordinates and test whether the slopes are equal; if the W3A16 line is not parallel to the others or its confidence interval overlaps W4A16, the claim that TopKLD elevates scaling by one level would be refuted without needing new experiments. Alternatively, repeat the SNR simulation replacing Gaussian noise with correlated noise matched to actual 3-bit quantization error; if the step-like tolerance disappears, the discrete-space explanation would be an artifact of the noise model.","supporting_citations":[],"review_version":1}