REVIEW 2 major objections 4 minor 1 cited by
Continuous Visual Autoregressive Generation via Score Maximization
T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Training an autoregressive Transformer to maximize the energy score generates visual tokens directly, reaching FID 1.97 on ImageNet 256x256 without vector quantization and cutting inference time to about one second.
desk verdict Energy-score autoregression works and is fast, but the paper overclaims 'continuous beats discrete' by confounding the objective with tokenizer quality. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the energy score as a strictly proper scoring rule, paired with an MLP generator as the implicit sampler. For $\alpha\in(0,2)$, the energy score $S(p,y)=E[|x_1-x_2|^\alpha]-2E[|x-y|^\alpha]$ is strictly proper because the generalized energy distance $2E[|x-y|^\alpha]-E[|x_1-x_2|^\alpha]-E[|y_1-y_2|^\alpha]$ is nonnegative and zero only when $p=q$; this unique optimizer is what turns the negative score into a principled loss rather than an ad hoc distance. The MLP generator consumes the Transformer hidden state and a random noise vector, injecting the noise through adaptive layer normalization in residual blocks, so that the conditional distribution of the next continuous token is defined by the sampling process itself. The energy loss then estimates the score with two draws from this sampler, making the whole objective likelihood-free and single-pass at inference.
What would settle it
Retrain the discrete autoregressive Transformer on a discrete tokenization that matches the KL tokenizer's reconstruction quality (reconstruction FID near 1.22 rather than 5.87), with the same model size, steps, and guidance, and compare FID with EAR-H's 1.97 on ImageNet 256x256; a match or a win for the discrete model would falsify the paper's central claim that continuous autoregression is what delivers the gain.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that continuous visual autoregressive generation can be trained as score maximization: select a strictly proper scoring rule and optimize its negative expectation at every autoregressive step. The concrete instantiation is the energy score $S(p,y)=E[|x_1-x_2|^\alpha]-2E[|x-y|^\alpha]$ with $\alpha\in(0,2)$, whose negative expected value is the generalized energy distance and equals zero only when the predicted distribution $p$ and the true conditional distribution $q$ coincide. Because the score can be estimated from two independent samples of the model distribution, the loss $\mathcal{L}=|x_1-y|^\alpha+|x_2-y|^\alpha-|x_1-x_2|^\alpha$ requires no explicit density, so an MLP generator with injected noise can define an implicit predictive distribution. Trained with masked autoregression, classifier-free guidance, and temperature scheduling on ImageNet 256x256, the largest model (EAR-H, 937M parameters) reaches FID 1.97 with a generation time of roughly one second, compared with about ten seconds for the per-token diffusion baseline (MAR) at comparable FID. The paper further shows GIVT and diffusion loss are special cases of the same framework via the logarithmic score and the Hyvarinen score, respectively.
Load-bearing premise
The load-bearing premise is that the quality gain over discrete autoregressive models comes from the continuous energy-score framework and not from the much better reconstruction of the KL tokenizer, since the head-to-head comparison changes both the objective and the tokenizer at once.
Editorial extensions
If this is right
- Discrete image tokenizers are no longer required for autoregressive generation; continuous tokens can be supervised directly, avoiding the reconstruction bottleneck imposed by vector quantization.
- Inference is one forward pass per token: a 256x256 image is generated in about one second, roughly ten times faster than per-token diffusion at comparable quality.
- Any strictly proper score for continuous variables is a valid training objective, so the framework opens a family of likelihood-free losses beyond the energy score.
- The framework absorbs GIVT and diffusion loss as special cases, which means improvements in scoring-rule theory translate into concrete new autoregressive objectives for visual data.
- Generation quality improves monotonically from EAR-B to EAR-H, so scaling the energy Transformer is a direct route to further FID reduction.
Reading between the lines
- The reported advantage over discrete autoregression is a bundled comparison: the energy objective and the continuous KL tokenizer (reconstruction FID 1.22) change together against a VQ tokenizer with reconstruction FID 5.87. A discrete autoregressive model trained on an equally strong tokenizer would isolate whether continuous autoregression itself, rather than the better tokenizer, is what drives
- The two-sample energy estimator is high-variance, and the paper's fixes, including a reduced learning rate for the MLP generator and the collapse for $\alpha<1$, suggest that estimator variance, not just strict propriety, is what actually limits trainability.
- A natural testable extension is applying the energy objective to continuous latent language modeling or video and audio tokens, where the same single-pass advantage could be measured against per-token diffusion baselines.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a Continuous Visual Autoregressive (VAR) framework that trains autoregressive models with strictly proper scoring rules, avoiding vector quantization. The main instantiation, Energy-based AutoRegression (EAR), maximizes the energy score for alpha in (0,2) via a two-sample unbiased estimator, with an MLP generator that samples continuous tokens in a single forward pass. Experiments on ImageNet 256x256 report EAR-H at FID 1.97 with 937M parameters and roughly one second per image, and the paper shows that alpha=2 (non-strictly proper) fails while alpha<2 succeeds. The paper also positions GIVT and diffusion loss as instances of the same framework via the logarithmic and Hyvärinen scores, respectively.
Significance. If the claims hold, the paper offers a principled, likelihood-free alternative to per-token diffusion for continuous autoregressive generation, with a large inference speed advantage. The theoretical connection between strictly proper scoring rules and autoregressive training is clearly presented, and the alpha=2 ablation is a clean empirical validation of strict propriety. The code release and detailed ablations (temperature, noise, CFG, learning rate, masking) support reproducibility. However, the headline comparison against discrete autoregressive models is not yet established because it conflates the training objective with tokenizer quality.
major comments (2)
- [Section 5.2, Figure 3, Table 1] The comparison supporting the claim of superiority over discrete autoregressive generation is confounded by tokenizer quality. The continuous KL-16 tokenizer used for EAR has a reconstruction FID of 1.22, while the discrete VQ-16 tokenizer used for the baseline has a reconstruction FID of 5.87. Because the tokenizer's reconstruction error upper-bounds generation quality, the gap in Figure 3 is expected even if the energy-score objective and cross-entropy were equally effective. Moreover, Table 1 lists discrete autoregressive models with better tokenizers (MAGVIT-v2 at FID 1.78 and VAR-d30 at FID 1.92) that match or beat EAR-H's 1.97. The paper acknowledges the reconstruction bottleneck in Section 5.2 but still concludes that continuous tokenization with the energy loss consistently outperforms discrete tokenization with cross-entropy. I request either a matched-tokenizer discrete baseline (for example, a discrete autoregressive model trained on the same KL-16 latent space after quantization, or a cross-entropy model using MAGVIT-v2 or FSQ tokens) or a clearly qualified claim that restricts the comparison to the specific VQ-16 baseline and presents EAR as a practical alternative rather than a generally superior paradigm.
- [Section 4.3 (Temperature) and Section 5.1] The actual training objective used for the final model is not exactly the strictly proper energy score. With tau_train=0.99, the fine-tuning loss becomes L = |x1-y|^alpha + |x2-y|^alpha - tau_train|x1-x2|^alpha, which is not an unbiased estimator of the negative energy score; the extra term (1-tau_train) E|x1-x2|^alpha penalizes the spread of the model distribution and breaks the strict-propriety rationale. The paper labels this as a fine-tuning heuristic, but the abstract's statement that 'all we need is to select a strictly proper score and set it as the training objective to optimize' is then not literally true for the reported final checkpoint. Please clarify how the temperature-weighted loss is justified within the proposed framework, or explicitly classify the temperature modification as an orthogonal regularization technique that falls outside the scoring-rule theory.
minor comments (4)
- [Figure 6 caption] The caption contains a typo: 'gudiance' should be 'guidance'.
- [Section 5.2] The description of EAR-H as having a 'relatively modest model size among the leading systems' is misleading given that 937M parameters is larger than DiT-XL (675M), MAGVIT-v2 (307M), and comparable to MAR (943M); please rephrase to specify the comparison set for which the size is modest.
- [Table 3] The conclusion that dnoise=64 performs better rests on small differences (e.g., w/ cfg Uniform: 3.55 at dnoise=64 vs. 3.89 at dnoise=32); please report multiple seeds or confidence intervals to support this choice.
- [References] The reference 'Alpha-VLLM' points to a GitHub repository rather than an archival publication; please cite the corresponding paper if available, or specify the exact version and commit used.
Circularity Check
No significant circularity: the energy-score objective is grounded in external statistical theory and evaluated on held-out FID; the only self-citation is non-load-bearing provenance.
full rationale
The paper's derivation chain does not reduce to its inputs. Equation (10) is a Monte Carlo estimator of the negative energy score defined in Equation (8), and its strict propriety for alpha in (0,2) is cited to Székely (2003) and Székely & Rizzo (2013), which are external mathematical results independent of this paper. The FID values in Table 1 are obtained by training on ImageNet and evaluating on held-out samples; no parameter is fitted to a target FID, so there is no fitted-input-called-prediction pattern. The sequence loss in Equation (5) cites Shao et al. (2024), an overlapping-author prior work, but that citation is only provenance for a straightforward per-step sum of scoring rules; the validity of EAR does not rest on any empirical claim from that citation. The derivations of GIVT and diffusion loss in Section 3.3 are post-hoc identifications with the logarithmic and Hyvärinen scores through standard identities (Equations 6 and 7); they are re-descriptions of known methods, not inputs used to produce the energy-score predictions, and they do not make the central claim circular. The discrete-versus-continuous comparison in Section 5.2 varies tokenizer and objective jointly, which is a potential confound and a correctness/interpretation risk, but not a circularity: the comparison is an empirical claim whose attribution could be challenged, not a derivation that assumes its conclusion. Therefore no circular step is identified; score 2 only to acknowledge a minor non-load-bearing self-citation.
Assumptions & free parameters
free parameters (7)
- alpha (energy loss exponent) =
1.0
- tau_train (training temperature on diversity term) =
0.99 during final 50 epochs
- tau_infer (inference temperature) =
0.7
- lambda (MLP generator learning rate multiplier) =
0.25
- cfg (classifier-free guidance scale) =
3.0 for ImageNet 256
- dnoise (noise dimension) =
64
- Masking ratio range / generation steps =
[0.7, 1.0] mask ratio, 64 steps
assumptions (6)
- standard math Strictly proper scoring rules have a unique maximizer at the true distribution.
- standard math The energy distance E_alpha(p,q) = 2E|x-y|^alpha - E|x1-x2|^alpha - E|y1-y2|^alpha is a metric and equals zero iff p=q for alpha in (0,2).
- standard math The two-sample loss L(p,y) is an unbiased estimator of the negative energy score.
- domain assumption The MLP generator's output distribution p is the implicit predictive distribution, and gradients of the loss with respect to parameters can be computed by backpropagating through the generator.
- domain assumption Masked autoregressive generation with random masking and a fixed schedule is a valid autoregressive sequence loss.
- domain assumption The continuous KL-16 tokenizer (reconstruction FID 1.22) is an appropriate target representation.
Cite this review
Pith. "Pith review of Continuous Visual Autoregressive Generation via Score Maximization." pith.science (2026). https://pith.science/paper/ZP6EVFXM
@misc{pith2026250507812,
author = {Pith},
title = {Pith review of: Continuous Visual Autoregressive Generation via Score Maximization},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZP6EVFXM}},
note = {Machine review of arXiv:2505.07812}
}
read the original abstract
Conventional wisdom suggests that autoregressive models are used to process discrete data. When applied to continuous modalities such as visual data, Visual AutoRegressive modeling (VAR) typically resorts to quantization-based approaches to cast the data into a discrete space, which can introduce significant information loss. To tackle this issue, we introduce a Continuous VAR framework that enables direct visual autoregressive generation without vector quantization. The underlying theoretical foundation is strictly proper scoring rules, which provide powerful statistical tools capable of evaluating how well a generative model approximates the true distribution. Within this framework, all we need is to select a strictly proper score and set it as the training objective to optimize. We primarily explore a class of training objectives based on the energy score, which is likelihood-free and thus overcomes the difficulty of making probabilistic predictions in the continuous space. Previous efforts on continuous autoregressive generation, such as GIVT and diffusion loss, can also be derived from our framework using other strictly proper scores. Source code: https://github.com/shaochenze/EAR.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
An MLLM-conditioned next-scale VAR decoder handles 15+ unified visual generation tasks with competitive quality and substantially lower latency than diffusion baselines.
Reference graph
Works this paper leans on
-
[7]
cc/paper_files/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper. pdf. Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In Gurevych, I. and Miyao, Y . (eds.),Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers) , pp. 32...
-
[8]
cc/paper_files/paper/2019/file/ 0234c510bc6d908b28c70ff313743079-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 0234c510bc6d908b28c70ff313743079-Paper. pdf. 11 Continuous Visual Autoregressive Generation via Score Maximization Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems...
arXiv 2019
-
[9]
cc/paper_files/paper/2019/file/ 8558cb408c1d76621371888657d2eb1d-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 8558cb408c1d76621371888657d2eb1d-Paper. pdf. Pacchiardi, L. and Dutta, R. Likelihood-free inference with generative neural networks via scoring rule minimiza- tion, 2022. URL https://arxiv.org/abs/2205. 15784. Pacchiardi, L., Adewoyin, R. A., Dueben, P., and Dutta, R. Probabilistic forecasti...
work page 2019
-
[10]
Razavi, A., van den Oord, A., and Vinyals, O
URL https://proceedings.mlr.press/ v139/ramesh21a.html. Razavi, A., van den Oord, A., and Vinyals, O. Generating di- verse high-fidelity images with vq-vae-2. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alch´e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.,
-
[11]
cc/paper_files/paper/2019/file/ 5f8e2fa1718d1bbcadf1cd9c7a54fb8c-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 5f8e2fa1718d1bbcadf1cd9c7a54fb8c-Paper. pdf. Roby, T. B. Belief states: A preliminary empirical study. Behavioral Sci, 10(3):255–270, 1965. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the I...
work page 2019
-
[14]
cc/paper_files/paper/2019/file/ 3001ef257407d5a371a96dcd947c7d93-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 3001ef257407d5a371a96dcd947c7d93-Paper. pdf. Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. In In- ternational Conference on Learning Representations ,
work page 2019
-
[15]
Sun, P., Jiang, Y ., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z
URL https://openreview.net/forum? id=PxTIG12RRHS. Sun, P., Jiang, Y ., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation, 2024a. URL https: //arxiv.org/abs/2406.06525. Sun, Y ., Bao, H., Wang, W., Peng, Z., Dong, L., Huang, S., Wang, J., and Wei, F. Multimodal latent language mod...
arXiv 2003
-
[17]
URL https://openreview.net/forum? id=gojL67CfS8. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023. Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Gener- ative infinite-voca...
arXiv 2023
Show all 20 references
-
[18]
Team, G., Anil, R., Borgeaud, S., Wu, Y ., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A
URL https://www.sciencedirect.com/ science/article/pii/S0378375813000633. Team, G., Anil, R., Borgeaud, S., Wu, Y ., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:231...
2023 arXiv
-
[19]
cc/paper_files/paper/2017/file/ 7a98af17e63a0ac09ce2e96d03992fbc-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ 7a98af17e63a0ac09ce2e96d03992fbc-Paper. pdf. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V ...
2017
-
[20]
cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper. pdf. Vincent, P. A connection between score matching and de- noising autoencoders. Neural Computation, 23(7):1661– 1674, 2011. doi: 10.1162/NECO a 00142. Weber, M., Yu, L.,...
2017 arXiv
-
[2014]
cc/paper_files/paper/2014/file/ 5ca3e9b122f61f8f06494c97b1afccf3-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2014/file/ 5ca3e9b122f61f8f06494c97b1afccf3-Paper. pdf. Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep autoregressive networks. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st Interna...
2014
-
[2016]
cc/paper_files/paper/2016/file/ 8a3363abe792db2d8761d6403605aeb7-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2016/file/ 8a3363abe792db2d8761d6403605aeb7-Paper. pdf. Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1–10, 2022. Scheuerer,...
2016
-
[2017]
cc/paper_files/paper/2017/file/ 8a1d694707eb0fefe65871369074926d-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ 8a1d694707eb0fefe65871369074926d-Paper. pdf. Ho, J. and Salimans, T. Classifier-free diffusion guid- ance, 2022. URL https://arxiv.org/abs/ 2207.12598. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabi...
2017 arXiv
-
[2019]
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W
URL https://openreview.net/forum? id=B1xsqj09Fm. Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 11315–11325, June 20...
2022
-
[2020]
cc/paper_files/paper/2020/file/ 9873eaad153c6c960616c89e54fe155a-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 9873eaad153c6c960616c89e54fe155a-Paper. pdf. Gruber, S. and Buettner, F. Better uncertainty calibration via proper scores for classification and beyond. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho,...
2020
-
[2021]
cc/paper_files/paper/2021/file/ 49ad23d1ec9fa4bd8d77d02681df5cfa-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ 49ad23d1ec9fa4bd8d77d02681df5cfa-Paper. pdf. Dinh, L., Krueger, D., and Bengio, Y . NICE: non-linear independent components estimation. In Bengio, Y . and LeCun, Y . (eds.),3rd International Conference on Learn- ...
2021 arXiv
-
[2022]
Yu, L., Lezama, J., Gundavarapu, N
URL https://openreview.net/forum? id=pfNyExj7z2. Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y ., Gupta, A., Gu, X., Haupt- mann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion - tokenizer...
2021
-
[2023]
Deng, H., Pan, T., Diao, H., Luo, Z., Cui, Y ., Lu, H., Shan, S., Qi, Y ., and Wang, X
URL https://proceedings.mlr.press/ v202/chang23b.html. Deng, H., Pan, T., Diao, H., Luo, Z., Cui, Y ., Lu, H., Shan, S., Qi, Y ., and Wang, X. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14169, 2024. Deng, J., Dong, W., Socher, R., Li...
2024 arXiv
-
[2024]
Shoemaker, P
URL https://openreview.net/forum? id=LALSZ88Xpx. Shoemaker, P. A note on least-squares learning procedures and classification by neural network models. IEEE Trans- actions on Neural Networks, 2(1):158–160, 1991. doi: 10.1109/72.80304. Singh, B., De, S., Zhang, Y ., Goldstein, ...
1991 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.