Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Spectral Image Tokenizer

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Image tokens ordered by wavelet scale enable coarse-to-fine generation.

desk verdict A genuinely useful tokenizer with a real attribution gap: the gains over ViT-VQGAN aren't isolated to the spectral input, but the multiscale applications are novel and worth refereeing. read the letter →

arxiv 2412.09607 v2 pith:VV4I4KE3 submitted 2024-12-12 cs.CV cs.LG

Carlos Esteves , Mohammed Suhail , Ameesh Makadia This is my paper · ORCID
classification cs.CVcs.LG
keywords imagetokenizationdiscretewavelettransformautoregressivegenerationcoarse-to-finesequencevectorquantizationscale-causalattentionmultiscalereconstructiontext-guidedupsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes replacing the usual raster-scan tokenization of image patches with a tokenizer that operates on discrete wavelet transform (DWT) coefficients, so the token sequence naturally runs from a coarse approximation of the whole image to finer detail. The claim is that this ordering is better suited to autoregressive transformers: the model predicts the next scale while conditioning on a coarse reconstruction of the full image, rather than on a partial row-by-row reconstruction. The authors show the tokenizer reconstructs images at multiple resolutions with a single training, that generation can be stopped early to yield a coarse image, and that the same machinery supports text-guided upsampling and editing. If correct, the paper provides a spectral, multiscale tokenizer that brings autoregressive image generation closer to the sequential structure of language models.

What carries the argument

The central object is DWT coefficient patchification combined with scale-causal attention. Each wavelet scale becomes one block of tokens: the approximation coefficients form the first scale, and horizontal, vertical, and diagonal detail coefficients are concatenated spatially so every scale has the same token count. The scale-causal attention mask, applied to the encoder or decoder or both, is what endows the representation with its multiscale behavior, since it forces each scale to depend only on itself and lower scales. The Approximation-Details Transformer (ADTransformer), with separate parameters and codebooks for the approximation and detail tokens, handles the fact that the two coefficient distributions are very different.

What would settle it

Train a SIT with scale-causal attention on both encoder and decoder at 256×256, then decode only the first-scale tokens into a full-resolution image and measure LPIPS and PSNR against ground truth. If the coarse reconstruction from the first scale alone is not a recognizable image, the claim that the representation is truly coarse-to-fine and decodable at partial scales would be falsified. The paper's own ablation, which shows LPIPS degrading from 0.143 to 0.184 when both masks are enabled, indicates the direction of the risk.

Watch

Extended reading notes

Core claim

The central claim is that tokenizing the spectrum rather than the pixels is a better inductive bias for autoregressive image modeling. SIT applies a Haar DWT, patchifies each scale separately with the same number of tokens per scale (so higher scales use larger patches and are compressed more), and feeds the resulting sequence through a transformer with separate parameters and codebooks for approximation versus detail coefficients. An optional scale-causal attention mask lets each scale attend only to itself and lower scales, which makes the representation multiscale without retraining. With this tokenizer, an autoregressive transformer (AR-SIT) models the coarse-to-fine sequence; the authors report improved reconstruction over the ViT-VQGAN baseline (LPIPS 0.135 versus 0.167 at 256×256 for SIT-5), improved class-conditional generation (FID 6.95 versus 8.37), and comparable text-to-image FID while adding the ability to partially decode, upsample, and edit images.

Load-bearing premise

The load-bearing premise is that the low-scale approximation tokens contain enough information that all higher-scale detail tokens can be predicted or generated from them, so the scale-causal mask that blocks high-frequency information from influencing low-frequency tokens does not cripple reconstruction quality.

Editorial extensions

If this is right

  • One model trained at a single resolution can tokenize and reconstruct images at lower and higher resolutions without retraining, because the scale-causal mask makes unused scales simply absent from the sequence.
  • Early termination of autoregressive generation yields a coarse image after only the first tokens, so a user can preview many drafts cheaply and refine only the selected one.
  • Text-guided upsampling becomes a continuation problem: encode the low-resolution input into the first tokens and let the autoregressive model generate the rest.
  • Text-guided editing becomes a freezing problem: keep the low scales of an input image and regenerate higher scales under a new caption.
  • Class-conditional generation improves over the comparable raster-order baseline, supporting the claim that coarse-to-fine conditioning is better for next-token prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the coarse-to-fine ordering is what matters, then other invertible multiscale transforms with energy compaction, such as DCT or learned filter banks, could yield similar gains; the paper only demonstrates DWT with Haar.
  • The reconstruction-quality cost of scale-causal attention suggests a possible refinement: allow high-frequency information to influence low-frequency tokens during encoding but decode with strict scale-causality, or predict detail tokens conditionally with a learned prior over the approximation coefficients.
  • The observation that bigger codebooks improve reconstruction but not generation hints at a compression-generation tradeoff that could be studied by varying the number of tokens per scale according to the energy of each scale.
  • A natural extension is to make the number of tokens per scale adaptive to image content, using more tokens where detail is actually present, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the Spectral Image Tokenizer (SIT), a ViT-VQGAN-style image tokenizer that operates on Haar discrete wavelet transform (DWT) coefficients instead of spatial pixel patches. The DWT output is organized into scales, an approximation scale followed by detail scales, and each scale is patchified with scale-specific linear embeddings, so higher-frequency scales are represented by larger patches and hence compressed more. The tokenizer transformer can use a scale-causal attention mask and separate parameters and codebooks for approximation versus detail coefficients (ADTransformer). The resulting token sequence is coarse-to-fine. The paper then trains an autoregressive transformer (AR-SIT) over SIT tokens and evaluates it for text-to-image generation, partial decoding, text-guided upsampling, text-guided editing, and class-conditional generation. Experiments report reconstruction metrics on ImageNet at multiple resolutions, text-to-image metrics on MS-COCO, and class-conditional FID on ImageNet, together with ablations over scales, sequence length, vocabulary size, wavelet family, and scale-causal attention. The authors state explicitly that text-to-image FID is similar to the Parti baseline and that the scale-causal mask reduces reconstruction accuracy.

Significance. If the central claim holds, SIT is a useful step toward making autoregressive image generation genuinely multiscale: a single tokenizer can handle multiple resolutions without retraining, supports partial decoding, and enables text-guided upsampling and editing. The paper is commendably transparent: it reproduces the ViT-VQGAN baseline, reports throughput and memory, includes ablations, and states its limitations (small AR models, similar text-to-image metrics, scale-causal tradeoff). The main weakness is attribution: SIT differs from ViT-VQGAN not only in the input representation but also in the transformer internals and quantizer, so the reported gains are not yet isolated to the spectral representation. With the missing ablation supplied, the paper would provide solid evidence for its claims.

major comments (3)
  1. [§4.1, Tables 1, 3, 5] The paper's central claim, stated in §1 and repeated in §2, is that tokenizing DWT coefficients rather than spatial patches is the 'crucial difference' from prior work. However, Tables 1, 3, and 5 compare SIT against ViT-VQGAN under a combined change: SIT uses DWT inputs, the ADTransformer with separate approximation/detail parameters, and two separate codebooks. No experiment isolates the input representation by ablating these components independently. For example, one could keep the ADTransformer and dual codebooks but feed a spatial multiscale representation (e.g., a Laplacian pyramid or downsampled image plus residuals), or keep the DWT input but revert to a shared transformer and single codebook. Without such a control, the reconstruction improvements (e.g., LPIPS 0.135 vs 0.167 at 256x256 in Table 1) and the class-conditional FID gain (6.95 vs 8.37 in Table 3) cannot be attributed specifically to the spectral input. This is an omitted control rather than an internal inconsistency, but it is load-bearing for the paper's stated contribution.
  2. [Table 5, Section 4.1] Table 5 shows that scale-causal attention is not free: for SIT-4, enabling both scale-causal encoder and decoder increases LPIPS from 0.143 to 0.184 and FID from 1.20 to 1.97 at 256x256, and enabling only the scale-causal decoder still increases LPIPS to 0.166 and FID to 1.45. Since properties P.2 and P.4-P.6 all depend on scale-causal masking, the paper should quantify the benefit side of this tradeoff with the same metrics. I would like to see, at minimum, reconstruction quality and FID as a function of the number of decoded scales for partial decoding, and a statement of how the full-resolution loss weighs against the multiscale capabilities. Without this quantification, the claim that the scale-causal attention enables different applications is supported only qualitatively.
  3. [Tables 1, 2, 3; Section 5.1] Most reported numbers are single-run and no error bars or significance tests are given. This matters because several headline comparisons are close (AR-SIT FID 12.6 vs Parti 12.4 in Table 2) and because the 512x512 ViT-VQGAN row in Table 1 is explicitly selected as the best checkpoint before divergence. The paper should state the number of seeds or runs used, report means and standard deviations (or an equivalent dispersion measure), and describe the checkpoint-selection protocol for the unstable baseline. This is standard practice and would materially strengthen the empirical claims in §5.1 and §5.2.
minor comments (5)
  1. [Section 6] The word 'realtively' in the conclusion should be 'relatively'.
  2. [Table 5] The SCE and SCD columns use both 'X' and '✓' without a legend; please state explicitly that '✓' means the scale-causal mask is enabled.
  3. [Section 1] Reference [37] appears twice in the citation list [15, 37, 37, 51, 57]; the duplicate should be removed.
  4. [Equation (6), Section 4.2] Equation (6) writes the categorical distribution over ⌊q^n_s⌋, but the notation ⌊q^n_s⌋ is introduced just before as 'its discrete position in the codebook'; consider using a distinct symbol such as k^n_s to avoid confusion with an integer-part or floor operation.
  5. [Section 5.3 and Table 4] The main text reports an FID of 6.2 for 32x32 upsampling, while the supplemental Table 4 lists 6.19; please state which number is used and confirm that the evaluation subset and ground-truth protocol are the same as in Table 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's claims are empirically tested against external baselines and its design choices are ablated rather than derived from the target results.

full rationale

The paper makes no derivation that reduces to its own inputs. The central contribution, tokenizing DWT coefficients instead of spatial patches, is an architectural proposal whose benefits are evaluated through reconstruction metrics, generation quality, and downstream applications, all compared with external baselines such as ViT-VQGAN and Parti. The design choices are explicitly tested in the ablation study (Table 5), which varies the number of scales, sequence length, vocabulary size, wavelet family, and the use of scale-causal attention on the encoder and decoder. These ablations show that the scale-causal mask costs reconstruction quality, so the multiscale properties are presented as trade-offs rather than as forced consequences of the target result. The paper also acknowledges the text-to-image metrics were similar to the fair baseline and that a better tokenizer does not necessarily lead to a better generative model, which further indicates the claims are not constructed to be true by definition. There are no load-bearing self-citations: the references to prior work are external baselines and contextual methods, not prior papers by the same authors invoked to justify the central premise. No fitted parameter is renamed as a prediction; the learned codebooks and transformer weights are trained and then evaluated on held-out data in the standard manner. The strongest concern, noted by the skeptic framing, is that the empirical comparison changes both the input representation and the transformer/quantizer internals simultaneously, so the specific attribution of gains to the spectral input is not fully isolated. However, this is an omitted control and a correctness or attribution risk, not circularity: the paper does not define the input representation in terms of the outcome, nor does it import uniqueness from self-citation, nor does it present a fit as a prediction. Therefore the paper is self-contained with respect to the circularity criteria, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on architectural choices (token count, number of scales, separate codebooks) and on the empirical assumption that DWT coefficients, especially low-frequency approximation, carry enough information for coarse-to-fine generation. No new physical entities are introduced.

free parameters (5)
  • tokens per scale N = 256 (varies 64 to 2048 in ablations)
    Chosen by hand; the sequence length per scale determines the coarse-to-fine resolution granularity and reconstruction quality (Table 5).
  • number of scales S = 4 or 5 (up to 6 tested)
    Selected per experiment; more scales increase sequence length and reconstruction quality up to a point, but hurt generation (Sec 5.1, Appendix A).
  • codebook size = 8192 per codebook (4096 and 16384 ablated)
    Follows ViT-VQGAN; ablation shows larger vocab improves reconstruction but may hurt generation.
  • loss weights = L2 1.0, perceptual 0.1, adversarial 0.1, commitment 0.25
    Copied from ViT-VQGAN protocol; AR-SIT-4* uses different weights (1.0, 1.0, 0.5, 0.25).
  • per-scale loss weighting in AR-SIT-4* = each scale 4x smaller than previous
    Found to improve performance slightly (Appendix B.3).
assumptions (4)
  • standard math 2D DWT with Haar filters is invertible
    Used to reconstruct images from quantized coefficients after decoding (Sec 4.1, Eq 3 and IDWT).
  • domain assumption Natural image power spectra concentrate energy in low frequencies
    Justifies assigning the same number of tokens per scale so high frequencies are more compressed (Sec 4.1, P.1).
  • domain assumption Approximation and detail DWT coefficients have distinct distributions that warrant separate codebooks and transformer parameters
    Central design choice (ADTransformer, Sec 4.1); validated only indirectly by reconstruction metrics.
  • domain assumption A scale-causal attention mask that restricts tokens to lower scales provides sufficient context for encoding and generating higher scales
    Enables multiscale decoding and partial generation (Sec 4.1); ablation shows it degrades reconstruction (Table 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spectral Image Tokenizer." pith.science (2026). https://pith.science/paper/VV4I4KE3

@misc{pith2026241209607,
  author       = {Pith},
  title        = {Pith review of: Spectral Image Tokenizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VV4I4KE3}},
  note         = {Machine review of arXiv:2412.09607}
}
read the original abstract

Image tokenizers map images to sequences of discrete tokens, and are a crucial component of autoregressive transformer-based image generation. The tokens are typically associated with spatial locations in the input image, arranged in raster scan order, which is not ideal for autoregressive modeling. In this paper, we propose to tokenize the image spectrum instead, obtained from a discrete wavelet transform (DWT), such that the sequence of tokens represents the image in a coarse-to-fine fashion. Our tokenizer brings several advantages: 1) it leverages that natural images are more compressible at high frequencies, 2) it can take and reconstruct images of different resolutions without retraining, 3) it improves the conditioning for next-token prediction -- instead of conditioning on a partial line-by-line reconstruction of the image, it takes a coarse reconstruction of the full image, 4) it enables partial decoding where the first few generated tokens can reconstruct a coarse version of the image, 5) it enables autoregressive models to be used for image upsampling. We evaluate the tokenizer reconstruction metrics as well as multiscale image generation, text-guided image upsampling and editing.

Figures

Figures reproduced from arXiv: 2412.09607 by the authors.

Figure 1
Figure 1. Left: we introduce a Spectral Image Tokenizer (SIT), that learns to encode and decode discrete wavelet transform (DWT) coefficients to and from a small set of discrete tokens, such that the sequence represents the image in a coarse-fine fashion. SIT is naturally multiscale and enables coarse-to-fine autoregressive image generation with our AR-SIT model. SIT also leverages the sparsity of high frequency coefficients … view at source ↗
Figure 2
Figure 2. Input patchification. Left: typical patchification for Vi￾sion Transformers (ViT) [13], where the image is split in equal￾sized patches. Right: we propose to patchify the coefficients of a discrete wavelet transform (DWT) instead. Each scale is shown in a different color. Scales other than the lowest contain three blocks representing horizontal, vertical and diagonal details; we concate￾nate the spatially correspond… view at source ↗
Figure 3
Figure 3. Coarse-to-fine text-to-image on MS-COCO [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Text-guided image upsampling on MS-COCO [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Text-guided image editing on MS-COCO [27]. Our coarse-to-fine generative models can do text-guided editing by encoding a given image but keeping only the lower scales, and using a pre-trained AR-SIT to re-generate the higher scales conditioned on the textual prompt. Ea…
Figure 6
Figure 6. Figure 6: Additional results for text-guided image editing on MS-COCO [ [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Multiscale reconstruction on ImageNet. Each triplet shows reconstruction from the ViT-VQGAN baseline [ [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Additional text-guided image upsampling results. Here we consider the more challenging task of upsampling from [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Samples of class-conditional generation with AR-SIT-4 on [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Samples of class-conditional generation with AR-SIT-5* on [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StrandDesigner: Towards Practical Strand Generation with Sketch Guidance

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Sketch drawings can directly control 3D hair strand generation through multi-scale latent upsampling with adaptive DINOv2 conditioning, yielding more accurate results than text- or image-guided baselines.

  2. Improving Progressive Generation with Decomposable Flow Matching

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DFM applies flow matching with per-stage noise schedules to Laplacian pyramid levels in one shared DiT, improving FID and FDD on ImageNet and video benchmarks over prior progressive generation baselines.

  3. SpectralAR: Spectral Autoregressive Visual Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    An autoregressive image generator that tokenizes images in the DCT frequency domain into nested 1D spectral sequences and generates them coarse-to-fine, reaching 3.02 gFID with 64 tokens on ImageNet-1K.

  4. Flow Along the K-Amplitude for Generative Modeling

    cs.LG 2025-04 reject novelty 6.0 of 10

    K-Flow trains flow-matching models with frequency scale as time, enabling competitive image and molecule generation plus scale-level control of outputs.

  5. DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Diffusion models trained on DCT-compressed image representations outperform pixel-based and latent (VAE) diffusion baselines at lower training cost.

  6. Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A fixed-size triplane scene representation tokenizes any number of camera images into 45-104 tokens, enabling faster autoregressive driving policies without sacrificing planning accuracy.

Reference graph

Works this paper leans on

62 extracted references · 49 canonical work pages · cited by 6 Pith papers

  1. [1]

    Im- proving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, † TimBrooks, Jian- feng Wang, Linjie Li, † LongOuyang, † JuntangZhuang, † JoyceLee, † YufeiGuo, † WesamManassra, † PrafullaDhari- wal, † CaseyChu, † YunxinJiao, and Aditya Ramesh. Im- proving image generation with better captions. 2

  2. [2]

    Multi-Stage Vari- ational Auto-Encoders for Coarse-to-Fine Image Genera- tion, page 630–638

    Lei Cai, Hongyang Gao, and Shuiwang Ji. Multi-Stage Vari- ational Auto-Encoders for Coarse-to-Fine Image Genera- tion, page 630–638. Society for Industrial and Applied Math- ematics, 2019. 3

    work page 2019

  3. [3]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022. 1, 3

    work page 2022

  4. [4]

    Freeman, Michael Ru- binstein, Yuanzhen Li, and Dilip Krishnan

    Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Ru- binstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text- to-image generation via masked generative transformers. In ICML, 2023. 1, 3, 9

    work page 2023

  5. [5]

    PixelSNAIL: An improved autoregressive genera- tive model

    XI Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixelSNAIL: An improved autoregressive genera- tive model. In ICML, 2018. 3

    work page 2018

  6. [6]

    Pali: A jointly-scaled multilingual language-image model, 2022

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Brad- bury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Ca...

    work page 2022

  7. [7]

    C. A. Christopoulos, T. Ebrahimi, and A. N. Skodras. Jpeg2000: the new still picture compression standard. In Proceedings of the 2000 ACM Workshops on Multimedia , page 45–49, New York, NY , USA, 2000. Association for Computing Machinery. 4

    work page 2000

  8. [8]

    Cohen, Ingrid Daubechies, and J.-C

    A. Cohen, Ingrid Daubechies, and J.-C. Feauveau. Biorthog- onal bases of compactly supported wavelets. Communi- cations on Pure and Applied Mathematics , 45(5):485–560,

    work page

Show all 62 references
  1. [9]

    Ten lectures on wavelets

    Ingrid Daubechies. Ten lectures on wavelets . Society for Industrial and Applied Mathematics, USA, 1992. 3

    1992

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 6

    2009

  3. [11]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021. 1

    2021

  4. [12]

    Diffusion is spectral autoregression, 2024

    Sander Dieleman. Diffusion is spectral autoregression, 2024. 2

    2024

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  6. [14]

    Quantised global autoencoder: A holistic approach to representing visual data,

    Tim Elsner, Paula Usinger, Victor Czech, Gregor Kobsik, Yanjiang He, Isaak Lim, and Leif Kobbelt. Quantised global autoencoder: A holistic approach to representing visual data,

  7. [15]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Taming transformers for high-resolution image synthesis. In CVPR,

  8. [16]

    Scaling rectified flow trans- formers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yan- nik Marek, and Robin Rombach. Scaling rectified flow tr...

    2024

  9. [17]

    Swagan: a style-based wavelet-driven generative model

    Rinon Gal, Dana Cohen Hochberg, Amit Bermano, and Daniel Cohen-Or. Swagan: a style-based wavelet-driven generative model. ACM TOG, 2021. 1

    2021

  10. [18]

    Susskind, and Navdeep Jaitly

    Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M. Susskind, and Navdeep Jaitly. Matryoshka diffusion mod- els. In ICLR, 2024. 3

    2024

  11. [19]

    Learnings from scaling visual tokenizers for reconstruction and generation,

    Philippe Hansen-Estruch, David Yan, Ching-Yao Chung, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vish- wanath, Peter Vajda, and Xinlei Chen. Learnings from scaling visual tokenizers for reconstruction and generation,

  12. [20]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In NeurIPS, 2017. 2

    2017

  13. [21]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. 1

    2022

  14. [22]

    Towards accurate image coding: Improved au- toregressive image generation with dynamic vector quantiza- tion

    Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yong- dong Zhang. Towards accurate image coding: Improved au- toregressive image generation with dynamic vector quantiza- tion. In CVPR, 2023. 2, 8

    2023

  15. [23]

    Imagen 3, 2024

    Imagen-Team-Google. Imagen 3, 2024. 2

    2024

  16. [24]

    Progressive growing of GANs for improved quality, stability, and variation

    Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018. 3

    2018

  17. [25]

    Sinddm: A single image denoising diffusion model

    Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. In ICML, pages 17920–17930, 2023. 3

    2023

  18. [26]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022. 2, 8

    2022

  19. [27]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV. 5, 6, 7, 8, 2, 3

  20. [28]

    Star: Scale-wise text-to- image generation via auto-regressive representations, 2024

    Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise text-to- image generation via auto-regressive representations, 2024. 2

    2024

  21. [29]

    A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way

    Stephane Mallat. A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way . Academic Press, Inc., USA, 3rd edition, 2008. 3

    2008

  22. [30]

    Wavelets are all you need for autoregressive image gener- ation, 2024

    Wael Mattar, Idan Levy, Nir Sharon, and Shai Dekel. Wavelets are all you need for autoregressive image gener- ation, 2024. 3

    2024

  23. [31]

    Spectral normalization for generative ad- versarial networks

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative ad- versarial networks. In ICLR, 2018. 5

    2018

  24. [32]

    Battaglia

    Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W. Battaglia. Generating images with sparse representations. In ICML, 2021. 3

    2021

  25. [33]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 2

    2023

  26. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. 2023. 8

    2023

  27. [35]

    Wavelet diffusion models are fast and scalable image generators

    Hao Phung, Quan Dao, and Anh Tran. Wavelet diffusion models are fast and scalable image generators. In CVPR,

  28. [36]

    When worse is better: Navigating the compression-generation tradeoff in visual to- kenization, 2024

    Vivek Ramanujan, Kushal Tirumala, Armen Aghajanyan, Luke Zettlemoyer, and Ali Farhadi. When worse is better: Navigating the compression-generation tradeoff in visual to- kenization, 2024. 1

    2024

  29. [37]

    Generat- ing diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generat- ing diverse high-fidelity images with vq-vae-2. In NeurIPS,

  30. [38]

    Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer

    Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2021. 1

    2021

  31. [39]

    Sin- gan: Learning a generative model from a single natural im- age

    Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Sin- gan: Learning a generative model from a single natural im- age. In ICCV, 2019. 3

    2019

  32. [40]

    Pyramidal denoising diffu- sion probabilistic models, 2022

    Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffu- sion probabilistic models, 2022. 3

    2022

  33. [41]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016. 2

    2016

  34. [42]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020. 8

    2020

  35. [43]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 1

    2015

  36. [44]

    Roformer: Enhanced transformer with rotary position embedding, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021. 8

    2021

  37. [45]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 2, 8

    2024 arXiv

  38. [46]

    Gemini: A family of highly capable multi- modal models, 2023

    Gemini Team. Gemini: A family of highly capable multi- modal models, 2023. 1, 2

    2023

  39. [47]

    The llama 3 herd of models, 2024

    Llama team. The llama 3 herd of models, 2024. 1, 2

    2024

  40. [48]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurIPS, 2024. 2, 8

    2024

  41. [49]

    Condi- tional image generation with pixelcnn decoders

    Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, ko- ray kavukcuoglu, Oriol Vinyals, and Alex Graves. Condi- tional image generation with pixelcnn decoders. In NeurIPS,

  42. [50]

    Pixel recurrent neural networks

    A ¨aron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML,

  43. [51]

    Neural discrete representation learning

    Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017. 1, 2, 3, 5

    2017

  44. [52]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 1

    2017

  45. [53]

    ”principal components” enable a new language of images, 2025

    Xin Wen, Bingchen Zhao, Ismail Elezi, Jiankang Deng, and Xiaojuan Qi. ”principal components” enable a new language of images, 2025. 2

    2025

  46. [54]

    Reconstruc- tion vs

    Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruc- tion vs. generation: Taming optimization dilemma in latent diffusion models. In CVPR, 2025. 1

    2025

  47. [55]

    Wave-vit: Unifying wavelet and transformers for visual representation learning

    Ting Yao, Yingwei Pan, Yehao Li, Chong-Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In ECCV, 2022. 2

    2022

  48. [56]

    Frequency autoregressive image generation with continuous tokens, 2025

    Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, and Feng Zhao. Frequency autoregressive image generation with continuous tokens, 2025. 2

    2025

  49. [57]

    Vector-quantized image modeling with improved VQGAN

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In ICLR, 2022. 1, 2, 3, 4, 5, 6, 7

    2022

  50. [58]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content...

    2022

  51. [59]

    Language model beats diffusion - tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to vi...

    2024

  52. [60]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. In NeurIPS,

  53. [61]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 2

    2018

  54. [62]

    small” encoder and “small

    Zhenhai Zhu and Radu Soricut. Wavelet-based image tok- enizer for vision transformers, 2024. 2 Spectral Image Tokenizer Supplementary Material A. Ablation study We conduct an ablation study to evaluate the effects of our design decisions. Starting from the model denoted “SIT- ...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.