{"id":"0945fd05-9a72-412f-b6b7-d1221e876e6b","arxiv_id":"2412.09607","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A spectral image tokenizer based on Haar DWT coefficients produces coarse-to-fine token sequences that enable multiscale autoregressive image generation, upsampling, and editing.","lead":"This paper introduces a discrete-wavelet-based image tokenizer that represents images as coarse-to-fine token sequences. It enables autoregressive models to generate, upsample, and edit images at multiple resolutions without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never isolates DWT patchification from the ADTransformer/dual-codebook changes, so the central claim that spectral input drives the reported gains is not yet empirically supported.","rationale":"The reader's weakest_assumption concerns the scale-causal attention mask and whether low-scale tokens suffice for high-scale prediction. That is a real and partially supported trade-off, but it is acknowledged in the paper ('scale-causal attention ... slightly reduces performance') and the applications still demonstrate the capability. The more load-bearing gap, in my reading, is that the paper's central comparative claim—that operating on the spectrum instead of spatial patches is what improves reconstruction and generation—is never isolated from the ADTransformer and dual-codebook modifications. The method is called the Spectral Image Tokenizer and the paper states that the crucial difference from prior multiscale latent approaches is operating on spectral coefficients, yet every head-to-head comparison varies two or more design axes at once. The absence of a spatial-control ablation leaves open the possibility that the empirical gains come from the extra per-coefficient-type parameters or the dual codebooks, not from the DWT input. This does not invalidate the method's useful multiscale properties (P.2, P.4, P.5, P.6), which are structurally enabled by the DWT, but it does mean the strongest interpretive claim is not yet established. The paper has strengths: the multiscale reconstruction results at native low resolutions are compelling, the scale-causal mask enables genuinely novel applications, and the ablations in Table 5 cover several design factors. Missing code and error bars also contribute to the conditional verdict. I therefore recommend keeping the verdict CONDITIONAL, with the explicit additional condition that the authors provide the spatial-control and architecture-control ablations described above, or clearly soften the claim to a multiscale-tokenizer contribution.","tokens_in":16058,"tokens_out":11235,"duration_ms":123888,"concrete_test":"Train SIT-4 under the exact Table 5 protocol in two additional conditions: (A) keep DWT patchification but remove the ADTransformer's specialized parameters and the dual codebooks, reverting to shared transformer parameters and a single 8192-entry codebook; (B) keep ADTransformer and dual codebooks but replace DWT coefficients with a spatial multiscale representation (e.g., a 4-scale Gaussian/pixel pyramid) using the same per-scale token counts and patch sizes as SIT-4. Compare LPIPS, PSNR, FID, and class-conditional FID against the dense SIT-4 row. If (A) reproduces the SIT-4 numbers, the spectral representation, not the auxiliary architecture changes, is the driver; if (B) matches or beats SIT-4, the reported gains cannot be uniquely attributed to DWT coefficients, and the central claim should be recast as a multiscale-tokenizer contribution rather than a specifically spectral one.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that tokenizing DWT coefficients rather than spatial patches yields a coarse-to-fine representation that improves reconstruction and generation. The empirical support for this claim is weakened by a missing control: relative to the ViT-VQGAN baseline, SIT changes not only the input representation but also the transformer internals (ADTransformer with separate parameters for approximation vs. details) and the quantizer (two codebooks instead of one). Tables 1 and 3 compare the combined system against an architecture that differs on both axes, so the reconstruction improvements (e.g., LPIPS 0.135 for SIT-5 vs. 0.167 for reproduced ViT-VQGAN at 256x256) and the class-conditional FID gain (6.95 vs. 8.37) cannot be attributed to the spectral input per se. The paper explicitly frames the 'crucial difference' from prior work as operating on spectral coefficients rather than latent features, but no ablation removes the ADTransformer/dual-codebook components while retaining DWT, nor replaces DWT coefficients with a spatial multiscale input while retaining those components. This is an omitted control, not an internal inconsistency. Separately, Table 5 shows scale-causal attention costs 0.143 to 0.184 LPIPS and 1.20 to 1.97 FID at 256, which is an acknowledged trade-off; the attribution gap is more fundamental because it concerns what the method's contribution actually is.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Spectral Image Tokenizer (SIT), a ViT-VQGAN-style image tokenizer that operates on Haar discrete wavelet transform (DWT) coefficients instead of spatial pixel patches. The DWT output is organized into scales, an approximation scale followed by detail scales, and each scale is patchified with scale-specific linear embeddings, so higher-frequency scales are represented by larger patches and hence compressed more. The tokenizer transformer can use a scale-causal attention mask and separate parameters and codebooks for approximation versus detail coefficients (ADTransformer). The resulting token sequence is coarse-to-fine. The paper then trains an autoregressive transformer (AR-SIT) over SIT tokens and evaluates it for text-to-image generation, partial decoding, text-guided upsampling, text-guided editing, and class-conditional generation. Experiments report reconstruction metrics on ImageNet at multiple resolutions, text-to-image metrics on MS-COCO, and class-conditional FID on ImageNet, together with ablations over scales, sequence length, vocabulary size, wavelet family, and scale-causal attention. The authors state explicitly that text-to-image FID is similar to the Parti baseline and that the scale-causal mask reduces reconstruction accuracy.","tokens_in":16363,"tokens_out":6695,"duration_ms":60353,"significance":"If the central claim holds, SIT is a useful step toward making autoregressive image generation genuinely multiscale: a single tokenizer can handle multiple resolutions without retraining, supports partial decoding, and enables text-guided upsampling and editing. The paper is commendably transparent: it reproduces the ViT-VQGAN baseline, reports throughput and memory, includes ablations, and states its limitations (small AR models, similar text-to-image metrics, scale-causal tradeoff). The main weakness is attribution: SIT differs from ViT-VQGAN not only in the input representation but also in the transformer internals and quantizer, so the reported gains are not yet isolated to the spectral representation. With the missing ablation supplied, the paper would provide solid evidence for its claims.","major_comments":[{"comment":"The paper's central claim, stated in §1 and repeated in §2, is that tokenizing DWT coefficients rather than spatial patches is the 'crucial difference' from prior work. However, Tables 1, 3, and 5 compare SIT against ViT-VQGAN under a combined change: SIT uses DWT inputs, the ADTransformer with separate approximation/detail parameters, and two separate codebooks. No experiment isolates the input representation by ablating these components independently. For example, one could keep the ADTransformer and dual codebooks but feed a spatial multiscale representation (e.g., a Laplacian pyramid or downsampled image plus residuals), or keep the DWT input but revert to a shared transformer and single codebook. Without such a control, the reconstruction improvements (e.g., LPIPS 0.135 vs 0.167 at 256x256 in Table 1) and the class-conditional FID gain (6.95 vs 8.37 in Table 3) cannot be attributed specifically to the spectral input. This is an omitted control rather than an internal inconsistency, but it is load-bearing for the paper's stated contribution.","section":"§4.1, Tables 1, 3, 5"},{"comment":"Table 5 shows that scale-causal attention is not free: for SIT-4, enabling both scale-causal encoder and decoder increases LPIPS from 0.143 to 0.184 and FID from 1.20 to 1.97 at 256x256, and enabling only the scale-causal decoder still increases LPIPS to 0.166 and FID to 1.45. Since properties P.2 and P.4-P.6 all depend on scale-causal masking, the paper should quantify the benefit side of this tradeoff with the same metrics. I would like to see, at minimum, reconstruction quality and FID as a function of the number of decoded scales for partial decoding, and a statement of how the full-resolution loss weighs against the multiscale capabilities. Without this quantification, the claim that the scale-causal attention enables different applications is supported only qualitatively.","section":"Table 5, Section 4.1"},{"comment":"Most reported numbers are single-run and no error bars or significance tests are given. This matters because several headline comparisons are close (AR-SIT FID 12.6 vs Parti 12.4 in Table 2) and because the 512x512 ViT-VQGAN row in Table 1 is explicitly selected as the best checkpoint before divergence. The paper should state the number of seeds or runs used, report means and standard deviations (or an equivalent dispersion measure), and describe the checkpoint-selection protocol for the unstable baseline. This is standard practice and would materially strengthen the empirical claims in §5.1 and §5.2.","section":"Tables 1, 2, 3; Section 5.1"}],"minor_comments":[{"comment":"The word 'realtively' in the conclusion should be 'relatively'.","section":"Section 6"},{"comment":"The SCE and SCD columns use both 'X' and '✓' without a legend; please state explicitly that '✓' means the scale-causal mask is enabled.","section":"Table 5"},{"comment":"Reference [37] appears twice in the citation list [15, 37, 37, 51, 57]; the duplicate should be removed.","section":"Section 1"},{"comment":"Equation (6) writes the categorical distribution over ⌊q^n_s⌋, but the notation ⌊q^n_s⌋ is introduced just before as 'its discrete position in the codebook'; consider using a distinct symbol such as k^n_s to avoid confusion with an integer-part or floor operation.","section":"Equation (6), Section 4.2"},{"comment":"The main text reports an FID of 6.2 for 32x32 upsampling, while the supplemental Table 4 lists 6.19; please state which number is used and confirm that the evaluation subset and ground-truth protocol are the same as in Table 2.","section":"Section 5.3 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle to acceptance is the attribution gap in the empirical section. The authors should be asked to add an ablation that holds the ADTransformer and dual-codebook design fixed and varies only the input representation (DWT versus a spatial multiscale input). If that ablation confirms that the spectral input matters, the paper would be a solid accept after revisions. I would also encourage the authors to report variance and to clarify the 512x512 baseline selection protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces SIT, a discrete wavelet transform (DWT) based image tokenizer with scale-causal attention and separate codebooks for approximation and detail coefficients. It's a well-executed empirical paper. The core idea is clever: by patchifying DWT coefficients rather than spatial patches, the token sequence becomes coarse-to-fine, which naturally supports partial decoding, multiscale reconstruction, text-guided upsampling, and editing. The experiments substantiate these capabilities. Reconstruction metrics improve over ViT-VQGAN (LPIPS 0.135 vs 0.167 at 256), class-conditional FID improves (6.95 vs 8.37), and the model handles variable resolutions without retraining, with substantial speedups at low resolutions. The ablations are honest—they show wavelet family choice (Haar wins), and they quantify the cost of scale-causal attention (LPIPS goes from 0.143 to 0.184 when both encoder and decoder use it). That trade-off is acknowledged and is reasonable given the multiscale features it buys.\n\nThe main soft spot is attribution. When comparing SIT to ViT-VQGAN, the paper changes more than just the input representation: it also adds ADTransformer (different parameters for approximation vs details) and uses two codebooks instead of one. There is no ablation that isolates the effect of the DWT input from the architectural changes. So the statement that the main contribution is a tokenizer operating on the image spectrum is not fully supported by the evidence. The gains could be due to the transformer/quantizer changes. This is an omitted control rather than an internal error, and it's fixable: run a DWT input with the standard transformer/codebook, or run a spatial multiscale input with ADTransformer. The paper would be much stronger with that experiment.\n\nOther soft spots are more minor. Text-to-image FID is similar to Parti, not better; the editing section has no quantitative evaluation, only qualitative examples; the 512x512 baseline comparison used a checkpoint selected before training diverged, which makes that table weak; there are no error bars or code release. These are typical for this venue but worth noting.\n\nOverall, I think the central idea is novel and the multiscale applications are a genuine contribution. The paper is not circular; it tests an architecture against baselines and ablates its key choices. I'd send it to peer review, but I'd ask for the isolating ablation and for quantitative editing results. It's a conditional accept, not a clear accept, but it deserves serious referee time.","headline":"A genuinely useful tokenizer with a real attribution gap: the gains over ViT-VQGAN aren't isolated to the spectral input, but the multiscale applications are novel and worth refereeing.","tokens_in":16866,"tokens_out":2875,"would_cite":true,"duration_ms":26182,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Image tokens ordered by wavelet scale enable coarse-to-fine generation.","keywords":["image tokenization","discrete wavelet transform","autoregressive generation","coarse-to-fine sequence","vector quantization","scale-causal attention","multiscale reconstruction","text-guided upsampling"],"falsifier":"Train a SIT with scale-causal attention on both encoder and decoder at 256×256, then decode only the first-scale tokens into a full-resolution image and measure LPIPS and PSNR against ground truth. If the coarse reconstruction from the first scale alone is not a recognizable image, the claim that the representation is truly coarse-to-fine and decodable at partial scales would be falsified. The paper's own ablation, which shows LPIPS degrading from 0.143 to 0.184 when both masks are enabled, indicates the direction of the risk.","tokens_in":15870,"feed_emoji":"🖼️","tokens_out":6060,"duration_ms":52559,"temperature":0.7,"pith_summary":"This paper proposes replacing the usual raster-scan tokenization of image patches with a tokenizer that operates on discrete wavelet transform (DWT) coefficients, so the token sequence naturally runs from a coarse approximation of the whole image to finer detail. The claim is that this ordering is better suited to autoregressive transformers: the model predicts the next scale while conditioning on a coarse reconstruction of the full image, rather than on a partial row-by-row reconstruction. The authors show the tokenizer reconstructs images at multiple resolutions with a single training, that generation can be stopped early to yield a coarse image, and that the same machinery supports text-guided upsampling and editing. If correct, the paper provides a spectral, multiscale tokenizer that brings autoregressive image generation closer to the sequential structure of language models.","feed_headline":"Image tokens ordered by wavelet scale enable coarse-to-fine generation.","feed_subtitle":"One autoregressive model can reconstruct, upsample, and edit images at multiple resolutions from a single training.","key_machinery":"The central object is DWT coefficient patchification combined with scale-causal attention. Each wavelet scale becomes one block of tokens: the approximation coefficients form the first scale, and horizontal, vertical, and diagonal detail coefficients are concatenated spatially so every scale has the same token count. The scale-causal attention mask, applied to the encoder or decoder or both, is what endows the representation with its multiscale behavior, since it forces each scale to depend only on itself and lower scales. The Approximation-Details Transformer (ADTransformer), with separate parameters and codebooks for the approximation and detail tokens, handles the fact that the two coefficient distributions are very different.","core_discovery":"The central claim is that tokenizing the spectrum rather than the pixels is a better inductive bias for autoregressive image modeling. SIT applies a Haar DWT, patchifies each scale separately with the same number of tokens per scale (so higher scales use larger patches and are compressed more), and feeds the resulting sequence through a transformer with separate parameters and codebooks for approximation versus detail coefficients. An optional scale-causal attention mask lets each scale attend only to itself and lower scales, which makes the representation multiscale without retraining. With this tokenizer, an autoregressive transformer (AR-SIT) models the coarse-to-fine sequence; the authors report improved reconstruction over the ViT-VQGAN baseline (LPIPS 0.135 versus 0.167 at 256×256 for SIT-5), improved class-conditional generation (FID 6.95 versus 8.37), and comparable text-to-image FID while adding the ability to partially decode, upsample, and edit images.","pith_inferences":["If the coarse-to-fine ordering is what matters, then other invertible multiscale transforms with energy compaction, such as DCT or learned filter banks, could yield similar gains; the paper only demonstrates DWT with Haar.","The reconstruction-quality cost of scale-causal attention suggests a possible refinement: allow high-frequency information to influence low-frequency tokens during encoding but decode with strict scale-causality, or predict detail tokens conditionally with a learned prior over the approximation coefficients.","The observation that bigger codebooks improve reconstruction but not generation hints at a compression-generation tradeoff that could be studied by varying the number of tokens per scale according to the energy of each scale.","A natural extension is to make the number of tokens per scale adaptive to image content, using more tokens where detail is actually present, which the paper does not explore."],"forward_implications":["One model trained at a single resolution can tokenize and reconstruct images at lower and higher resolutions without retraining, because the scale-causal mask makes unused scales simply absent from the sequence.","Early termination of autoregressive generation yields a coarse image after only the first tokens, so a user can preview many drafts cheaply and refine only the selected one.","Text-guided upsampling becomes a continuation problem: encode the low-resolution input into the first tokens and let the autoregressive model generate the rest.","Text-guided editing becomes a freezing problem: keep the low scales of an input image and regenerate higher scales under a new caption.","Class-conditional generation improves over the comparable raster-order baseline, supporting the claim that coarse-to-fine conditioning is better for next-token prediction."],"supporting_citations":[{"why":"Supplies the ViT-VQGAN tokenizer architecture and training protocol that SIT modifies and compares against.","marker":"[57]"},{"why":"Supplies the Parti autoregressive generation architecture and training protocol used for AR-SIT and its main fair baseline.","marker":"[58]"},{"why":"Introduced vector quantization of latents, the quantization mechanism SIT inherits.","marker":"[51]"},{"why":"Added perceptual and adversarial losses to VQ tokenizers, which SIT uses for training; also a baseline generative model.","marker":"[15]"},{"why":"Provides the wavelet transform background and the inverse DWT that SIT relies on for reconstruction.","marker":"[29]"},{"why":"Provides the compression precedent of aggressive high-frequency quantization (JPEG2000) that motivates equal token counts per scale.","marker":"[7]"},{"why":"Recent multiscale residual-latent autoregressive baseline that SIT compares against in class-conditional generation.","marker":"[48]"},{"why":"Interprets diffusion as spectral autoregression, motivating the paper's literal spectral autoregression approach.","marker":"[12]"}],"fun_headline_variants":["Image generation gets coarse-to-fine with spectral tokens","Tokenize the spectrum, not pixels, for better image generation","Wavelet-based tokens enable multiscale image generation","Spectral tokens: one model for generation, upsampling, editing","Coarse-to-fine image tokens from wavelet transforms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the low-scale approximation tokens contain enough information that all higher-scale detail tokens can be predicted or generated from them, so the scale-causal mask that blocks high-frequency information from influencing low-frequency tokens does not cripple reconstruction quality.","fun_headline_variants_meta":{"raw":{"variants":["Image generation gets coarse-to-fine with spectral tokens","Tokenize the spectrum, not pixels, for better image generation","Wavelet-based tokens enable multiscale image generation","Spectral tokens: one model for generation, upsampling, editing","Coarse-to-fine image tokens from wavelet transforms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3937,"prompt_tokens":939,"completion_tokens":2998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2916}},"tokens_in":555,"tokens_out":2998,"duration_ms":19900,"temperature":1.0,"reasoning_tokens":2916,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:52:08.366060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a SIT with scale-causal attention on both encoder and decoder at 256×256, then decode only the first-scale tokens into a full-resolution image and measure LPIPS and PSNR against ground truth. If the coarse reconstruction from the first scale alone is not a recognizable image, the claim that the representation is truly coarse-to-fine and decodable at partial scales would be falsified. The paper's own ablation, which shows LPIPS degrading from 0.143 to 0.184 when both masks are enabled, indicates the direction of the risk.","supporting_citations":[{"cited_title":"Vector-quantized image modeling with improved VQGAN","cited_arxiv_id":null,"evidence_quote":"Supplies the ViT-VQGAN tokenizer architecture and training protocol that SIT modifies and compares against."},{"cited_title":"Scaling autoregressive models for content-rich text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the Parti autoregressive generation architecture and training protocol used for AR-SIT and its main fair baseline."},{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Introduced vector quantization of latents, the quantization mechanism SIT inherits."},{"cited_title":"Taming transformers for high-resolution image synthesis","cited_arxiv_id":null,"evidence_quote":"Added perceptual and adversarial losses to VQ tokenizers, which SIT uses for training; also a baseline generative model."},{"cited_title":"A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way","cited_arxiv_id":null,"evidence_quote":"Provides the wavelet transform background and the inverse DWT that SIT relies on for reconstruction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the compression precedent of aggressive high-frequency quantization (JPEG2000) that motivates equal token counts per scale."},{"cited_title":"Visual autoregressive modeling: Scalable image generation via next-scale prediction","cited_arxiv_id":null,"evidence_quote":"Recent multiscale residual-latent autoregressive baseline that SIT compares against in class-conditional generation."},{"cited_title":"Diffusion is spectral autoregression, 2024","cited_arxiv_id":null,"evidence_quote":"Interprets diffusion as spectral autoregression, motivating the paper's literal spectral autoregression approach."}],"review_version":1}