Pith. sign in

REVIEW 4 major objections 5 minor 72 references

Switching codebook routing from Euclidean distance to cosine similarity prevents codebook collapse and scales visual tokenizers to 131,072 entries with 0.42 rFID.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:38 UTC pith:Y6YDWXYE

load-bearing objection HSQ is a genuinely effective recipe for scaling discrete tokenizers, but the paper's claim that it preserves magnitude is not actually demonstrated, and one key ablation is missing. the 4 major comments →

arxiv 2607.22148 v1 pith:Y6YDWXYE submitted 2026-07-24 cs.CV cs.AI

dRAE: Representation Autoencoder with Hyper-Spherical Codes

Tianren Ma , Lin Long , Chuyan Chen , Mu Zhang , Junbo Zhao , Tong Zhang , Qixiang Ye This is my paper
classification cs.CV cs.AI
keywords hyper-spherical quantizationvector quantizationcodebook collapserepresentation autoencoderdiscrete visual tokenizationmultimodal understandingimage generationcosine similarity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that codebook collapse in high-dimensional visual quantization is a metric problem, not a capacity problem: Euclidean distance lets feature magnitude dominate assignment, so high-norm codes hijack the codebook. Hyper-Spherical Quantization (HSQ) assigns each feature to the codebook entry with the highest cosine similarity—routing by semantic direction—while keeping a Euclidean commitment loss so the decoder still receives magnitude. The authors report that this decoupling lets the vocabulary scale to 131,072 entries with over 90% utilization and 0.42 reconstruction FID, where a Euclidean baseline plateaus at 16K. If true, this means discrete tokenizers built on semantic representations can scale without collapse, a step toward unified models that understand and generate images.

Core claim

Stated on the paper's own terms: the root cause of collapse is the mismatch between the Euclidean codebook objective and the anisotropic geometry of pretrained representation space. Representation features concentrate on a thin spherical shell—semantics live in direction, while magnitude carries structural detail needed for pixels. HSQ decouples these by using cosine routing for assignment, a spherical codebook loss, and an ℓ2 commitment loss that preserves magnitude for the decoder. The result is a codebook with stable magnitude scale and angular uniformity, which resists collapse and keeps improving as vocabulary grows to 131,072 (0.42 rFID, >90% utilization).

What carries the argument

Hyper-Spherical Quantization (HSQ): for each feature z, the assigned code is argmax over cosine similarity sim(z, c_i), the codebook loss is 1 − cosine similarity between normalized z and the normalized retrieved code (with stop-gradient), and the commitment loss stays Euclidean ‖z − sg[z_q]‖². The routing operates on direction; the commitment keeps magnitude. This split is what prevents scale-based hijacking while retaining reconstruction information.

Load-bearing premise

The whole method leans on the claim that semantic understanding survives near-complete loss of magnitude information—so if any encoder or task encodes meaning in feature length, HSQ's angular routing silently throws that meaning away.

What would settle it

Reproduce the diagnostic in Table 1a with a different vision encoder family (for instance, a masked-autoencoder or detection-oriented head): if ℓ2-normalizing its features drops an understanding benchmark by more than roughly 2%, the hyperspherical-semantics premise is violated and HSQ should not be expected to preserve semantics. Alternatively, run HSQ on an encoder whose feature norms are not concentrated on a thin shell and observe whether codebook utilization or rFID collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Vocabulary size becomes a usable scaling axis: going from 16K to 131K codes improves reconstruction (rFID from 0.69 to 0.42) instead of plateauing.
  • Codebook utilization stays above 90% without stochastic sampling or anti-collapse tricks, so the full codebook budget is actually used.
  • The quantized tokens preserve semantic fidelity: multimodal understanding benchmarks hold or improve as the codebook grows, and generation (class-conditional and text-to-image) benefits from the HSQ tokenizer.
  • The method transfers beyond pixel reconstruction to feature reconstruction and to joint image-semantic modeling with a language decoder, where it shows higher utilization and faster convergence than Euclidean VQ and stochastic-sampling quantizers.
  • The diagnostic result—ℓ2-normalizing features costs about 1% on understanding benchmarks while hurting reconstruction—supports the direction/magnitude split the quantizer exploits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the metric-mismatch explanation is general, angular routing should also prevent collapse in other high-dimensional discrete bottlenecks (audio, video, text embeddings); the paper only demonstrates vision, leaving this a testable extension.
  • The ablations show a fully spherical objective fails (rFID 12.7), so magnitude is not ignorable—only the assignment is angular. This suggests the specific split is delicate; varying the degree of normalization (e.g., only at lookup, or partial sphere projection) is a natural next experiment.
  • A testable falsification: apply HSQ to an encoder whose features are not thin-shell concentrated, or to a task where magnitude is semantically meaningful; if utilization or downstream accuracy drops, the hyperspherical assumption is the binding constraint.
  • The paper's own diagnostic uses one MLLM architecture; whether the ~1% normalization cost holds across other encoders and tasks is an open question that determines how broadly the angular-routing principle applies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. dRAE proposes Hyper-Spherical Quantization (HSQ) for discretizing high-dimensional visual features from pretrained encoders. HSQ routes each feature to the codebook entry with maximum cosine similarity and updates codebook entries with a cosine (spherical) loss, while keeping an ℓ2 commitment loss to guide feature magnitudes. The resulting tokenizer is evaluated for image reconstruction, MLLM understanding, C2I/T2I generation, and in ablations with SigLIP2 and DINOv2 encoders. The main empirical claims are that HSQ avoids codebook collapse, reaches >90% codebook utilization, improves continuously as the vocabulary scales to 131,072, and achieves rFID 0.42 while retaining semantic fidelity.

Significance. If the scaling and utilization results hold under controlled comparison, HSQ would be a useful step toward unified representation tokenizers: the hybrid metric design (angular routing + spherical codebook loss + Euclidean commitment) is simple, and the paper provides consistent ablations across two encoder families (Tables 7, 10, 11), a simplified single-stage training pipeline, and explicit correction of an SSIM evaluation bug in prior work. However, several load-bearing claims—magnitude preservation, the causal role of Euclidean assignment, and understanding superiority over VQRAE—are not yet backed by the reported experiments. The paper is thus promising but needs substantial experimental and conceptual tightening.

major comments (4)
  1. [§4.2, Eqs. (5)–(6); §4.1 Table 1b] The central claim that HSQ 'preserves essential magnitude information for the decoder' is not supported by the proposed mechanism. In Eq. (5), routing uses only normalized vectors, so the norm of z never affects code selection. The quantized vector z_q is a codebook entry, and the ℓ2 commitment loss in Eq. (3) pulls z toward sg[z_q], training the encoder to match codebook magnitudes rather than transmit input magnitudes. Since Table 1b shows that ℓ2-normalization degrades rFID from 4.62 to 9.57, a mechanism that discards input norms would be expected to lose exactly the information H2 identifies as essential. No experiment measures whether learned codebook magnitudes recover the input magnitude distribution (e.g., histogram of ||z|| vs ||c|| for routed codes, or reconstruction with an auxiliary scalar magnitude channel). Please provide such evidence or temper the claim.
  2. [§6, Table 7] The ablation isolates the combined change from (ℓ2, ℓ2, ℓ2) to (θ, θ, ℓ2) but omits the cell (ℓ2 assignment, θ codebook, ℓ2 commitment). Without this cell, one cannot determine whether the improvement comes from angular routing or from the spherical codebook update. The 'core insight' in §4.2 is that both are needed, and Figure 2 attributes collapse to Euclidean assignment; a direct test of the assignment mechanism is missing. If a Euclidean-routing + spherical-codebook variant matches line 3, the metric-mismatch explanation would be unsupported and the contribution would reduce to a codebook update regularizer. Please add this ablation, and also report utilization/active codes for all cells.
  3. [§5.3, Table 3; §A.4 Table 14] The understanding comparison is not controlled. VQRAE is evaluated with its tuned encoder at 512 resolution, while dRAE uses an encoder optimized for reconstruction; the text states the two methods follow different tuning protocols. The controlled variant in Table 14 uses only LLaVA-1.5 data and Vicuna-7B but omits VQRAE and dRAE at 512. Thus the claim that dRAE 'maximally preserved semantic knowledge' compared with VQRAE is not established under identical encoder-tuning, resolution, data, and LLM settings. Please report a matched comparison (same data, same resolution, same backbone) with VQRAE included.
  4. [Fig. 1c and §4.2] The diagnostic for the root cause is partly circular. HSQ by construction optimizes codebook entries with a cosine loss (Eq. 6), which spreads embeddings over the sphere; showing in Fig. 1c that HSQ's codebook is more angularly uniform than VQ's therefore does not independently confirm that angular uniformity is the causal factor preventing collapse. The paper should separate the descriptive claim (HSQ yields uniform codes) from the causal claim (Euclidean metric causes collapse) by, e.g., adding a codebook-only angular diversity regularizer to VQ and measuring whether collapse disappears. This does not invalidate the method, but it weakens the 'root cause' narrative in the abstract and Section 1.
minor comments (5)
  1. [Abstract vs §5.2] The abstract claims '100% codebook utilization', but §5.2 reports '>90% global code utilization'. Please reconcile these numbers.
  2. [Table 12] The tokenizer hyperparameter table omits β (commitment loss weight), which is defined in Eq. (3) and used in all experiments. It also omits the entropy regularization temperature τ introduced in §7. Please include all loss hyperparameters.
  3. [Algorithm 2 and Eq. (3)] Algorithm 2 uses a learned projection W on the codebook for routing and lookup (Z_q ← Ĉ[I]), while the preliminary formulation in Eqs. (2)–(3) does not mention projection. Clarify whether W applies only to the codebook or also to the features, and whether the commitment loss is computed against the projected or unprojected codebook entry.
  4. [Table 2 and §A.6] The corrected SSIM values for UniLIP and VQRAE are reported based on the authors' re-evaluation. To make this reproducible, please provide the exact checkpoints/evaluation code used for the correction and the raw (uncorrected) values.
  5. [Global] Minor typos and formatting issues: 'anti-collasping' in §8; 'Eucildean' in Table 7 header; 'V ector' in §3 header; 'fine-rained' in §2; Figure 3 axis labels '214' should read '2^14', etc. Fig. 1 uses 'Sim(·)' before Eq. (5) defines it; define the notation in the caption.

Circularity Check

0 steps flagged

No significant circularity: HSQ’s central claims are tested against external benchmarks, and the only author-overlapping citation is non-load-bearing.

full rationale

The paper’s derivation chain is not circular in the relevant sense. The central performance claims—reconstruction fidelity (rFID/PSNR/SSIM), multimodal understanding benchmarks, and text-to-image and class-to-image generation metrics—are all measured on external data and are not derived from the method’s own equations. Equations (5) and (6) define angular routing and a cosine codebook loss, so it is unsurprising that the learned codebook exhibits angular alignment; Fig. 1c is an illustration of the designed objective, not an independent prediction used to prove the root-cause story. The root-cause claim is instead supported by ablations (Table 7) that vary routing and loss metrics and by scaling experiments (Fig. 3, Tables 2, 5, 8). The H1/H2 diagnostic is an empirical premise, not a fitted input: Table 1 compares raw versus ℓ2-normalized features on external understanding and reconstruction tasks. No parameter is fitted to a hidden target and then renamed as a prediction, and no uniqueness theorem or load-bearing result is imported from the authors’ own prior work. The only author-overlapping citation is [35], used for a discrete prediction head in the generative pipeline; it is not load-bearing for HSQ. The skeptic’s proposed missing ablation (Euclidean routing with spherical codebook loss and ℓ2 commitment) would sharpen the causal attribution, but its absence is an experimental gap, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on the thin-shell geometry assumption, the direction-carries-semantics assumption, and the assumption that baseline numbers are trustworthy. No new physical entities are introduced; HSQ is a training objective and routing rule. Four hand-set loss weights (β, λ, ωp/ωd, τ) are load-bearing for the reported numbers.

free parameters (4)
  • β (commitment loss weight) = not reported
    Needed in Algorithm 2 line 7; Table 7 shows the l2 commitment is critical (full-spherical variant fails with rFID 12.7), yet β is absent from Table 12.
  • λ (distillation weight) = 1.0
    Table 12 sets λ=1.0; distillation to teacher T is part of the pipeline and no ablation isolates HSQ from this semantic-preservation mechanism.
  • ωp / ωd (perceptual and discriminator loss weights) = 1.0 / 0.1
    Eq. 1 and Table 12; reconstruction quality in Table 2 depends on these auxiliary losses, which are fixed by hand and are not part of HSQ.
  • τ (entropy regularization temperature) = 0.01
    Used in Sec. 7 joint-modeling experiments; essential for codebook utilization when an LLM is the decoder, and added to all methods for fairness.
axioms (5)
  • domain assumption VFM features concentrate on a thin spherical shell and encode semantics in direction plus radius.
    Invoked in Sec. 4.1 to justify cosine routing; supported by refs [57,25,26,58] and the paper's own Tab. 1b, not by direct geometry measurement.
  • domain assumption Direction alone preserves semantic understanding performance (H1).
    Tab. 1a shows ~1% gap on three benchmarks, but this is one architecture and three tasks; the whole Eq. 5 assignment assumes it.
  • domain assumption Magnitude is necessary for reconstruction but can be left to the decoder/commitment loss (H2).
    Tab. 1b supports the first half; the second half is an architectural bet that the Euclidean commitment loss in Algorithm 2 is enough.
  • domain assumption Reported baseline numbers, including corrected SSIM for UniLIP/VQRAE, are accurate.
    A.6 states a ~10% SSIM bug in previous implementations; no corrected evaluation code is released, so the adjusted baseline values in Table 2 cannot be independently checked.
  • domain assumption Baselines in Table 3 are comparable despite different encoder tuning and data mixtures.
    The paper notes VQRAE is trained with original encoder but tested with tuned one; dRAE uses tuned encoder and extra 1M OneVision pairs, so the understanding comparison assumes these differences do not bias the result.

pith-pipeline@v1.3.0-alltime-deepseek · 16495 in / 14972 out tokens · 151749 ms · 2026-08-01T05:38:30.691719+00:00 · methodology

0 comments
read the original abstract

In this work, we aim to discretize the high-dimensional visual representations to bridge the gap with language models - a non-trivial challenge, as existing quantization methods suffer from codebook collapse, failing to scale while preserving semantic coherence. We identify the root cause as metric mismatch: standard Euclidean codebook objectives are fundamentally misaligned with the anisotropic geometry of representation space, leading to codebook embeddings with high-variance magnitude scales and uneven angular distributions that hinder scalability. To address this, we propose Hyper-Spherical Quantization (HSQ), which decouples semantic content from feature magnitude via angular routing, preventing code assignment from being dominated by scale rather than meaning. The resulting discrete Representation Autoencoder (dRAE) achieves high-fidelity reconstruction while preserving semantic integrity and supporting scalable codebook budget. Extensive experiments demonstrate consistent performance gains as the vocabulary size scales to 131{,}072, along with 100\% codebook utilization, simplified training pipeline, and strong performance across understanding and generation tasks.

Figures

Figures reproduced from arXiv: 2607.22148 by Chuyan Chen, Junbo Zhao, Lin Long, Mu Zhang, Qixiang Ye, Tianren Ma, Tong Zhang.

Figure 1
Figure 1. Figure 1: Illustration of the code space defined by VQ ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Left: Training curve under identical set [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Scaling behavior with K = 214 ∼ 2 17 vocabulary size. From left to right: active code per batch, reconstruction PSNR, and rFID. 5 Experiment 5.1 Setups We provide a brief overview of the implementation in this section. Full experimental details and hyperparameters are included in Sec. A. Throughout the experimental section, VQ denotes vector quantization trained with the conventional ℓ2 objective. Unless s… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of quantization methods under the joint image-semantic optimization. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A demonstration of optimizing the codebook embeddings, following VQ-VAE [56]. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 22 linked inside Pith

  1. [1]

    Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture.arXiv preprint arXiv:2301.08243, 2023

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture.arXiv preprint arXiv:2301.08243, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis

    Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. InICLR, 2025

    2025

  3. [3]

    Qwen2.5-VL Technical Report, February 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report, F...

    2025

  4. [4]

    Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density.arXiv preprint arXiv:2510.05949, 2025

    Randall Balestriero, Nicolas Ballas, Mike Rabbat, and Yann LeCun. Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density.arXiv preprint arXiv:2510.05949, 2025

    arXiv 2025

  5. [5]

    InfoNCE Induces Gaussian Distribution

    Roy Betser, Eyal Gofer, Meir Yossef Levi, and Guy Gilboa. InfoNCE Induces Gaussian Distribution. In ICLR, 2026

    2026

  6. [6]

    Megalith-10m

    Ollin Boer Bohan. Megalith-10m. https://huggingface.co/datasets/madebyollin/ megalith-10m, June 2024

    2024

  7. [7]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InICCV, pages 9650–9660, 2021

    2021

  8. [8]

    Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan

    Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-To-Image Generation via Masked Generative Transformers. InICML, volume 202, pages 4055–4075, 2023

    2023

  9. [9]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Scaling instruction-finetuned language models.J

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.J. Mach. Learn. Res., 25(70):1–53, 2024

    2024

  11. [11]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, pages 248–255, 2009

    2009

  12. [12]

    Kelix Technical Report.arXiv preprint arXiv:2602.09843, 2026

    Boyang Ding, Chenglong Chu, Dunju Zang, Han Li, Jiangxia Cao, Kun Gai, Muhao Wei, Ruiming Tang, Shiyao Wang, Siyang Mao, Xinchen Luo, Yahui Liu, Zhixin Ling, Zhuoran Yang, Ziming Li, Chengru Song, Guorui Zhou, Guowang Zhang, Hao Peng, Hao Wang, Jiaxin Deng, Jin Ouyang, Jinghao Zhang, Lejian Ren, Qianqian Wang, Qigen Hu, Tao Wang, Xingmei Wang, Yiping Yang...

    arXiv 2026

  13. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021

    2021

  14. [14]

    Vqrae: Representation quantization autoencoders for multimodal understanding, generation and reconstruction.arXiv preprint arXiv:2511.23386, 2025

    Sinan Du, Jiahao Guo, Bo Li, Shuhao Cui, Zhengzhuo Xu, Yifu Luo, Yongxian Wei, Kun Gai, Xinggang Wang, Kai Wu, et al. Vqrae: Representation quantization autoencoders for multimodal understanding, generation and reconstruction.arXiv preprint arXiv:2511.23386, 2025

    arXiv 2025

  15. [15]

    Taming Transformers for High-Resolution Image Synthesis

    Patrick Esser, Robin Rombach, and Björn Ommer. Taming Transformers for High-Resolution Image Synthesis. InCVPR, pages 12873–12883, 2021

    2021

  16. [16]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023. 10

    Pith/arXiv arXiv 2023

  17. [17]

    X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again.arXiv preprint arXiv 2507.22058, 2025

    Zigang Geng, Yibing Wang, Yeyao Ma, Chen Li, Yongming Rao, Shuyang Gu, Zhao Zhong, Qinglin Lu, Han Hu, Xiaosong Zhang, Linus, Di Wang, and Jie Jiang. X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again.arXiv preprint arXiv 2507.22058, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Geneval: An object-focused framework for evaluating text-to-image alignment.NeurIPS, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.NeurIPS, 36:52132–52152, 2023

    2023

  19. [19]

    Vision as a dialect: Unifying visual understanding and generation via text-aligned representations

    Jiaming Han, Hao Chen, Yang Zhao, Hanyu Wang, Qi Zhao, Ziyan Yang, Hao He, Xiangyu Yue, and Lu Jiang. Vision as a dialect: Unifying visual understanding and generation via text-aligned representations. arXiv preprint arXiv:2506.18898, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 15979–15988, 2022

    2022

  21. [21]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InNeurIPS, pages 6626–6637, 2017

    2017

  22. [22]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InCVPR, pages 6700–6709, 2019

    2019

  24. [24]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. InICLR, 2014

    2014

  25. [25]

    Number 89

    Michel Ledoux.The concentration of measure phenomenon. Number 89. American Mathematical Soc., 2001

    2001

  26. [26]

    The double-ellipsoid geometry of CLIP

    Meir Yossef Levi and Guy Gilboa. The double-ellipsoid geometry of CLIP. InICML, volume 267, pages 33999–34019, 2025

    2025

  27. [27]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Toklip: Marry visual tokens to CLIP for multimodal comprehension and generation.CoRR, abs/2505.05422, 2025

    Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to CLIP for multimodal comprehension and generation.CoRR, abs/2505.05422, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. InECCV, pages 366–384. Springer, 2024

    2024

  32. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26296–26306, 2024

    2024

  33. [33]

    Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233. Springer, 2024

    2024

  34. [34]

    Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321, 2025

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321, 2025

    arXiv 2025

  35. [35]

    ReDDiT: Rehashing Noise for Discrete Visual Generation

    Tianren Ma, Xiaosong Zhang, Boyu Yang, Junlan Feng, and Qixiang Ye. ReDDiT: Rehashing Noise for Discrete Visual Generation. InICLR, 2026

    2026

  36. [36]

    Finite scalar quantization: VQ-V AE made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-V AE made simple. InICLR, 2024. 11

    2024

  37. [37]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick L...

    2024

  38. [38]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, pages 4172–4182, 2023

    2023

  39. [39]

    Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification.arXiv preprint arXiv:2606.18249, 2026

    Wujian Peng, Lingchen Meng, Yuxuan Cai, Xianwei Zhuang, Yuhuan Yang, Rongyao Fang, Chenfei Wu, Junyang Lin, Zuxuan Wu, and Shuai Bai. Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification.arXiv preprint arXiv:2606.18249, 2026

    Pith/arXiv arXiv 2026

  40. [40]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In CVPR, pages 2545–2555, 2025

    2025

  41. [41]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026

    2026

  42. [42]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763, 2021

    2021

  43. [43]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, pages 10674–10685, 2022

    2022

  44. [44]

    Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T

    Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and V olodymyr Kuleshov. Simple and Effective Masked Diffusion Language Models. In NeurIPS, 2024

    2024

  45. [45]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. InNeurIPS, pages 2226–2234, 2017

    2017

  46. [46]

    Scalable Image Tokenization with Index Backpropagation Quantization.arXiv preprint arXiv:2412.02692, 2025

    Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Scalable Image Tokenization with Index Backpropagation Quantization.arXiv preprint arXiv:2412.02692, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InCVPR, pages 8317–8326, 2019

    2019

  48. [48]

    What matters for Representation Alignment: Global Information or Spatial Structure?arXiv preprint arXiv:2512.10794, 2025

    Jaskirat Singh, Xingjian Leng, Zongze Wu, Liang Zheng, Richard Zhang, Eli Shechtman, and Saining Xie. What matters for Representation Alignment: Global Information or Spatial Structure?arXiv preprint arXiv:2512.10794, 2025

    arXiv 2025

  49. [49]

    Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324, 2026

    Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang, Eli Shechtman, and Saining Xie. Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324, 2026

    Pith/arXiv arXiv 2026

  50. [50]

    Autore- gressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Unilip: Adapting clip for unified multimodal understanding, generation and editing.arXiv preprint arXiv:2507.23278, 2025

    Hao Tang, Chenwei Xie, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. Unilip: Adapting clip for unified multimodal understanding, generation and editing.arXiv preprint arXiv:2507.23278, 2025

    arXiv 2025

  52. [52]

    LongCat-Next: Lexicalizing Modalities as Discrete Tokens.arXiv preprint arXiv:2603.27538, 2026

    Meituan LongCat Team. LongCat-Next: Lexicalizing Modalities as Discrete Tokens.arXiv preprint arXiv:2603.27538, 2026

    arXiv 2026

  53. [53]

    Zettlemoyer, Koustuv Sinha, Yann LeCun, and Saining Xie

    Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Th’eophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke S. Zettlemoyer, Koustuv Sinha, Yann LeCun, and Saining Xie. Beyond language modeling: An exploration of mult...

    arXiv 2026

  54. [54]

    Scaling text-to-image diffusion transformers with representation autoencoders.arXiv preprint arXiv:2601.16208, 2026

    Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, and Saining Xie. Scaling text-to-image diffusion transformers with representation autoencoders.arXiv preprint arXiv:2601.16208, 2026. 12

    arXiv 2026

  55. [55]

    Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J

    Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J. Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense featur...

    Pith/arXiv arXiv 2025

  56. [56]

    Neural discrete representation learning.NeurIPS, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.NeurIPS, 30, 2017

    2017

  57. [57]

    Cambridge Series in Statistical and Probabilistic Mathematics

    Roman Vershynin.High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018

    2018

  58. [58]

    Understanding contrastive representation learning through alignment and uniformity on the hypersphere

    Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. InICML, volume 119, pages 9929–9939, 2020

    2020

  59. [59]

    Representation Forcing for Bottleneck-Free Unified Multimodal Models.arXiv preprint arXiv:2605.31604, 2026

    Yuqing Wang, Zhijie Lin, Ceyuan Yang, Yang Zhao, Fei Xiao, Hao He, Qi Zhao, Zihan Ding, Fuyun Wang, Shuai Wang, Youliang Zhang, Haoqi Fan, and Xihui Liu. Representation Forcing for Bottleneck-Free Unified Multimodal Models.arXiv preprint arXiv:2605.31604, 2026

    Pith/arXiv arXiv 2026

  60. [60]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Pith/arXiv arXiv 2024

  61. [61]

    VILA-U: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-U: a unified foundation model integrating visual understanding and generation. InICLR, 2025

    2025

  62. [62]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    Muse-vl: Modeling unified vlm through semantic discrete encoding

    Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vlm through semantic discrete encoding. InICCV, pages 24135–24146, 2025

    2025

  64. [64]

    Towards scalable pre-training of visual tokenizers for generation.arXiv preprint arXiv:2512.13687, 2025

    Jingfeng Yao, Yuda Song, Yucong Zhou, and Xinggang Wang. Towards scalable pre-training of visual tokenizers for generation.arXiv preprint arXiv:2512.13687, 2025

    arXiv 2025

  65. [65]

    Reconstruction vs

    Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. InCVPR, pages 15703–15712, 2025

    2025

  66. [66]

    Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G

    Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language Model Beats Diffusion – Tokenizer is Key to Visual Generation.arXiv preprint arXiv:2310.05737, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.arXiv preprint arXiv:1801.03924, 2018

    Pith/arXiv arXiv 2018

  68. [68]

    Spherical leech quantization for visual tokenization and generation.CoRR, abs/2512.14697, 2025

    Yue Zhao, Hanwen Jiang, Zhenlin Xu, Chutong Yang, Ehsan Adeli, and Philipp Krähenbühl. Spherical leech quantization for visual tokenization and generation.CoRR, abs/2512.14697, 2025

    arXiv 2025

  69. [69]

    Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation.arXiv preprint arXiv:2502.05178, 2025

    Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and De-An Huang. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation.arXiv preprint arXiv:2502.05178, 2025

    Pith/arXiv arXiv 2025

  70. [70]

    Diffusion transformers with representation autoencoders.CoRR, abs/2510.11690, 2025

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders.CoRR, abs/2510.11690, 2025

    Pith/arXiv arXiv 2025

  71. [71]

    Addressing representation collapse in vector quantized models with one linear layer

    Yongxin Zhu, Bocheng Li, Yifei Xin, Zhihua Xia, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. InICCV, pages 22968–22977, 2025

    2025

  72. [72]

    Advancing aesthetic image generation via composition transfer.Int

    Kai Zou, Zhiwei Zhao, Bin Liu, and Nenghai Yu. Advancing aesthetic image generation via composition transfer.Int. J. Comput. Vis., 134:252, 2026. 13 Enc. … Dec. 𝐶𝐶 𝑖𝑖 𝐼𝐼𝑖𝑖 Retrieve ̂𝐶𝐶 𝐼𝐼 , where 𝐼𝐼𝑖𝑖 ← argmax𝑗𝑗 𝑍𝑍𝑖𝑖 𝑍𝑍𝑖𝑖 2 � 𝐶𝐶𝑗𝑗 C𝑗𝑗 2 ℒcodebook ← 1 − sg[ 𝑍𝑍 𝑍𝑍 2 ] � 𝑍𝑍𝑞𝑞 𝑍𝑍𝑞𝑞 2 𝑍𝑍𝑞𝑞𝑍𝑍 ℒcommit ← 𝑍𝑍 − sg[𝑍𝑍𝑞𝑞] 2 2 Continuous latents Quantized latents Code...

    2026