Pith. sign in

REVIEW 4 cited by

GIVT: Generative Infinite-Vocabulary Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.02116 v4 pith:LVT4BEDV submitted 2023-12-04 cs.CV

Michael Tschannen , Cian Eastwood , Fabian Mentzer This is my paper
classification cs.CV
keywords givttransformerslatentmodelsequencesdiscretegenerationgenerative
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We introduce Generative Infinite-Vocabulary Transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT, where transformers are used to model the discrete latent sequences of a VQ-VAE, we use GIVT to model the unquantized real-valued latent sequences of a $\beta$-VAE. In class-conditional image generation GIVT outperforms VQ-GAN (and improved variants thereof) as well as MaskGIT, and achieves performance competitive with recent latent diffusion models. Finally, we obtain strong results outside of image generation when applying GIVT to panoptic segmentation and depth estimation with a VAE variant of the UViM framework.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enhancing next token prediction based pre-training for jet foundation models

    hep-ph 2025-12 conditional novelty 6.0 of 10

    Using continuous particle features as input and combining next-token with masked-token pre-training markedly improves classification accuracy of the OmniJet jet foundation model without visibly hurting its generative quality.

  2. Transition Matching: Scalable and Flexible Generative Modeling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Transition Matching unifies flow matching and continuous autoregressive generation as discrete-time Markov processes, with three variants that improve text-to-image quality and speed.

  3. DiSA: Diffusion Step Annealing in Autoregressive Image Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Autoregressive image diffusion models can use far fewer denoising steps for later tokens without losing quality, yielding 1.4-2.5x speedup from step annealing and up to 10x when combined with fewer autoregressive steps.

  4. MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    An empirical testbed shows that allocating more transformer layers to autoregressive conditioning improves FID under low inference budgets, while block size should scale with image resolution.

Pith tools