Pith. sign in

REVIEW 3 major objections 7 minor 4 cited by

CAT: Content-Adaptive Image Tokenization

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A content-adaptive image tokenizer spends fewer tokens on simple images and still improves generation quality.

desk verdict Useful tokenizer method, but the headline FID gain is confounded by more training images per step. read the letter →

arxiv 2501.03120 v1 pith:RUCVRZSJ submitted 2025-01-06 cs.CV

Junhong Shen , Kushal Tirumala , Michihiro Yasunaga , Ishan Misra , Luke Zettlemoyer , Lili Yu , Chunting Zhou This is my paper · ORCID
classification cs.CV
keywords content-adaptivetokenizationimagetokenizervariable-lengthlatentrepresentationvariationalautoencoderLLM-basedcomplexityscoringdiffusiontransformerNetgenerationtokenallocation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most image tokenizers encode every image into the same number of tokens, even though simple images need far less capacity. This paper argues that content complexity can be predicted from a text description and that using that prediction to allocate tokens improves both reconstruction and generation. To do this, CAT uses an LLM to score captions and perception-focused answers, then routes each image through a nested variational autoencoder that can compress it 8x, 16x, or 32x. On ImageNet class-conditional generation, a diffusion transformer trained on CAT latents reaches FID 4.56 versus 4.78 for a fixed-ratio tokenizer at the same training token budget, while using 23% fewer tokens per image on average and boosting inference throughput by 18.5%.

What carries the argument

The mechanism is a nested variational autoencoder with a shared middle block: intermediate encoder outputs at different downsampling depths are routed through channel-matching ResNet blocks to produce Gaussian latents of spatial size corresponding to 8x, 16x, or 32x compression in a single model. The routing decision comes from an LLM-based complexity evaluator that takes a caption plus yes/no answers about human faces and text, outputs an integer score from 1 to 9, and maps score intervals to compression ratios chosen so the training average is 16x. This design lets CAT train one tokenizer on all three ratios and later emit variable-length latents without retraining.

What would settle it

Retrain the fixed 16x DiT baseline with the same per-step image count as CAT (about 1,358 images at 197.44 tokens each, instead of 1,024 images at 256 tokens), holding total token FLOPs and steps fixed; if the FID gap to 4.56 disappears, the improvement must be credited to extra data diversity rather than to adaptive token allocation.

Watch

Extended reading notes

Core claim

The central claim is that adaptive tokenization improves both the quality and the efficiency of image modeling. The paper shows that existing complexity metrics (JPEG size, MSE, LPIPS) correlate poorly with how much an image can be compressed before quality visibly degrades: repetitive textures score as complex, while text and faces score as simple. A caption-based LLM score reverses this, and CAT trained with ratios chosen by that score keeps reconstruction quality on natural images while using fewer tokens, and improves reconstruction on faces and text-heavy images (rFID drops roughly 12% on CelebA and 39% on ChartQA relative to fixed 16x baselines). In generation, DiT-CAT achieves FID 4.56, sFID 10.55, and Inception Score 191.09 at 512x512 ImageNet, beating fixed-ratio baselines trained with the same global token batch size, and it supports controllable generation by letting the user choose how many tokens to spend.

Load-bearing premise

The comparison assumes that matching the global token batch size and step count makes CAT and the fixed-ratio baseline cost-matched, even though CAT's shorter token sequences mean it sees more images per training step.

Editorial extensions

If this is right

  • With the same global token batch size and number of training steps, DiT-CAT beats the fixed 16x baseline on FID (4.56 vs 4.78), sFID (10.55 vs 11.81), Inception Score (191.09 vs 187.47), and precision (0.75 vs 0.72).
  • Because simpler images average 216 tokens at inference instead of 256, CAT increases inference throughput by 18.5%.
  • On natural images CAT preserves reconstruction quality while using roughly 16% fewer tokens, and on perceptually sensitive images (faces, text) it improves rFID by about 12% on CelebA and 39% on ChartQA relative to fixed 16x baselines.
  • Users can control generation complexity at inference time by specifying a token count, trading detail against compute without retraining.
  • Adaptive allocation only needs the text description, so the approach carries over to text-to-image settings where the original image is unavailable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because CAT runs 23% fewer tokens per image at the same global token batch, it sees roughly 30% more images per training step; a matched-image-count control experiment would separate the benefit of adaptive allocation from the benefit of extra data diversity.
  • The caption-based complexity signal could be transferred to other modalities with high temporal redundancy, such as video, where a single LLM description might set per-clip or per-frame token budgets.
  • If the LLM scorer is the bottleneck, a smaller distilled scorer or a regression model trained on LLM scores could provide the same adaptive routing at lower cost and with more controllable thresholds.
  • The nested-VAE design suggests a natural extension to discrete tokenizers, where variable-length token sequences are already common, so adaptive ratios could be added without changing the downstream sequence model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents Content-Adaptive Tokenizer (CAT), a VAE-based image tokenizer that selects one of three compression ratios (8x, 16x, or 32x) per image using a three-stage pipeline: an InstructBlip caption and yes/no answers about faces/text are fed to Llama-3-70B, which outputs a 1–9 complexity score; thresholds (a,b) map scores to ratios. The tokenizer is a nested VAE with shared middle blocks and skip connections, trained on 380M Shutterstock images with a reconstruction loss plus KL, LPIPS/MoCo, and GAN terms. Reconstruction is evaluated on COCO, ImageNet, CelebA, and ChartQA. The authors then train DiT-XL on ImageNet using CAT latents and report 512x512 class-conditional FID of 4.56 versus 4.78 for a fixed 16x tokenizer at matched global token batch size and steps, with 18.5% higher inference throughput.

Significance. If the adaptive token allocation is genuinely responsible for the reported gains, CAT is a useful contribution: it provides a practical way to allocate representational capacity according to content complexity, enables controllable token counts at inference, and improves reconstruction on text- and face-heavy images. The paper's strengths include the clean nested-VAE design that reuses one model for several compression ratios, the external benchmark of caption complexity against existing metrics, the breadth of reconstruction datasets (including ChartQA), and the explicit reporting of training configurations. The main caveat is that the headline generation comparison does not cleanly separate the effect of adaptivity from the effect of seeing more images per training step, so the FID gain could be explained by data diversity rather than by the adaptive allocation itself.

major comments (3)
  1. [Sections 5.1 and 5.2, Table 4] The claim that CAT improves FID over fixed-ratio baselines 'trained with the same flops' is confounded by a difference in the number of images seen per step. The setup fixes the global token batch size at 262,144 tokens and the number of steps at 400K. Since CAT uses on average 197.44 tokens per image during training while the fixed 16× tokenizer uses 256, DiT-CAT processes approximately 30% more images per step (1328 vs 1024) and therefore approximately 1.3x more total training images over the run. The paper itself acknowledges this in Section 5.2 ('more extensive and diverse training within the same computational budget'). A fixed-ratio control with the same average token count (e.g., a fixed 18x tokenizer or a fixed 16x tokenizer with random token dropout to 197 tokens per image, trained at the same token batch size and steps) is needed to determine whether adaptive allocation rather than extra data diversity is responsible for the 4.56 vs 4.78 FID gap.
  2. [Section 5.2, Table 4] The reported FID gap (4.56 vs 4.78) is small and is presented without error bars or multiple seeds; DiT training is known to be sensitive to seed, so a 0.22 difference may not be statistically significant. Additionally, the token allocation is different between training and inference: training captions are generated with InstructBlip, while inference uses the template 'this is an image of [label]', yielding an average token count of 216 rather than 197.44. The authors should report mean and standard deviation over at least three seeds and should test whether the inference-time caption/token distribution is consistent with training, since the throughput and FID claims rely on that distribution.
  3. [Section 3.2, Eq. (2) and Table 2] The caption-based complexity score is validated only against the 'maximum acceptable compression ratio' defined by MSE tolerance of a specific pretrained LDM-VAE (τ = 0.0015) on COCO, and the reported Pearson correlation is 0.55. The thresholds (a, b) are then fitted to achieve an average 16x compression on the Shutterstock training set. This is a reasonable internal calibration, but it leaves open how sensitive the downstream results are to τ and to the threshold selection. A sensitivity analysis over τ and over alternative threshold settings (beyond the (4,7) vs (2,8) comparison in Table 3) would strengthen the claim that the LLM score, rather than the calibration, drives the reported gains.
minor comments (7)
  1. [Section 3.1.1, Eq. (1)] Equation (1) defines the maximum acceptable compression ratio using an argmax over f of the expression MSE_f - min_{f'} MSE_{f'}; as written this always selects f=32 and does not express the intended notion. The inequality should be inside a set-builder or the requested ratio should be the largest f for which the difference is below τ.
  2. [Section 5.2] The 18.5% inference throughput gain is reported in samples per second without specifying batch size, hardware, or the exact measurement protocol, so the number is not independently reproducible from the current text.
  3. [Appendix 8.1] The discriminator is described as a 'pretrained StyleGAN architecture' without a specific checkpoint or training configuration; please provide the exact reference and any modifications.
  4. [Section 4.1] There is a typo: 'emprical' should be 'empirical'.
  5. [Table 2 caption] The table caption refers to 'Average spatial dimension (r/f)' but the column is labeled 'Latent Dim'; please align the notation.
  6. [Throughout] The paper inconsistently uses 'flops' and 'FLOPs'; please standardize the capitalization.
  7. [Abstract and Introduction] The introduction lists 'Faster Generative Learning' as a contribution, but the paper reports only inference throughput and relative FLOPs, not training wall-clock time; if training speedup is intended, please report it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's complexity predictor, threshold selection, and generation comparisons are empirical design choices rather than derivations that reduce to their own inputs.

full rationale

No load-bearing circular step is present. The caption-complexity score is not derived from the target reconstruction or generation results: it is an LLM rating of a caption (Section 3.2) validated against an external benchmark, the LDM-VAE-based maximum acceptable compression ratio (Pearson r = 0.55, Table 2). The thresholds (a, b) that map scores to 8x/16x/32x compression are chosen to meet a target average compression ratio via Eq. (2), and the choice between (4,7) and (2,8) is made on validation performance (Table 3); this is parameter fitting, but the reported rFID/FID numbers are measured outcomes, not quantities forced by that fit. The nested VAE is an architectural contribution and is compared against both fixed-ratio baselines and a JPEG-complexity adaptive baseline, so it is not a renamed version of a known result. In Section 5, DiT-CAT and the fixed-16x DiT baseline are trained with the same global token batch and step count, which equalizes token throughput but gives CAT more distinct images per step (197.44 vs 256 training tokens per image, Section 5.2); the paper explicitly attributes part of the FID gain to this extra data diversity. That is a legitimate experimental confound for the claim that adaptivity itself drives the gain, but it is not circularity: the FID improvement is an empirical observation, not an algebraic consequence of the fitting equations. Self-citations by the authors appear only in related-work and future-direction contexts and are not used as load-bearing justification for the central claims. No uniqueness theorem, ansatz smuggled through citation, or definitional equivalence of input and output occurs.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities; its new machinery is the nested VAE routing and the LLM-based caption scorer. The free parameters are the compression ratio set, score thresholds, target average ratio, tolerance, latent channel, and loss weights. The main domain assumptions are that captions plus a hand-designed LLM prompt predict human-perceptual complexity, that a shared middle block handles all scales, and that the proprietary training distribution transfers to the evaluation benchmarks.

free parameters (6)
  • Compression ratio set = {8, 16, 32}
    Chosen in Section 3.1.1 based on LDM-VAE reconstruction analysis; not derived from a theory.
  • Thresholds (a, b) for LLM score classification = (4, 7)
    Calibrated to achieve a target average compression of 16x on training data; selected over (2,8) after observing Table 3 outcomes (Section 4.1).
  • Target average compression ratio = 16x
    Set to enable fair comparison with fixed 16x baselines (Section 3.2).
  • Tolerance tau for max acceptable compression = 0.0015 (text also says 0.0001)
    Defines the ground-truth 'maximum acceptable compression ratio' in Section 3.1.1; the paper is internally inconsistent about its value.
  • Latent channel dimension c = 16 (main), 8 (best generation)
    Ablated in Tables 5 and 7; the choice materially affects reconstruction-generation trade-off.
  • Loss weights (KL, perceptual, MoCo, GAN) = 1e-6, 1.0, 0.2, 0.5
    Tuned hyperparameters listed in Appendix 8.2; no sensitivity analysis is reported.
assumptions (5)
  • domain assumption The LLM-assigned complexity score from a caption plus yes/no answers about text and faces is a reliable proxy for human-perceptual reconstruction difficulty.
    Central to the method (Section 3.2); supported only by a single correlation measurement (r=0.55 on COCO) against LDM-VAE MSE-defined acceptable compression.
  • domain assumption Captions generated by InstructBlip (or provided text at inference) are sufficient for complexity scoring, including for ImageNet where 'this is an image of [label]' is used.
    Depends on InstructBlip's caption quality and on whether a class-name template captures content complexity (Section 5.1).
  • domain assumption A single shared middle block in the nested VAE can represent multi-scale features for all compression ratios without per-ratio specialization beyond extra attention layers.
    Assumed in Section 3.3; the paper provides no dedicated ablation of this architectural choice.
  • domain assumption The 380M Shutterstock training distribution, together with the LLM-selected compression ratio distribution, transfers to COCO, ImageNet, CelebA, and ChartQA.
    The paper evaluates on those datasets but trains on proprietary Shutterstock data (Section 4.1).
  • domain assumption The 'maximum acceptable compression ratio' computed from fixed LDM-VAE MSEs under a tolerance is a valid oracle for image complexity.
    Used to validate the caption complexity metric in Section 3.1.1; the oracle itself is a particular tokenizer's reconstruction error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAT: Content-Adaptive Image Tokenization." pith.science (2026). https://pith.science/paper/RUCVRZSJ

@misc{pith2026250103120,
  author       = {Pith},
  title        = {Pith review of: CAT: Content-Adaptive Image Tokenization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUCVRZSJ}},
  note         = {Machine review of arXiv:2501.03120}
}
read the original abstract

Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%.

Figures

Figures reproduced from arXiv: 2501.03120 by the authors.

Figure 1
Figure 1. Content-Adaptive Tokenization. CAT uses an LLM to evaluate the content complexity and determine the optimal compression ratio based on the image’s text description. The image is processed by a nested VAE architecture that dynamically routes the input according to the selected compression ratio. The resulting latent representations thus have varying spatial dimensions. Images shown in the figure are taken from COCO 2… view at source ↗
Figure 2
Figure 2. Left: Maximum acceptable compression ratios for COCO images under different error tolerance. We can compress most images more aggressively without compromising reconstruc￾tion quality. Right: Pearson correlation between various metrics and max acceptable compression ratio with tolerance 0.0015. to obtain variable-length token sequences. However, these methods are tailored for visual understanding tasks and can￾not b… view at source ↗
Figure 3
Figure 3. Existing metrics can misjudge image complexity. Metrics like JPEG size, MSE, and LPIPS consider images with high contrast and repetitive patterns as complex but underestimate the complexity of text-heavy images that are more challenging for human perception (note the distortion in the bottom two rows). Images shown in the figure are taken from COCO 2014 [9]. in a zebra image can toggle pixel values between black and… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: We highlight the compression ratio selected by our proposed caption complexity in red. On simpler images (top two rows), adjusting the CAT compression ratio does not significantly affect quality. On more complex images (bottom three rows), the impact is substantial. Al…
Figure 5
Figure 5. Figure 5: Increasing token count (left→right) for CAT leads to better image quality and higher complexity. c FID↓ sFID↓ IS↑ Precision↑ Recall↑ 4 5.11 10.84 158.80 0.75 0.49 Fixed 16x 8 4.96 10.39 221.85 0.76 0.51 16 4.78 11.81 187.47 0.72 0.49 4 5.12 11.12 152.39 0.72 0.48 CAT 8…
Figure 6
Figure 6. Figure 6: On COCO 2014 test set, the minimum JPEG size [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: More CAT reconstruction examples. We highlight the compression ratio selected by our proposed caption complexity in red. Images shown in the figure are taken from COCO 2014 [9] and ChartQA [14]. 5 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: More DiT-CAT generation examples. Increasing token count (left→right) generally leads to better image quality and higher complexity. 6 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Single-pass Adaptive Image Tokenization for Minimum Program Search

    cs.CV 2025-07 conditional novelty 7.0 of 10

    KARL conditions a tokenizer on a target reconstruction loss and learns halting probabilities that produce an adaptive token count in a single forward pass.

  2. ELT: Elastic Looped Transformers for Visual Generation

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Weight-shared looped transformers trained with intra-loop self-distillation match MaskGIT-class FID/FVD at roughly 4x fewer parameters and support any-time inference across loop counts.

  3. Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.

  4. Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity

    cs.LG 2025-01 conditional novelty 5.0 of 10

    Modality-specific projection weights let a Mamba model match dense multimodal baselines at the same loss using 25% to 65% of the training compute.

Reference graph

Works this paper leans on

66 extracted references · 32 canonical work pages · cited by 4 Pith papers

  1. [1]

    Taming transformers for high-resolution image synthesis, 2020

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Taming transformers for high-resolution image synthesis, 2020. 1, 5

    work page 2020

  2. [2]

    Auto-encoding varia- tional bayes, 2014

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes, 2014. 2, 5

    work page 2014

  3. [3]

    Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vigh- nesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G

    Lijun Yu, Jos ´e Lezama, Nitesh B. Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vigh- nesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats dif- fusion – tokenizer is key to visual generation, 2024. URL https://arxiv.org/abs/2310.05737

    arXiv 2024

  4. [4]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arxiv: 2406.07550, 2024

    arXiv 2024

  5. [5]

    Ef- ficient architecture search for diverse tasks

    Junhong Shen, Mikhail Khodak, and Ameet Talwalkar. Ef- ficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    work page 2022

  6. [6]

    NAS-bench- 360: Benchmarking neural architecture search on diverse tasks

    Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS-bench- 360: Benchmarking neural architecture search on diverse tasks. In Advances in Neural Information Processing Sys- tems (NeurIPS) Datasets and Benchmarks Track, 2022

    work page 2022

  7. [7]

    Finite scalar quantization: Vq-vae made simple, 2023

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple, 2023. URL https://arxiv.org/abs/2309. 15505. 1, 2

    work page 2023

  8. [8]

    G.K. Wallace. The jpeg still picture compression standard. IEEE Transactions on Consumer Electronics , 38(1):xviii– xxxiv, 1992. doi: 10.1109/30.125072. 1, 2

    doi:10.1109/30.125072 1992

Show all 66 references
  1. [9]

    Lawrence Zitnick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll ´ar. Microsoft coco: Common objects in context, 2015. URL https: //arxiv.org/abs/1405.0312. 2, 3, 4, 6, 7, 5

    2015 arXiv

  2. [10]

    Elastictok: Adaptive tok- enization for image and video

    Wilson Yan, Matei Zaharia, V olodymyr Mnih, Pieter Abbeel, Aleksandra Faust, and Hao Liu. Elastictok: Adaptive tok- enization for image and video. arXiv preprint, 2024. 1, 3, 5

    2024

  3. [11]

    High-resolution image syn- thesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021. 1, 3, 8, 9

    2021

  4. [12]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009. 2, 6

    2009

  5. [13]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , De- cember 2015. 2, 6

    2015

  6. [14]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning, 2022. URL https://arxiv.org/abs/2203.10244 . 2, 6, 7, 5

    2022 arXiv

  7. [15]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022. 2, 8

    2022 arXiv

  8. [16]

    Neural discrete representation learning, 2018

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018. URL https://arxiv.org/abs/1711.00937. 2

    2018 arXiv

  9. [17]

    Wiegand, G.J

    T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the h.264/avc video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7):560–576, 2003. doi: 10.1109/TCSVT.2003.815165. 2

    2003

  10. [18]

    An im- age is worth 16x16 words: Transformers for image recog- nition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An im- age is worth 16x16 words: Transformers for image recog- nit...

    2010 arXiv

  11. [19]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2

    2021

  12. [20]

    A-ViT: Adaptive tokens for ef- ficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for ef- ficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022

    2022

  13. [21]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023

    2023

  14. [22]

    Efficient video action detection with token dropout and context refinement, 2023

    Lei Chen, Zhan Tong, Yibing Song, Gangshan Wu, and Limin Wang. Efficient video action detection with token dropout and context refinement, 2023. URL https:// arxiv.org/abs/2304.08451. 2

    2023 arXiv

  15. [23]

    Vision trans- formers with mixed-resolution tokenization, 2023

    Tomer Ronen, Omer Levy, and Avram Golbert. Vision trans- formers with mixed-resolution tokenization, 2023. URL https://arxiv.org/abs/2304.00287. 2

    2023 arXiv

  16. [24]

    Shivam Duggal, Phillip Isola, Antonio Torralba, and William T. Freeman. Adaptive length image tokenization via recurrent allocation, 2024. URL https://arxiv.org/ abs/2411.02393. 3

    2024 arXiv

  17. [25]

    U-net: Convolutional networks for biomedical image segmentation,

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation,

  18. [26]

    Matryoshka representation learning

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. In Advances in Neural Information Processing Systems, December 2022. 3, 5

    2022

  19. [27]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430, 2024

    2024 arXiv

  20. [28]

    Matryoshka diffusion models, 2024

    Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh Susskind, and Navdeep Jaitly. Matryoshka diffusion models, 2024. URL https://arxiv.org/abs/2310.15111. 3 10

    2024 arXiv

  21. [29]

    Transframer: Arbitrary frame prediction with generative models, 2022

    Charlie Nash, Jo ˜ao Carreira, Jacob Walker, Iain Barr, An- drew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models, 2022. URL https : / / arxiv . org / abs / 2203.09494. 3

    2022 arXiv

  22. [30]

    Matryoshka query trans- former for large vision-language models, 2024

    Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query trans- former for large vision-language models, 2024

    2024

  23. [31]

    Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar

    Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: align then refine. In Proceedings of the 40th International Conference on Machine Learning , 2023

    2023

  24. [32]

    Ups: Towards foundation models for pde solving via cross-modal adaptation

    Junhong Shen, Tanya Marwah, and Ameet Talwalkar. Ups: Towards foundation models for pde solving via cross-modal adaptation. arXiv preprint arXiv:2403.07187, 2024

    2024 arXiv

  25. [33]

    Nicholas Roberts, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, Minwen Deng, Johannes Hog, Alexander Pfefferle, Sushil Ammanaghatta Shivakumar, Ar- jun Krishnakumar, Yubo Wang, Rhea Sanjay Sukthanker, Fr...

  26. [34]

    Tag-llm: Repurposing general-purpose llms for specialized domains, 2024

    Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-llm: Repurposing general-purpose llms for specialized domains, 2024. 3

    2024

  27. [35]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 3, 5

    2018

  28. [36]

    Very deep con- volutional networks for large-scale image recognition, 2015

    Karen Simonyan and Andrew Zisserman. Very deep con- volutional networks for large-scale image recognition, 2015. URL https://arxiv.org/abs/1409.1556. 3

    2015 arXiv

  29. [37]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  30. [38]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  31. [39]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385. 5

    2015 arXiv

  32. [40]

    Momentum contrast for unsupervised visual rep- resentation learning, 2020

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning, 2020. URL https://arxiv.org/ abs/1911.05722. 5

    2020 arXiv

  33. [41]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https://arxiv.org/abs/1406.2661. 5

    2014 arXiv

  34. [42]

    Image quality metrics: Psnr vs

    Alain Hor ´e and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pages 2366–2369, 2010. doi: 10.1109/ICPR. 2010.579. 6

    2010 doi

  35. [43]

    Emu: Enhancing image generation models using photogenic nee- dles in a haystack, 2023

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Mot- wani, ...

    2023 arXiv

  36. [44]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium, 2018

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium, 2018. URL https://arxiv.org/abs/1706. 08500. 8

    2018

  37. [45]

    Welch, and Z

    Xin Ding, Yongwei Wang, Zuheng Xu, William J. Welch, and Z. Jane Wang. Continuous conditional generative ad- versarial networks: Novel empirical losses and label input mechanisms, 2023. URL https://arxiv.org/abs/ 2011.07466. 8

    2023 arXiv

  38. [46]

    Improved techniques for training gans, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016. URL https://arxiv.org/ abs/1606.03498. 8

    2016 arXiv

  39. [47]

    Improved precision and recall met- 12 ric for assessing generative models, 2019

    Tuomas Kynk ¨a¨anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall met- 12 ric for assessing generative models, 2019. URL https: //arxiv.org/abs/1904.06991. 8

    2019 arXiv

  40. [48]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL https://arxiv.org/abs/ 2207.12598. 8

    2022 arXiv

  41. [49]

    Junhong Shen and Lin F. Yang. Theoretically principled deep rl acceleration via nearest neighbor function approxi- mation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11):9558–9566, May 2021. doi: 10.1609/ aaai.v35i11.17151. URL https://ojs.aaai.org/ inde...

    2021

  42. [50]

    Scribeagent: Towards specialized web agents us- ing production-scale workflow data, 2024

    Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Am- lekar, Mouad Hadji, Aaron Podolny, and Ameet Tal- walkar. Scribeagent: Towards specialized web agents us- ing production-scale workflow data, 2024. URL https: //arxiv.org/abs/2411.15004. 9

    2024 arXiv

  43. [51]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024. URL https://arxiv.org/ abs/2405.09818. 9

    2024 arXiv

  44. [52]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model, 2024. URL https://arxiv.org/abs/2408. 11039. 9

    2024

  45. [53]

    A style-based generator architecture for generative adversarial networks,

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks,

  46. [54]

    The open images dataset v4: Uni- fied image classification, object detection, and visual rela- tionship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Uni- fied image classification, object detection, and ...

    1956

  47. [60]

    • Are there faces in the image? → Yes/No

    Prompt for LLM Scorer Our caption complexity pipeline works as follows: Step 1: Use Salesforce/instructblip-vicuna-7b to generate caption, with the following prompts: • What’s in the image? → Caption • Are there text or numbers in the image? → Yes/No. • Are there faces in the ...

  48. [61]

    Number of distinct objects

  49. [62]

    Foreground and background

  50. [63]

    Symmetry and repetition

  51. [64]

    Score: ? out of 9

    Human perception factors, like the presence of human faces or text You will be given the caption, whether there are text or numbers, and whether there are faces in the image. Assign a complexity score such that a higher number means the image is more complex. Note that text an...

  52. [65]

    Architecture We implement the nested V AE similar to theAutoencoderKL implementation of the diffusers library

    Reconstruction Experiments 8.1. Architecture We implement the nested V AE similar to theAutoencoderKL implementation of the diffusers library. The network configuration is: • sample size: 512 • in channels: 3 • out channels: 3 • down block types: [DownEncoderBlock2D] × 6 • up ...

    2014

  53. [66]

    Architecture We use DiT-XL architecture with a patchify downsampler and patch size of 2

    Generation Experiments 9.1. Architecture We use DiT-XL architecture with a patchify downsampler and patch size of 2. The model size depends on the latent channel, but is generally around 431M parameters. The model Tflops is 22.0. 9.2. Training & Inference The training configur...

    2014

  54. [1405]

    URL http: //dx.doi.org/10.1007/s11263-020-01316-z

    doi: 10.1007/s11263-020-01316-z. URL http: //dx.doi.org/10.1007/s11263-020-01316-z . 3 13 CA T: Content-Adaptive Image Tokenization Supplementary Material

    doi

  55. [2015]

    URL https://arxiv.org/abs/1505.04597. 3, 5

    arXiv

  56. [2019]

    URL https://arxiv.org/abs/1812.04948. 2

    arXiv

  57. [2021]

    URL https://api.semanticscholar.org/ CorpusID:265536645

  58. [2023]

    URL https://arxiv.org/abs/2305.06500. 4

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.