Pith. sign in

REVIEW 4 major objections 6 minor 62 references

Images are Worth Variable Length of Representations

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A vision encoder that lets each image choose its own token count produces better reconstructions and VQA answers with fewer tokens than fixed-length tokenizers.

desk verdict A genuine EOS-based dynamic tokenizer with broad experiments, but the headline efficiency claim needs per-dataset token counts and EOS-quality numbers before I'd trust the aggregate. read the letter →

arxiv 2506.03643 v2 pith:ZXO4C6FB submitted 2025-06-04 cs.CV

Lingjun Mao , Rodolfo Corona , Xin Liang , Wenhao Yan , Zineng Tang This is my paper · ORCID
classification cs.CV
keywords dynamicvisionencodervariable-lengthtokenizationend-of-sequencepredictionimagereconstructionvision-languagemodelsvisualquestionansweringtokencompressionemergentsemantics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that assigning every image the same number of tokens is wasteful, because images carry different amounts of information. It introduces DOVE (Dynamic Output Vision Encoder), a vision encoder that generates a variable-length sequence of continuous tokens per image and learns to emit an end-of-sequence token once its reconstruction is good enough. The central claim is that this dynamic policy uses fewer tokens on average than fixed-length tokenizers while matching or improving reconstruction, classification, and visual-question-answering performance. A query-conditioned variant, Q-DOVE, concentrates tokens on regions relevant to a text query and compresses further. If the paper is right, token count stops being a fixed hyperparameter and becomes a per-image, even per-question, decision.

What carries the argument

The central mechanism is a learned end-of-sequence (EOS) token inside an autoregressive token generator. The generator is a 70M-parameter transformer (Pythia) that, given VQGAN encoder features plus timestamp encodings, produces continuous visual tokens and can output EOS at any of up to K positions; all slots after EOS are replaced with zero vectors, and a second transformer decodes the padded sequence for the VQGAN decoder. The EOS training loss compares the current sample's reconstruction loss to an exponential moving average of the last 100 losses, rewarding earlier termination when reconstruction already meets the threshold and later termination when it does not. This threshold rule is what turns token length into a learned, per-sample decision rather than a constant.

What would settle it

Run DOVE on a set of solid-color or otherwise trivially simple images and record its EOS positions; the central claim predicts that these images will terminate at very few tokens, so if the average length stays near the general training average (for example above 100), the threshold rule is not tracking image complexity.

Watch

Extended reading notes

Core claim

DOVE extends an autoencoder-style tokenizer with a transformer-based dynamic token generator. For each image, the generator produces tokens one at a time and may stop at any position by emitting an end-of-sequence symbol; tokens after that point are zeroed and a fixed decoder reconstructs the image from the truncated sequence. Training couples sequence length to reconstruction quality by comparing each image's reconstruction loss with a moving average of recent losses: when the loss is already below the threshold the model is encouraged to end sooner, and when it is above the threshold the model is encouraged to continue. The paper reports an average of about 121.6 tokens per image instead of 256, a Pearson correlation of 0.742 between EOS position and image complexity, and higher scores than VQGAN, TiTok, and ALIT on most VQA benchmarks and linear-probing sets even at 32 tokens. Query-conditioned Q-DOVE adds a bounding-box weighted loss so tokens focus on query-relevant regions, reducing the average to about 82.4 tokens while improving VQA accuracy further.

Load-bearing premise

The load-bearing premise is that comparing an image's reconstruction loss to a moving average of the last 100 training losses reliably indicates how many tokens that image needs, and that this rule continues to work on new images and datasets.

Editorial extensions

If this is right

  • A vision-language model using DOVE needs roughly half the visual tokens (121.6 average versus 256) and still scores higher than VQGAN and TiTok on VQAv2, GQA, OK-VQA, and ScienceQA in the paper's experiments.
  • Because the encoder can be truncated at any position in a single forward pass, one model can serve different compute budgets at inference time without retraining.
  • Q-DOVE shows that conditioning token generation on a text query further cuts token count (82.4 average) while improving VQA accuracy, making task-driven compression a working alternative to fixed token budgets.
  • DOVE's semantic advantage appears in linear probing, where it outranks fixed-length autoencoder tokenizers on all seven reported benchmarks, and in PCA visualizations that resemble semantic segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same EOS-plus-threshold recipe could be carried over to video, audio, or 3D tokenizers, where per-sample information varies even more than in images.
  • The moving-average threshold is a training heuristic; a learned complexity estimator or a reward model for token length might make the policy transfer more reliably across datasets, which the paper does not test.
  • If the reported semantic emergence is real and not an artifact of the threshold rule, variable-length autoencoders could serve as cheap unsupervised feature extractors for segmentation and retrieval, not only as QA backbones.
  • Q-DOVE's bounding-box supervision could be relaxed to attention or saliency signals, which would let query-conditioned compression apply where box annotations are unavailable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces DOVE, a dynamic vision encoder that generates a variable-length sequence of continuous tokens for each image and terminates the sequence at a predicted EOS position, trained with a reconstruction loss plus a heuristic EOS loss based on an EMA threshold. A query-conditioned variant (Q-DOVE) refines this idea by supervising reconstruction with bounding boxes around answer-relevant regions. The authors evaluate DOVE on image reconstruction FID, token-length distributions, frozen-feature classification and linear probing, and as the vision encoder of a Vicuna-7B VLM across eight VQA benchmarks, reporting large gains over TiTok, VQGAN, and ALIT at equal or smaller token counts.

Significance. The idea of allowing token count to vary with image complexity is timely, and a working dynamic tokenizer that can be dropped into VLM pipelines would be practically valuable. The paper's breadth is a strength: it evaluates reconstruction, classification, linear probing, and downstream VQA, and it promises code and checkpoints. If the efficiency-quality trade-off is confirmed, DOVE would be a useful alternative to fixed-length tokenizers for multimodal systems. The main weaknesses are that the central claim is not directly evidenced at the dynamic operating point, and several technical details of the EOS mechanism and evaluation protocol are underspecified.

major comments (4)
  1. [Sec. 2.2, Table 1, Eq. (3)] The EOS training objective is not reproducible as written. Table 1's pseudocode treats the generated sequence D as containing an EOS token ('Find the first index j such that D[j]=EOS') even though the model is described as producing continuous visual tokens, and it does not say how p_eos(i) is produced. After the while loop, i is K+1, so the branch 'L_eos ← p_eos(i)' in Table 1 uses an invalid position, whereas Eq. (3) is written in terms of the EOS position m. The pseudocode also has no branch for the case where no EOS token is generated before the maximum length K. Please specify the EOS prediction head, the exact index used in the loss, and the inference-time behavior when no EOS is emitted.
  2. [Sec. 3.2-3.3, Table 3] The headline claim that DOVE 'significantly reduces the average number of tokens while maintaining high reconstruction quality' is supported only by a single aggregate token count (121.6 in Table 3). No per-benchmark breakdown of EOS lengths is given, and the token-length distribution in Figure 6a is not tied to the downstream benchmarks; the reconstruction-loss curve in Figure 6b is for fixed lengths on COCO, not for sequences truncated at predicted EOS positions. Table 2 reports FID only at fixed lengths. As a result, the reader cannot determine whether the dynamic-length output actually maintains reconstruction quality or whether the 121.6 average is dominated by particular datasets. Please report per-dataset (or at least per-benchmark) average token counts, the EOS-length distribution on those benchmarks, and a reconstruction metric (FID or L1) for the EOS-truncated sequences.
  3. [Sec. 3.4, Appendix C] The linear probing results in Table 5 are not interpretable without specifying which hidden layer is probed and the training protocol. The main text says probing is done on 'model's hidden layers' and Appendix C refers to 'a selected hidden layer,' but neither the layer index nor the classifier training details (epochs, learning rate, pooling) are given. Since linear probing accuracy varies substantially across layers, the cross-model comparison may be unfair, and this weakness directly affects the 'emergent semantics' claim.
  4. [Sec. 2.2-2.3] The EOS policy is determined by a training heuristic with several unexamined free parameters: the EMA window W (set to 100), the λ_eos schedule, and, for Q-DOVE, λ_o=1e-10. Because the token-length distribution is entirely a function of the EMA threshold, the robustness of the efficiency claim requires a sensitivity analysis or, at minimum, a report of the variance of the average token count across training runs and datasets. In addition, Section 2.3's description of L_pen is ambiguous: it says L_irr is compared to a threshold after defining L_eos via L_rel, and the penalty term appears identical to the second branch of Eq. (3). Please clarify whether L_pen is an extra term and which loss is used for the threshold comparison.
minor comments (6)
  1. [Table 2] Table 2's column layout is garbled; the values cannot be unambiguously assigned to (dataset, token length) pairs. Please reformat the table so each FID score has a clear header.
  2. [Sec. 3.2] The classification subsection says accuracy improves with token count but no quantitative accuracy table is provided; please add numeric results or a table in addition to Figure 5.
  3. [Table 3] Please define what '121.6 (Avg)' and '82.4 (Avg)' average over, and report the standard deviation or per-benchmark values; the Q-DOVE rows labeled '256#' and '256' should also be disambiguated in the table header.
  4. [Table 4] Please report speed and FLOPs at the EOS-truncated average length in addition to fixed lengths, since that is the claimed operating point.
  5. [Sec. 3.1] Please specify which parameters are frozen versus fine-tuned (VQGAN encoder/decoder, the two Pythia models) and give the exact training configurations used for the baselines (TiTok, ALIT) to support the 'same configuration' claim.
  6. [Throughout] The word 'significantly' is used throughout without statistical tests or error bars; at minimum, state the number of seeds and report standard deviations for the main tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DOVE's efficiency and downstream claims rest on external benchmarks and independent measurements, not on a derivation that reduces to its own inputs.

full rationale

The paper's derivation chain is as follows: the dynamic token generator f_phi is trained with a reconstruction loss L_rec (Eq. 2) and an EOS-control loss L_eos (Eq. 3), where L_eos uses an EMA of recent reconstruction losses as a threshold to encourage earlier or later EOS. This is a self-referential training heuristic, since the token-length policy is shaped by the model's own reconstruction quality, but it is not circular in the sense of defining a prediction in terms of the quantity it is claimed to predict. The reported average token count (121.6, Table 3) is a measured outcome on downstream benchmarks, not an input to the training objective. The claim that more complex images receive longer sequences is validated with an independent proxy (Laplacian variance, Figure 6c), not with the training loss itself. Reconstruction quality is evaluated with FID on fixed token lengths (Table 2), and downstream performance uses external VQA and linear-probing benchmarks (Tables 3 and 5) with a frozen encoder and standard fine-tuning protocol. The paper cites prior work (VQGAN, TiTok, ALIT, Pythia, Vicuna, etc.) for components and baselines, but no load-bearing claim is justified solely by a self-citation, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The skeptic's concerns about per-benchmark EOS lengths and missing EOS-truncated reconstruction metrics are evidentiary gaps rather than circularities: those omissions do not show that any predicted quantity is equivalent to a fitted input by construction. Overall, the central claims have independent empirical content, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rely on a standard autoencoder backbone, a heuristic EOS training rule with several hand-chosen hyperparameters, and a domain assumption linking reconstruction difficulty to image complexity.

free parameters (5)
  • EMA window W = 100 steps
    Used as the dynamic reconstruction-loss threshold for EOS supervision (Section 2.2, pseudocode). The value is chosen by hand and not ablated.
  • lambda_o = 1e-10
    Weight for the irrelevant-region loss in Q-DOVE (Eq. 7). Setting it near zero essentially tells the model to ignore background, which drives the compression gain.
  • lambda_eos schedule = small initially, gradually increased
    Controls how strongly EOS prediction is learned relative to reconstruction (Section 2.2). The schedule is not precisely specified.
  • GAN loss weight = 5e-10
    Chosen by validation L1 comparison (Appendix A.3); the paper shows larger weights hurt reconstruction.
  • Maximum token K = 256
    Maximum sequence length in the token generator; all experiments truncate at 256. Not ablated.
assumptions (5)
  • domain assumption Images with more visual complexity require more tokens for faithful reconstruction.
    This is the motivating premise of the paper (Section 1). The paper does not prove it; it relies on a post-hoc correlation with Laplacian variance (Figure 6c).
  • domain assumption Reconstruction loss (MSE plus perceptual plus a tiny GAN term) is a sufficient training signal for representations that are useful in downstream classification and VQA.
    Section 2.2 defines L_rec and uses it as the only supervision for representation quality, apart from the EOS loss.
  • domain assumption The pretrained VQGAN encoder and decoder can serve as the base autoencoder and can be fine-tuned to support dynamic tokenization.
    Section 2.1 builds on a pretrained VQGAN with codebook size 8192; the paper does not ablate the choice.
  • domain assumption For Q-DOVE, bounding boxes in Visual Genome and Open Images correctly mark regions relevant to the text query.
    Section 2.3 uses these boxes to reweight reconstruction loss; errors in box-query correspondence would misguide the model.
  • standard math A causal mask over timestamp positions, with no input token values, is sufficient to generate coherent variable-length visual tokens.
    Section 2.1 describes the transformer generator; the exact input-output structure is not fully specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Images are Worth Variable Length of Representations." pith.science (2026). https://pith.science/paper/ZXO4C6FB

@misc{pith2026250603643,
  author       = {Pith},
  title        = {Pith review of: Images are Worth Variable Length of Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXO4C6FB}},
  note         = {Machine review of arXiv:2506.03643}
}
read the original abstract

Most existing vision encoders map images into a fixed-length sequence of tokens, overlooking the fact that different images contain varying amounts of information. For example, a visually complex image (e.g., a cluttered room) inherently carries more information and thus deserves more tokens than a simple image (e.g., a blank wall). To address this inefficiency, we propose DOVE, a dynamic vision encoder that produces a variable number of visual tokens (i.e., continuous representation vectors) to reconstruct each image. Our results show that DOVE significantly reduces the average number of tokens while maintaining high reconstruction quality. In several linear probing and downstream multimodal tasks, it outperforms existing autoencoder-based tokenization methods when using far fewer tokens, capturing more expressive semantic features compared to fixed-length encoding. We further extend DOVE with query-conditioned tokenization. By guiding the model to focus on query-relevant regions, it achieves more efficient and targeted semantic extraction. Our code and checkpoints are available at https://dove-encoder.github.io/dove-encoder.

Figures

Figures reproduced from arXiv: 2506.03643 by the authors.

Figure 1
Figure 1. Dynamic Visual Representations. As the number of tokens used by DOVE increases, the reconstructed images shows finer and high frequency details. 1 Introduction Image representation learning [56] is a fundamental component of computer vision; it plays a pivotal role in various visual tasks, including image classification [38, 12], object detection [62, 61], and semantic segmentation [26, 27]. Vision representation mo… view at source ↗
Figure 2
Figure 2. Dynamic Tokenizer. only to its current or preceding timestamps. The dynamic token generation process from timestamp t0 to ti is defined as: D = fϕ(Hv, t1, t2, . . . , ti) = (d1, d2, . . . , di) (1) where D denotes the generated token sequence, and di is the token produced by the model at ti . We introduce dynamic length variation by detecting the EOS token from the model’s discrete output and replacing all visual to… view at source ↗
Figure 3
Figure 3. Query Conditioning. DOVE is trained with a bounding-box based loss, learning to focus its dynamic token resources on representing query-relevant image regions. 3.1 Experimental Setup Training Details. We use a pretrained VQGAN [21] with a codebook size of 8192 and a lightweight Pythia-70M [7] language model as the backbone of our framework. The model is fine-tuned on ImageNet-1K [16] for 20 epochs using two NVIDIA R… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Reconstructed images on ImageNet-1K using different methods. As the token length [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Classification accuracy with different visual tokenizers under varying token lengths. DOVE [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Token length analysis that when the input query is “null”, the model clearly reconstructs the entire image. In contrast, when a relevant question or description is provided, the reconstruction focuses on the semantically related regions and produces lower frequency out…
Figure 7
Figure 7. Figure 7: Reconstructed images from the Q-DOVE. When the text query is set to “null”, the model [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Semantics Visualization with PCA on latent features. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Effect of varying GAN loss weight on image reconstruction quality. A small weight (e.g., [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Model predictions under varying token counts. As the number of tokens increases, both [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Reconstruction results of DOVE and DOVE (Gaussian) under varying token budgets. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 22 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    2023

  3. [3]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  4. [4]

    Revisiting active perception.Autonomous Robots, 42:177–196, 2018

    Ruzena Bajcsy, Yiannis Aloimonos, and John K Tsotsos. Revisiting active perception.Autonomous Robots, 42:177–196, 2018

    work page 2018

  5. [5]

    Blur image detection using laplacian operator and open-cv

    Raghav Bansal, Gaurav Raj, and Tanupriya Choudhury. Blur image detection using laplacian operator and open-cv. In2016 International Conference System Modeling & Advancement in Research Trends (SMART), pages 63–67. IEEE, 2016

    work page 2016

  6. [6]

    Statistical inference for probabilistic functions of finite state markov chains.The annals of mathematical statistics, 37(6):1554–1563, 1966

    Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains.The annals of mathematical statistics, 37(6):1554–1563, 1966

    work page 1966

  7. [7]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. InInternational Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    work page 2023

  8. [8]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. InThe Eleventh International Conference on Learning Representations, 2023

    2023

Show all 62 references
  1. [9]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. InEuropean Conference on Computer Vision, 2014

    2014

  2. [10]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  3. [11]

    Efficient large multi-modal models via visual context compression

    Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Efficient large multi-modal models via visual context compression. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  4. [12]

    Review of image classification algorithms based on convolutional neural networks.Remote Sensing, 13(22):4712, 2021

    Leiyu Chen, Shaobo Li, Qiang Bai, Jing Yang, Sanlong Jiang, and Yanming Miao. Review of image classification algorithms based on convolutional neural networks.Remote Sensing, 13(22):4712, 2021

    2021

  5. [13]

    An empirical study of smoothing techniques for language modeling

    Stanley F Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–394, 1999

    1999

  6. [14]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014

    2014

  7. [15]

    An analysis of single layer networks in unsupervised feature learning aistats

    Adam Coates, Honglak Lee, and AY Ng. An analysis of single layer networks in unsupervised feature learning aistats. 2011

    2011

  8. [16]

    Scaling up dataset distillation to imagenet-1k with constant memory

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. InInternational Conference on Machine Learning, pages 6565–6590. PMLR, 2023

    2023

  9. [17]

    Top-down control of eye movements: Yarbus revisited.Visual Cognition, 17(6-7):790–811, 2009

    Marianne DeAngelus and Jeff B Pelz. Top-down control of eye movements: Yarbus revisited.Visual Cognition, 17(6-7):790–811, 2009

    2009

  10. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 10

    2009

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  12. [20]

    Adaptive length image tok- enization via recurrent allocation

    Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tok- enization via recurrent allocation. InFirst Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024

    2024

  13. [21]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  14. [22]

    Multimodal autoregressive pre-training of large vision encoders, 2024

    Enrico Fini*, Mustafa Shukor*, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Louis Béthune, Zhe Gan, Victor Turrisi, Alexander Toshev, Marcin Eichner, Yinfei Yang, Moin Nabi, Josh Susskind, and Alaaeldin El-Nouby*. Multimodal autoregressive pre-train...

    2024

  15. [23]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017

    2017

  16. [24]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  17. [25]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  18. [26]

    A review of semantic segmentation using deep neural networks.International journal of multimedia information retrieval, 7:87–93, 2018

    Yanming Guo, Yu Liu, Theodoros Georgiou, and Michael S Lew. A review of semantic segmentation using deep neural networks.International journal of multimedia information retrieval, 7:87–93, 2018

    2018

  19. [27]

    A brief survey on semantic segmentation with deep learning

    Shijie Hao, Yuan Zhou, and Yanrong Guo. A brief survey on semantic segmentation with deep learning. Neurocomputing, 406:302–321, 2020

    2020

  20. [28]

    Hierarchical cross-modal agent for robotics vision-and-language navigation

    Muhammad Zubair Irshad, Chih-Yao Ma, and Zsolt Kira. Hierarchical cross-modal agent for robotics vision-and-language navigation. In2021 IEEE international conference on robotics and automation (ICRA), pages 13238–13246. IEEE, 2021

    2021

  21. [29]

    Perceiver io: A general architecture for structured inputs & outputs.arXiv preprint arXiv:2107.14795, 2021

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs.arXiv preprint arXiv:2107.14795, 2021

    2021 arXiv

  22. [30]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. InInternational conference on machine learning, pages 4651–

  23. [31]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013

    2013

  24. [32]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer v...

    2017

  25. [33]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  26. [34]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection a...

    1956

  27. [35]

    The roles of vision and eye movements in the control of activities of daily living.Perception, 28(11):1311–1328, 1999

    Michael Land, Neil Mennie, and Jennifer Rusted. The roles of vision and eye movements in the control of activities of daily living.Perception, 28(11):1311–1328, 1999

    1999

  28. [36]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings,...

    2014

  29. [37]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 11

    2023

  30. [38]

    A survey of image classification methods and techniques for improving classification performance.International journal of Remote sensing, 28(5):823–870, 2007

    Dengsheng Lu and Qihao Weng. A survey of image classification methods and techniques for improving classification performance.International journal of Remote sensing, 28(5):823–870, 2007

    2007

  31. [39]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    2022

  32. [40]

    Fine-grained visual classification of aircraft, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft, 2013

    2013

  33. [41]

    Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    2019

  34. [42]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    2022

  35. [43]

    V Jawahar

    Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021

    2021

  36. [44]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021

    2021

  37. [45]

    Stl-10, nov 2024

    N/A. Stl-10, nov 2024

    2024

  38. [46]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  39. [47]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  40. [48]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification.Advances in neural information processing systems, 34:13937–13949, 2021

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification.Advances in neural information processing systems, 34:13937–13949, 2021

    2021

  41. [49]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019

    2019

  42. [50]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  43. [51]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019

    2019

  44. [52]

    Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial intelligence, 112(1-2):181–211, 1999

    Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial intelligence, 112(1-2):181–211, 1999

    1999

  45. [53]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  46. [54]

    Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    2017

  47. [55]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural ...

    2017

  48. [56]

    Supervised hashing for image retrieval via image representation learning

    Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. Supervised hashing for image retrieval via image representation learning. InProceedings of the AAAI conference on artificial intelligence, volume 28, 2014

    2014

  49. [57]

    Ehinger, Aude Oliva, and Antonio Torralba

    Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485–3492, 2010. 12

    2010

  50. [58]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809–10818, 2022

    2022

  51. [59]

    Scaling autoregressive models for content-rich text-to-image generation.arXiv preprint arXiv:2206.10789, 2(3):5, 2022

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation.arXiv preprint arXiv:2206.10789, 2(3):5, 2022

    2022 arXiv

  52. [60]

    An image is worth 32 tokens for reconstruction and generation.Advances in Neural Information Processing Systems, 37:128940–128966, 2024

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation.Advances in Neural Information Processing Systems, 37:128940–128966, 2024

    2024

  53. [61]

    Object detection with deep learning: A review.IEEE transactions on neural networks and learning systems, 30(11):3212–3232, 2019

    Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object detection with deep learning: A review.IEEE transactions on neural networks and learning systems, 30(11):3212–3232, 2019

    2019

  54. [62]

    STOP” on a sign as “SHOP

    Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey.Proceedings of the IEEE, 111(3):257–276, 2023. 13 A Implementation Details A.1 Model Architecture Our framework builds on a pretrained VQGAN and two instances of the light...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.