pith. sign in

arxiv: 2606.01503 · v1 · pith:VZMLLGAWnew · submitted 2026-05-31 · 💻 cs.CV · cs.AI· cs.CL

On the Limits of Token Reduction for Efficient Unified Vision Language Training

Siyi Chen , Weiming Zhuang , Jingtao Li , Lingjuan Lv This is my paper

Pith reviewed 2026-06-28 16:56 UTC · model grok-4.3

classification 💻 cs.CV cs.AIcs.CL
keywords token reductionunified vision-language modelsefficient trainingattention allocationsynergy lossjoint optimizationvisual understandingvisual generation
0
0 comments X

The pith

Task-specific token reduction eliminates mutual performance gains in unified vision-language model training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates token reduction as a way to speed up training of unified vision-language models that handle both understanding and generation in one backbone. Analysis of attention across layers reveals that understanding tasks can drop image tokens late in the network while generation tasks require them at every depth. Separate task-specific reduction delivers speedups when tasks train alone, but the same approach in joint training forces the model to develop separate parameter routes for each task and removes the accuracy improvements that usually come from optimizing both together. The work concludes that preserving shared structures across tasks is essential for efficient unified training.

Core claim

Unified VLMs exhibit a fundamental asymmetry in layerwise attention: visual understanding shows substantial late-layer redundancy in image tokens, whereas visual generation maintains persistent dependence on image tokens across all depths. Task-specific token dropping therefore produces large efficiency gains in isolated training but, under unified training, necessitates divergent parameter pathways that eliminate the mutual performance gains normally observed in joint optimization.

What carries the argument

Layerwise attention allocation patterns that expose the asymmetry between late-layer visual redundancy in understanding and persistent image-token dependence in generation.

If this is right

  • Task-specific token accelerators deliver efficiency gains when each objective is trained in isolation.
  • The same accelerators produce consistent synergy loss when both objectives train jointly.
  • Task-specific dropping forces the model onto divergent parameter pathways.
  • Efficient unified modeling requires preservation of shared cross-task structures.
  • Acceleration methods must be designed to be synergy-aware rather than task-isolated.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same attention-asymmetry limit may appear in other multimodal unified models that combine understanding and generation.
  • Methods that keep a single shared backbone while allowing selective computation could avoid the observed divergence.
  • Empirical checks on larger-scale or differently pretrained unified VLMs would test whether the asymmetry is architecture-independent.
  • Efficiency research for unified models should prioritize joint-optimization compatibility from the design stage.

Load-bearing premise

The layerwise attention patterns observed are general properties of unified VLMs rather than artifacts of the tested architectures or training setups.

What would settle it

Apply the same task-specific token dropping to a different unified VLM architecture or training regime and check whether the mutual performance gains of joint optimization disappear or remain.

Figures

Figures reproduced from arXiv: 2606.01503 by Jingtao Li, Lingjuan Lv, Siyi Chen, Weiming Zhuang.

Figure 1
Figure 1. Figure 1: Unified autoregressive VLM. A single Transformer backbone processes multimodal sequences under a unified next￾token prediction objective. (a) In visual understanding, the model predicts text tokens conditioned on image and textual context. (b) In visual generation, the model autoregressively predicts image tokens conditioned on preceding text and image tokens. serving shared cross-task structures and caref… view at source ↗
Figure 2
Figure 2. Figure 2: Asymmetric depth-wise attention patterns in unified VLMs. Visualization of self-attention heatmaps across layers for understanding (a) and generation (b). Understanding exhibits strong early cross-modal interactions followed by a sharp decay in image￾token attention. Generation, however, preserves substantial image-token attention throughout depth, highlighting a fundamental asymmetry in token utilization.… view at source ↗
Figure 3
Figure 3. Figure 3: Quantitative attention allocation reveals depth-dependent visual redundancy asymmetry. Left: Attention mass distribution over token segments (system, image, instruction, output) at representative shallow and deep layers. Right: Layerwise attention allocation across the full transformer depth. For visual understanding (top), attention to image tokens sharply decreases in deeper layers, while in￾struction an… view at source ↗
Figure 4
Figure 4. Figure 4: Task-specific token-reduction-based acceleration mechanisms for unified VLM training. From left to right: (1) Vanilla Transformer layer, where both text and image tokens participate fully in self-attention and feed-forward computation. (2) HiMix (Under￾standing) reduce image tokens from the query stream while retaining them in key/value projections, eliminating quadratic image-to-image attention while pres… view at source ↗
Figure 5
Figure 5. Figure 5: Inference-time-only HMGen. Qualitative comparison between the original model (bottom) and inference-time-only HM￾Gen (top), where image-token computation in shallow layers is skipped without retraining. Visual quality and semantic consis￾tency are largely preserved despite reduced computation. HMGen: Hierarchical Mixture for Generation. Mo￾tivated by the hierarchical attention structure observed in Sec. 4,… view at source ↗
Figure 6
Figure 6. Figure 6: Separate image projection strategy. To reduce inter￾ference between task-specific routing, we decouple image-related projection parameters (e.g., WQ v , W K v , WV v ) for HiMix (high￾lighted by yellow) and HMGen-Full (highlighted by blue), while keeping the backbone shared. This design aims to stabilize hi￾erarchical image representations when token participation differs across tasks. indicating negative … view at source ↗
read the original abstract

Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective. In this work, we study the feasibility and limits of token-reduction-based acceleration for unified VLM training. Through a systematic analysis of layerwise attention allocation, we uncover a fundamental asymmetry: visual understanding exhibits substantial late-layer visual redundancy, whereas visual generation maintains persistent dependence on image tokens across depth. Guided by this observation, we design task-specific accelerators that selectively reduce image-token computation for each objective. While these methods achieve significant efficiency gains in isolated settings, we observe a consistent synergy loss under unified training -- task-specific token dropping necessitates divergent parameter pathways and eliminates the mutual performance gains typically observed in joint optimization. Our findings suggest that efficient unified modeling requires preserving shared cross-task structures, highlighting the need for synergy-aware acceleration strategies. Project page: https://chicychen.github.io/TokenReductionUnifiedVLM/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript claims that unified vision-language models exhibit a fundamental asymmetry in layerwise attention allocation—visual understanding shows substantial late-layer visual redundancy while visual generation maintains persistent image-token dependence across depth. This leads to the design of task-specific token-reduction accelerators that yield efficiency gains in isolated settings but produce a consistent synergy loss under joint training, as task-specific dropping forces divergent parameter pathways and removes the mutual performance benefits of unified optimization. The authors conclude that efficient unified modeling requires preserving shared cross-task structures and thus needs synergy-aware acceleration strategies.

Significance. If the asymmetry and resulting synergy-loss observation hold, the work is significant for identifying a concrete limit on naive token-reduction approaches in multi-task VLMs and for providing a diagnostic (layerwise attention analysis) that could inform better efficiency methods. The emphasis on preserving joint-optimization benefits rather than maximizing per-task speedups is a useful corrective for the field. No machine-checked proofs or parameter-free derivations are present, but the experimental framing of the synergy loss is a clear, falsifiable contribution if the supporting data and controls are robust.

major comments (1)
  1. [Abstract] Abstract: the central claim that task-specific token dropping 'necessitates divergent parameter pathways and eliminates the mutual performance gains' is load-bearing for the recommendation of synergy-aware strategies. This inference depends on the layerwise attention asymmetry being a general property of unified VLMs rather than an artifact of the tested architectures or training setups; the abstract provides no indication of cross-architecture ablations or controls that would establish this generality.
minor comments (2)
  1. The manuscript should clarify in the methods or experimental sections whether the attention-allocation analysis was performed on multiple model scales or only a single backbone, as this directly affects the scope of the 'fundamental asymmetry' claim.
  2. Ensure that all quantitative efficiency gains and synergy-loss measurements are accompanied by the exact token-reduction ratios, layer ranges, and training budgets used, to allow direct reproduction.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the concern about the abstract's framing of generality below and will make corresponding revisions.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that task-specific token dropping 'necessitates divergent parameter pathways and eliminates the mutual performance gains' is load-bearing for the recommendation of synergy-aware strategies. This inference depends on the layerwise attention asymmetry being a general property of unified VLMs rather than an artifact of the tested architectures or training setups; the abstract provides no indication of cross-architecture ablations or controls that would establish this generality.

    Authors: We agree that the abstract should better contextualize the experimental scope. Our layerwise attention analysis and synergy-loss observations were obtained across multiple representative unified VLM backbones and training regimes, with the asymmetry appearing consistently. We will revise the abstract to explicitly name the architectures and setups employed, thereby indicating the basis for our claims. Comprehensive cross-architecture ablations would strengthen generality but lie outside the current scope; the reported results already demonstrate that task-specific reduction disrupts joint-optimization benefits in the tested unified settings, supporting the call for synergy-aware methods. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical observations with no derivations or self-referential reductions

full rationale

The paper contains no equations, derivations, fitted parameters presented as predictions, or load-bearing self-citations. All claims derive from direct experimental measurements of attention allocation and training outcomes across the tested models and setups. The observed asymmetry and synergy loss are reported as experimental results rather than constructed from prior definitions or ansatzes within the work. No step reduces by construction to its inputs, satisfying the default expectation of non-circularity for observation-driven studies.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical derivations, free parameters, axioms, or invented entities are described; the work is an empirical study of training dynamics.

pith-pipeline@v0.9.1-grok · 5706 in / 948 out tokens · 23954 ms · 2026-06-28T16:56:26.557329+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 6 canonical work pages · 4 internal anchors

  1. [1]

    Token merging: Your vit but faster, 2023

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster, 2023. 1, 2

    2023

  2. [2]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387. Springer, 2024. 5

    2024

  3. [3]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024. 1, 2, 3

    2024

  4. [4]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. InThirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 1

    2023

  5. [5]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

  6. [6]

    Taming transformers for high-resolution image synthesis, 2021

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Taming transformers for high-resolution image synthesis, 2021. 1

    2021

  7. [7]

    Scaling rectified flow trans- formers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yan- nik Marek, and Robin Rombach. Scaling rectified flow trans- formers for high-resolution image synthesis, 2024. 1

    2024

  8. [8]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. MME: A comprehensive evaluation benchmark for multimodal large language models. InThe Thirty-ninth An- nual Conference on Neural Information Processing Systems Datasets and Benchmarks Trac...

    2025

  9. [9]

    Making llama see and draw with seed tokenizer, 2023

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer, 2023. 1

    2023

  10. [10]

    When Attention Sink Emerges in Language Models: An Empirical View

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781, 2024. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Denoising diffu- sion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models, 2020. 1

    2020

  12. [12]

    Matryoshka query trans- former for large vision-language models, 2024

    Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query trans- former for large vision-language models, 2024. 1, 2

    2024

  13. [13]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. 6

    2019

  14. [14]

    Unified language-vision pretraining in llm with dynamic discrete visual tokenization, 2024

    Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jian- chao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, Di Zhang, Wenwu Ou, Kun Gai, and Yadong Mu. Unified language-vision pretraining in llm with dynamic discrete visual tokenization, 2024. 1

    2024

  15. [15]

    Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023. 6

    2023

  16. [16]

    Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion, 2024

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion, 2024. 6, 7

    2024

  17. [17]

    Tokenpacker: Efficient visual projector for multimodal llm, 2024

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm, 2024. 2

    2024

  18. [18]

    Evaluating object hallucination in large vision-language models, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023. 6

    2023

  19. [19]

    Llama-vid: An image is worth 2 tokens in large language models, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023. 2

    2023

  20. [20]

    Boosting multimodal large language models with visual to- kens withdrawal for rapid inference, 2025

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual to- kens withdrawal for rapid inference, 2025. 2

    2025

  21. [21]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 1

    2023

  22. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 1

    2023

  23. [23]

    World model on million-length video and language with blockwise ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. InThe Thirteenth International Conference on Learning Representations, 2025. 1

    2025

  24. [24]

    Cheap and quick: Efficient vision- language instruction tuning for large language models, 2023

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision- language instruction tuning for large language models, 2023. 1, 2

    2023

  25. [25]

    Unitok: A uni- fied tokenizer for visual generation and understanding, 2025

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Ze- huan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A uni- fied tokenizer for visual generation and understanding, 2025. 1

    2025

  26. [26]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, and Chong Ruan. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation, 2024. 1

    2024

  27. [27]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 1

    2021

  28. [28]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. InAdvances in Neural Information Processing Systems (NeurIPS), 2021. 1, 2

    2021

  29. [29]

    High-resolution image syn- thesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2022. 1, 2

    2022

  30. [30]

    Llava-prumrge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388, 2024

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388,

    work page arXiv

  31. [31]

    Journeydb: A benchmark for generative im- age understanding.Advances in neural information process- ing systems, 36:49659–49678, 2023

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative im- age understanding.Advances in neural information process- ing systems, 36:49659–49678, 2023. 7

    2023

  32. [32]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024. 1, 2

    2024

  33. [33]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024. 1

    2024

  34. [34]

    Llama: Open and efficient foundation lan- guage models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation lan- guage models, 2023. 5

    2023

  35. [35]

    Neural discrete representation learning,

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning,

  36. [36]

    Emu3: Next-token prediction is all you need, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need, 2024. 1, 2

    2024

  37. [37]

    Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024. 1, 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [38]

    arXiv preprint arXiv:2412.04332 , year=

    Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Heng- shuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liq- uid: Language models are scalable and unified multi-modal generators.arXiv preprint arXiv:2412.04332, 2024. 1

    work page arXiv 2024

  39. [39]

    VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model inte- grating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024. 1, 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    Efficient streaming language models with attention sinks.arXiv, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv, 2023. 2

    2023

  41. [41]

    Show-o: One Single Transformer to Unify Multimodal Understanding and Generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024. 1, 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Scaling autoregressive multi-modal mod- els: Pretraining and instruction tuning, 2023

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Rus- sell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilm...

    2023

  43. [43]

    Anygpt: Unified multimodal llm with discrete sequence modeling, 2025

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu- Gang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal llm with discrete sequence modeling, 2025. 1

    2025

  44. [44]

    A-vl: Adaptive attention for large vision- language models, 2025

    Junyang Zhang, Mu Yuan, Ruiguang Zhong, Puhan Luo, Huiyou Zhan, Ningkang Zhang, Chengchen Hu, and Xi- angyang Li. A-vl: Adaptive attention for large vision- language models, 2025. 1, 2

    2025

  45. [45]

    Llava-mini: Efficient image and video large multimodal models with one vision token, 2025

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token, 2025. 2

    2025

  46. [46]

    Himix: Reducing computational com- plexity in large vision-language models, 2025

    Xuange Zhang, Dengjie Li, Bo Liu, Zenghao Bao, Yao Zhou, Baisong Yang, Zhongying Liu, Yujie Zhong, Zheng Zhao, and Tongtong Yuan. Himix: Reducing computational com- plexity in large vision-language models, 2025. 2, 5

    2025

  47. [47]

    Ar- gus: A compact and versatile foundation model for vision

    Weiming Zhuang, Chen Chen, Zhizhong Li, Sina Sajad- manesh, Jingtao Li, Jiabo Huang, Vikash Sehwag, Vivek Sharma, Hirotaka Shinozaki, Felan Carlo Garcia, et al. Ar- gus: A compact and versatile foundation model for vision. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4418–4429, 2025. 1

    2025