Pith. sign in

REVIEW 3 major objections 6 minor 96 references

Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Turning the user's question into visual prompts that are injected early into a frozen vision encoder improves multimodal LLM performance, especially on vision-centric benchmarks.

desk verdict Sensible, mostly solid empirical paper; the instruction-conditioning mechanism is under-supported because the decisive random-prompt control is missing, and one table-text mismatch needs fixing. read the letter →

arxiv 2411.13909 v2 pith:ZQCR6P5F submitted 2024-11-21 cs.CV

Honglin Li , Yuting Gao , Chenglu Zhu , Jingdong Chen , Ming Yang , Lin Yang This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsinstruction-guidedvisualpromptingprompttuningvision-centricbenchmarksmulti-turntokenpruningAmblyopiafrozenvisionencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most multimodal LLMs run the image through a frozen vision encoder and only let the user's question influence the language decoder, so the visual features can lose exactly the details the question asks about. This paper proposes Panther, which instead turns the question into visual prompts that are prepended to image tokens inside the vision encoder, so the question shapes what the encoder attends to from early layers. On the standard LLaVA-1.5 setup with Vicuna-7B, the method raises VQAv2 from 78.5 to 80.8, GQA from 62.0 to 65.2, and MMVP from 24.7 to 30.0; with Llama 3-8B it raises CV-Bench 3D from 61.9 to 68.9. The paper argues that instruction-guided visual prompting is an effective and largely orthogonal improvement that works across different vision encoders and language decoders without retraining the visual backbone.

What carries the argument

The load-bearing object is the instruction-aware visual prompt, written as $\Gamma_{T\to I}[F_T(q_k)]$: the current question $q_k$ is passed through a frozen CLIP text encoder and a two-layer MLP projector, then the resulting vectors are prepended to the visual tokens entering the frozen vision Transformer, alongside 24 shared learnable prompts. The shared prompts give the model a stable training slot, while the projected instruction prompts carry question content into the visual self-attention that computes the features the LLM later reads. To keep multi-turn training within the 2048-token budget, Panther-Bridge prunes per-turn visual tokens whose cosine similarity to the previous turn's tokens exceeds a threshold of $\tau = 0.95$.

What would settle it

Run the same training recipe with the instruction prompt replaced by a fixed, question-independent prompt of the same length and parameter budget; if VQAv2, GQA, and MMVP gains match the full model, the content of the question is not doing the work, while if the gains collapse, cross-modal prompt steering is what carries the result.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Amblyopia problem of encoder-decoder MLLMs, where visual details relevant to the user's instruction are filtered out before text and image meet, can be mitigated by injecting instruction information into the vision encoder itself. The concrete mechanism is an instruction-aware visual prompt: a frozen text encoder encodes the current question, a small MLP projects those embeddings into the visual token space, and the resulting prompt tokens are prepended to the image patch tokens inside a frozen ViT. Because the prompt is regenerated for every turn, the same image yields different visual features for different questions. The paper reports that this change, combined with a token-pruning bridge and interleaved decoder training, consistently beats the base model on general question answering and by larger margins on vision-centric benchmarks.

Load-bearing premise

The method rests on the assumption that a CLIP text embedding projected into visual space and prepended to a frozen ViT actually steers the visual features the LLM sees, rather than merely adding trainable capacity that would help regardless of the question text.

Editorial extensions

If this is right

  • Vision-centric error patterns caused by overly diffuse attention can be reduced by moving instruction fusion earlier, without changing image resolution or unfreezing the vision backbone.
  • The same image can legitimately produce different visual tokens for different questions, and those differences matter for answering correctly in multi-turn conversations.
  • Cosine-similarity pruning of per-turn visual tokens can cut multi-turn training cost while retaining most of the benefit of instruction-aware encoding.
  • The improvement transfers across vision encoders such as CLIP, DINOv2, and SigLIP and across LLMs such as OpenLlama-3B, Vicuna-7B, Mistral-7B, and Llama 3-8B, so the method is orthogonal to those architectural choices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's experiments, earlier instruction conditioning may also help Q-former-style connectors that currently cross-attend to visual features only at the final encoder layer, since the paper's result suggests that late conditioning is the failure point.
  • A testable extension would be to regenerate prompts from multi-turn conversational context rather than only the current question, since follow-up questions such as 'no, the other one' depend on earlier answers.
  • The token-pruning idea suggests a cheap way to decide when a follow-up question needs new visual information: turns whose prompted visual tokens are nearly identical to the previous turn's could skip re-encoding entirely, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Panther, an MLLM architecture whose central novelty is Panther-VE: user instructions are encoded by a frozen CLIP text encoder, projected by a small MLP into per-sample instruction-aware visual prompts (IP), and prepended to the patch tokens of a frozen ViT together with shared prompts (SP). A Panther-Bridge prunes redundant visual tokens across multi-turn training, and a Panther-Decoder uses interleaved autoregressive training. The authors evaluate on general VQA, instruction-following, and vision-centric benchmarks, reporting gains over LLaVA-1.5 on VQAv2, GQA, MMVP, and CV-Bench with both Vicuna-7B and Llama 3-8B backbones, and they ablate the SP/IP components, vision encoders, LLMs, and the pruning threshold.

Significance. If the instruction-conditioning mechanism is real, the paper makes a useful and largely orthogonal contribution: it offers a way to inject textual intent into a frozen visual encoder without multiple encoders or high-resolution inputs, and it does so with the same training data and base model as LLaVA-1.5. The experimental program is thorough in coverage: the method is tested across three vision encoders and four LLMs, and the controlled LLaVA-1.5 baseline makes the headline comparisons interpretable. The significance of the central claim, however, rests on whether the IP prompts actually carry instruction information into visual features rather than merely adding tunable capacity, and the paper's own ablation does not yet settle that question.

major comments (3)
  1. [Section 4.3 / Table 2] The text states 'Our Panther surpasses Cambrian-dev in all the 4 benchmarks,' but Table 2 shows a tie on MMVP for the Vicuna-7B configuration: Panther achieves 30.0 and Cambrian-dev achieves 30.0. The sentence should be corrected to 'matches or exceeds' or should explicitly say Panther exceeds Cambrian-dev on three of the four benchmarks.
  2. [Section 4.4 / Table 3] The ablation does not isolate instruction alignment from added capacity. On VQAv2, GQA, and VQA-T, the alignment-free BGE-b text encoder matches the CLIP text encoder within 1.0 point in both LLM settings (e.g., Vicuna-7B: 80.7 vs 80.8, 65.6 vs 65.2, 58.8 vs 59.6; Llama-3B: 79.0 vs 79.4, 63.8 vs 64.2, 52.2 vs 51.2). Since the IP branch adds a trained two-layer MLP and 77 per-sample tokens, the headline gains on VQAv2 and GQA are compatible with a pure-capacity explanation. The decisive missing control is 'w/ SP + per-sample random prompts' with matched token count and a trained projector; without it, the paper's own data support instruction injection only on MMVP, not on the larger benchmarks.
  3. [Section 4.4 / Table 3, MMVP] The only benchmark on which CLIP-based IP clearly beats BGE-based IP is MMVP, which contains 150 questions and is reported as a single run without variance. For Vicuna-7B the CLIP advantage over BGE is 2.0 points (30.0 vs 28.0), about 3 questions, which is within one binomial standard error; for Llama-3B the 7.4-point gap (22.7 vs 15.3) is roughly two standard errors. To make this result load-bearing, the authors should report multiple seeds or confidence intervals, and ideally also the random-prompt control requested above.
minor comments (6)
  1. [Section 3.2.1 / Eq. (4)] The manuscript does not specify at which ViT layers the SP and IP tokens are inserted; the equation only describes prepending to the visual embedding, while Figure 3(b) may be read as prompt insertion at every Transformer layer. Please state the insertion depth explicitly.
  2. [Section 3.2.2 / Algorithm 1] In the pseudocode for `prune_tokens`, the line `ref_token = cur_T['tensor'][j]` should be `ref_T['tensor'][j]`; as written, the reference token is taken from the current turn's tensor rather than from the reference turn.
  3. [Section 3.2.1 / Section 4.1] The method text says the lightweight text encoder is 'for instance, BERT [11]', but the implementation uses CLIP's text encoder; please make this consistent.
  4. [Section 4.4 / Token-Pruning Analysis] The text refers to 'Table 3.2.2' when presenting the pruning analysis; this should be Table 6.
  5. [References] Reference [81] is cited for BGE-base, but the given entry describes C-Pack rather than the BGE text embedding model; please cite the correct source.
  6. [Throughout] There are several typos and grammatical errors, including 'we does not make pruning' in Section 3.3, 'the remained texts' in Section 3.2.2, and 'pervious models' in Section 4.3; a language edit would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Panther's central claims are empirical results on external benchmarks, not consequences of its own definitions or fitted parameters.

full rationale

Panther is an empirical architecture paper. Its central claims—that instruction-aware visual prompts improve general VQA and vision-centric benchmarks—are substantiated by comparisons against LLaVA-1.5, Cambrian-dev, and other baselines on held-out benchmark suites (VQAv2, GQA, MMVP, CV-Bench, etc.). These benchmarks are not used to define the visual-prompt generator, the loss, or the architecture; the ViT is frozen and the instruction-prompt generator is trained with the standard supervised instruction-tuning objective. There is no equation in the paper in which an output quantity is defined in terms of the benchmark result it is said to predict, and no fitted parameter is renamed as a prediction. Hyperparameters such as the 24 shared prompts and the pruning threshold tau = 0.95 are selected empirically on the reported evaluation sets, which is a test-set selection concern rather than definitional circularity. The only self-references are related-work citations to the authors' prior prompt-learning papers, and those citations are not load-bearing for any architectural or empirical claim. The skeptic's point that the CLIP-versus-BGE ablation in Table 3 leaves an added-capacity confound is a valid experimental-control limitation, but it does not reduce any stated result to its inputs by construction: the reported numbers could have come out differently and are not forced by the method's definitions.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method introduces no new mathematical axioms. Its central claim rests on CLIP alignment assumptions, on the LLaVA-1.5 training recipe as a reproduced baseline, and on several hand-chosen hyperparameters (prompt counts, tau, learning rate). The only invented conceptual object is the named Amblyopia condition, which lacks independent quantitative evidence.

free parameters (4)
  • Number of shared prompts (sp) = 24
    Hand-chosen default; these learned vectors are applied across all samples and updated during instruction tuning, with no ablation varying the count.
  • Number of instruction-aware prompts (IP) = 77 with masking
    Set from CLIP text encoder maximum length and practical instruction length; masking details are not fully specified in the paper.
  • Pruning threshold tau = 0.95
    Selected from Table 6 as the best cost-performance balance in a threshold sweep (0.90, 0.95, 0.97, 1.00), so it is empirically fitted rather than derived.
  • Panther-VE base learning rate = 1e-4
    Used for the shared prompts and the instruction-prompt generator during instruction tuning; a hand-chosen training hyperparameter.
assumptions (4)
  • domain assumption CLIP image-text alignment transfers when text-derived vectors are prepended as visual prompts in a frozen ViT.
    Section 3.2.1 and Section 4.1: the IP generator uses a frozen CLIP text encoder and an MLP projector; the BGE text-encoder ablation in Table 3 shows smaller MMVP gains, confirming dependence on CLIP alignment.
  • domain assumption The LLaVA-1.5 pretraining and instruction-tuning datasets (558K and 665K samples) are sufficient and the reproduced baseline matches published LLaVA-1.5 numbers.
    Section 4.1: Panther is trained only during the instruction-tuning stage on the same data as LLaVA-1.5; the validity of every delta in Tables 1 and 2 depends on this reproduction.
  • domain assumption Cosine-similarity pruning during training retains enough visual information for the final model.
    Section 3.2.2 and Table 6: tokens with similarity above tau are dropped in multi-turn training, and Panther-Bridge is omitted at inference, so the assumption is that pruned training signal is still sufficient.
  • domain assumption Frozen ViT plus learned prompts is a valid substitute for visual encoder fine-tuning.
    Section 4.1: ViT parameters stay frozen while only prompts and the IP generator are updated; this is standard visual prompt tuning but is not independently justified in this paper.
invented entities (1)
  • Amblyopia (named visual impairment of MLLMs)
    purpose: Explains why late-fusion encoder-decoder MLLMs miss instruction-relevant visual details: text and image are fused only after the visual encoder has already produced features.
    Supported only by qualitative attention heatmaps in Figures 1 and 4; no quantitative benchmark isolates Amblyopia, so it is a framing device rather than an independently verified entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts." pith.science (2026). https://pith.science/paper/ZQCR6P5F

@misc{pith2026241113909,
  author       = {Pith},
  title        = {Pith review of: Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQCR6P5F}},
  note         = {Machine review of arXiv:2411.13909}
}
read the original abstract

Multimodal large language models (MLLMs) are closing the gap to human visual perception capability rapidly, while, still lag behind on attending to subtle images details or locating small objects precisely, etc. Common schemes to tackle these issues include deploying multiple vision encoders or operating on original high-resolution images. Few studies have concentrated on taking the textual instruction into improving visual representation, resulting in losing focus in some vision-centric tasks, a phenomenon we herein termed as Amblyopia. In this work, we introduce Panther, a MLLM that closely adheres to user instruction and locates targets of interests precisely, with the finesse of a black panther. Specifically, Panther comprises three integral components: Panther-VE, Panther-Bridge, and Panther-Decoder. Panther-VE integrates user instruction information at the early stages of the vision encoder, thereby extracting the most relevant and useful visual representations. The Panther-Bridge module, equipped with powerful filtering capabilities, significantly reduces redundant visual information, leading to a substantial savings in training costs. The Panther-Decoder is versatile and can be employed with any decoder-only architecture of LLMs without discrimination. Experimental results, particularly on vision-centric benchmarks, have demonstrated the effectiveness of Panther.

Figures

Figures reproduced from arXiv: 2411.13909 by the authors.

Figure 1
Figure 1. A comparative analysis of visual feature heatmaps be [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of MLLM Architectures: (a) Unified MLLM (decoder-only) enables early fusion of instructions and images, preserving low-level image features before fusion but lacks pretrained vision-language knowledge. (b) Typical MLLM (encoder-decoder) performs late fusion, leading to the Amblyopia issue where important visual details may be filtered out. (c) Our Panther MLLM convert instructions as visual prompts to gui… view at source ↗
Figure 3
Figure 3. The overall framework. (a) The Panther MLLM instruction tuning on multi-turn visual QA data, the visual tokens are generated [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons for representative scenarios. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 33 canonical work pages

  1. [1]

    Fuyu-8b: A multimodal architecture for ai agents

    Adept AI. Fuyu-8b: A multimodal architecture for ai agents. https://www.adept.ai/blog/fuyu-8b, 2024. 1

    2024

  2. [2]

    Flamingo: a vi- sual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a vi- sual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 1, 14

    arXiv 2022

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 2, 6

    arXiv 2023

  4. [4]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 14

    arXiv 2022

  5. [5]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 3

    arXiv 2005

  6. [6]

    Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Mak- ing large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pat- tern Recognition, 2024. 2, 3

    2024

  7. [7]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Sain- ing Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024. 14

    arXiv

  8. [8]

    Towards unifying medical vision-and-language pre-training via soft prompts, 2023

    Zhihong Chen, Shizhe Diao, Benyou Wang, Guanbin Li, and Xiang Wan. Towards unifying medical vision-and-language pre-training via soft prompts, 2023. 3

    2023

Show all 96 references
  1. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 6, 7

    2023

  2. [10]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 2, 6, 14

    2023 arXiv

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 3, 4

    2018 arXiv

  4. [12]

    Internlm-xcomposer2- 4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2- 4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.0...

    2024 arXiv

  5. [13]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2020. 3

    2020

  6. [14]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, et al. The llama 3 herd of models, 2024. 6, 7

    2024

  7. [15]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 1, 3, 6

    2023 arXiv

  8. [16]

    Domain adaptation via prompt learning

    Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. IEEE Transactions on Neural Networks and Learning Systems, 2023. 3

    2023

  9. [17]

    Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation. arXiv preprint arXiv:2404.14396, 2024. 2

    2024 arXiv

  10. [18]

    Openllama: An open reproduc- tion of llama, 2023

    Xinyang Geng and Hao Liu. Openllama: An open reproduc- tion of llama, 2023. 7

    2023

  11. [19]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–691...

    2017

  12. [20]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  13. [21]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  14. [22]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 1, 3, 6

    2019

  15. [24]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In ECCV, 2022. 3

    2022

  16. [25]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, L ´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, 9 T...

    2023

  17. [26]

    Enhancing multimodal large language models with vision detection models: An empirical study

    Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, and Ying Shen. Enhancing multimodal large language models with vision detection models: An empirical study. arXiv preprint arXiv:2401.17981, 2024. 2, 3

    2024 arXiv

  18. [27]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023. 3

    2023

  19. [28]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. arXiv preprint arXiv:2304.02643, 2023. 1, 3

    2023 arXiv

  20. [29]

    Spvit: Enabling faster vision transformers via soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. Spvit: Enabling faster vision transformers via soft token pruning. In ECCV, 2022. 14

    2022

  21. [30]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP,

  22. [31]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 1

    2023 arXiv

  23. [32]

    Task-specific fine-tuning via variational information bottle- neck for weakly-supervised pathology whole slide image classification

    Honglin Li, Chenglu Zhu, Yunlong Zhang, Yuxuan Sun, Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bottle- neck for weakly-supervised pathology whole slide image classification. In Proceedings of the IEEE/CVF Confer...

    2023

  24. [33]

    Rethinking transformer for long contextual histopathology whole slide image analysis, 2024

    Honglin Li, Yunlong Zhang, Pingyi Chen, Zhongyi Shui, Chenglu Zhu, and Lin Yang. Rethinking transformer for long contextual histopathology whole slide image analysis, 2024. 3

    2024

  25. [34]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1, 2, 3, 6

    2023 arXiv

  26. [35]

    Tokenpacker: Efficient visual projector for multimodal llm, 2024

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm, 2024. 6, 14

    2024

  27. [36]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL (Long Paper),

  28. [37]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 6

    2023 arXiv

  29. [38]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  30. [39]

    Not all patches are what you need: Expediting vision transformers via token reorganiza- tions

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganiza- tions. ICLR, 2022. 14

    2022

  31. [40]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 14

    2023 arXiv

  32. [41]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023. 6

    2023

  33. [42]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want,

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want,

  34. [43]

    Rethinking visual prompting for multimodal large language models with exter- nal knowledge

    Yuanze Lin, Yunsheng Li, Dongdong Chen, Weijian Xu, Ronald Clark, Philip Torr, and Lu Yuan. Rethinking visual prompting for multimodal large language models with exter- nal knowledge. arXiv preprint arXiv:2407.04681, 2024. 2, 3, 6

    2024 arXiv

  35. [44]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, 2023

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Ji- aming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hong- sheng Li, and Yu Qiao. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal l...

    2023

  36. [45]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 2, 6, 14

    2023

  37. [46]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 3, 14

    2024

  38. [47]

    Llava-1.6: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-1.6: Improved reasoning, ocr, and world knowledge, 2024. 14

    2024

  39. [48]

    World model on million-length video and language with blockwise ringattention, 2024

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention, 2024. 2, 6

    2024

  40. [49]

    P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL (Short Paper), 2022. 3

    2022

  41. [50]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 1, 3, 6

    2023 arXiv

  42. [51]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  43. [52]

    Learn to explain: Multimodal reasoning via 10 thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via 10 thought chains for science question answering. Advances in Neural Information Processing Systems, 2022. 6

    2022

  44. [53]

    An empirical study of scal- ing instruct-tuned large multimodal models

    Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jian- feng Gao, and Yelong Shen. An empirical study of scal- ing instruct-tuned large multimodal models. arXiv preprint arXiv:2309.09958, 2023. 1

    2023 arXiv

  45. [54]

    Visual percep- tion by large language model’s weights

    Feipeng Ma, Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Visual percep- tion by large language model’s weights. arXiv preprint arXiv:2405.20339, 2024. 14

    2024 arXiv

  46. [55]

    Token pooling in vision transformers

    Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, An- ish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers. arXiv:2110.03860 [cs.CV],

    arXiv

  47. [56]

    Chatgpt plugins

    OpenAI. Chatgpt plugins. https://openai.com/ blog/chatgpt-plugins, 2023. 1

    2023

  48. [57]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://cdn. openai.com/papers/GPTV_System_Card.pdf ,

  49. [58]

    V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, et al

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, et al. Dinov2: Learning robust visual features without supervision, 2023. 1, 3, 7

    2023

  50. [59]

    Less is more: Pay less attention in vision transform- ers

    Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jian- fei Cai. Less is more: Pay less attention in vision transform- ers. In AAAI, 2022. 14

    2022

  51. [60]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 3

    2019

  52. [61]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 6

    2021

  53. [62]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. arXiv preprint arXiv:2103.00020, 2021. 2, 3, 7

    2021 arXiv

  54. [63]

    Tokenlearner: Adaptive space-time tokenization for videos

    Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: Adaptive space-time tokenization for videos. In NeurIPS, 2021. 14

    2021

  55. [64]

    Autoprompt: Eliciting knowl- edge from language models with automatically generated prompts

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV , Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowl- edge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020. 3

    2010 arXiv

  56. [65]

    What does clip know about a red cir- cle? visual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and An- drea Vedaldi. What does clip know about a red cir- cle? visual prompt engineering for vlms. arXiv preprint arXiv:2304.06712, 2023. 3

    2023 arXiv

  57. [66]

    Unleashing the power of prompt-driven nu- cleus instance segmentation, 2024

    Zhongyi Shui, Yunlong Zhang, Kai Yao, Chenglu Zhu, Sunyi Zheng, Jingxiong Li, Honglin Li, Yuxuan Sun, Ruizhe Guo, and Lin Yang. Unleashing the power of prompt-driven nu- cleus instance segmentation, 2024. 3

    2024

  58. [67]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6

    2019

  59. [68]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 1, 2

    2024 arXiv

  60. [69]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024. 1

    2024

  61. [70]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 1, 3, 6

    2024 arXiv

  62. [71]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 1, 3, 6

    2024

  63. [72]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  64. [73]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3

    2017

  65. [74]

    What makes for good visual to- kenizers for large language models? arXiv preprint arXiv:2305.12223, 2023

    Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankan- halli, and Ying Shan. What makes for good visual to- kenizers for large language models? arXiv preprint arXiv:2305.12223, 2023. 3

    2023 arXiv

  66. [75]

    Tarsier: Recipes for training and evaluating large video description models, 2024

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models, 2024. 14

    2024

  67. [76]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024

  68. [77]

    Learning to prompt for con- tinual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jen- nifer Dy, and Tomas Pfister. Learning to prompt for con- tinual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139–149,

  69. [78]

    Mio: A foun- dation model on multimodal tokens, 2024

    Zekun Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jiashuo Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, and Wenhao Huang. Mio: A foun- dation model on multimodal tokens, 2024. 1

    2024

  70. [79]

    Vila-u: a unified foundation model inte- grating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model inte- grating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 1, 2, 6 11

    2024 arXiv

  71. [80]

    Grok-1.5 vision preview

    X.ai. Grok-1.5 vision preview. https://x.ai/blog/ grok-1.5v, 2024. 1, 3, 6

    2024

  72. [81]

    C-pack: Packaged resources to advance general chi- nese embedding, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muen- nighoff. C-pack: Packaged resources to advance general chi- nese embedding, 2023. 7

    2023

  73. [82]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528 , 2024. 1, 2, 6

    2024 arXiv

  74. [83]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 14

    2024 arXiv

  75. [84]

    Libra: Building decoupled vision system on large lan- guage models

    Yifan Xu, Xiaoshan Yang, Yaguang Song, and Changsheng Xu. Libra: Building decoupled vision system on large lan- guage models. In Forty-first International Conference on Machine Learning, 2024. 2

    2024

  76. [85]

    Efficient model personalization in federated learning via client-specific prompt generation

    Fu-En Yang, Chien-Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in federated learning via client-specific prompt generation. In 2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 19102–19111, 2023. 3

    2023

  77. [86]

    Minicpm-v: A gpt-4v level mllm on your phone, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. M...

    2024

  78. [87]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  79. [88]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 1, 3

    2023 arXiv

  80. [89]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1, 3, 7

    2023

  81. [90]

    Anygpt: Unified multimodal llm with dis- crete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with dis- crete sequence modeling. arXiv preprint arXiv:2402.12226,

    arXiv

  82. [91]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 14

    2023 arXiv

  83. [92]

    Long context transfer from language to vision, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision, 2024. 14

    2024

  84. [93]

    Treat visual tokens as text? but your mllm only needs fewer efforts to see

    Zeliang Zhang, Phu Pham, Wentian Zhao, Kun Wan, Yu-Jhe Li, Jianing Zhou, Daniel Miranda, Ajinkya Kale, and Chen- liang Xu. Treat visual tokens as text? but your mllm only needs fewer efforts to see. arXiv preprint arXiv:2410.06169,

    arXiv

  85. [94]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  86. [95]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2 12 Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guide...

    2023 arXiv

  87. [96]

    utilizes selective token attention, inactive head pruning, and layer dropping to streamline token processing. VLoRA

  88. [97]

    uniquely converts visual tokens into LLM weights, providing a novel approach to token efficiency. In video un- derstanding, AURORACAP [7] implement the token merg- ing [4] strategy, reducing the number of input visual tokens to address the overhead caused by lengthy video sequ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.