Pith. sign in

REVIEW 2 major objections 4 minor 142 references

Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective

T0 review · 2 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey claims that the training paradigm—single-stage, two-stage, or direct adaptation—determines both benchmark performance and parameter efficiency, with two-stage tuning scoring highest and direct adaptation using the fewest…

desk verdict A useful survey taxonomy and honest replication of Direct Adaptation, but the paradigm ranking is confounded by scale and evaluation protocol. read the letter →

arxiv 2502.01524 v1 pith:STFWCON6 submitted 2025-02-03 cs.CV cs.AIcs.CLcs.LG

Xiaorui Ma , Haoran Xie , S. Joe Qin This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords visionlargelanguagemodelsparameter-efficientadaptationtrainingparadigmsmodalityintegratorinstructiontuningmultimodalsurveydirecttwo-stage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey reviews 34 vision large language models and argues that their training pipeline can be sorted into three paradigms: single-stage tuning, two-stage tuning, and direct adaptation. It claims that two-stage tuning, which adds an instruction-tuning phase, delivers the highest benchmark performance, while direct adaptation, which skips pretraining and tunes only a modality integrator, is the most parameter-efficient and can match or beat full fine-tuning with far fewer trainable parameters. Single-stage tuning is the lightest historical approach but scores lowest on downstream tasks. The survey's point is that paradigm choice, not just architecture, shapes both accuracy and cost, and that efficient integrator design matters across all three.

What carries the argument

The load-bearing object is the Modality Integrator (MI), the trainable connector that maps visual embeddings from a frozen vision encoder into the LLM's semantic space, together with the three-paradigm training taxonomy. The MI comes in two families: Out-of-block Integrators (attention-based abstractors such as the Resampler and Q-Former, dense projectors, convolution-based abstractors, and VSS-based abstractors) that sit outside the LLM, and In-block Integrators (bottleneck adapters, attention-based adapters, degree-adaptive prefixes, and unimodal linear adapters) inserted into the LLM's transformer blocks. The training paradigm determines which parts of this system are updated, and the MI is what carries both the performance and the parameter-efficiency comparisons in the paper's central claim.

What would settle it

A controlled re-run of the same benchmarks with one base LLM repeated across all three paradigms, or a re-analysis of the paper's tables stratified by LLM size; if Single-stage Tuning or Direct Adaptation matches or beats Two-stage Tuning within matched backbones, the central ranking fails.

Watch

Extended reading notes

Core claim

In the paper's own framing, the central discovery is a ranking and a taxonomy. The evaluated paradigms order as Two-stage Tuning above Direct Adaptation above Single-stage Tuning on traditional benchmarks, and Direct Adaptation's modality-integrator designs match or exceed full fine-tuning while updating only a small fraction of parameters, typically under ten percent of a 220M T5 model. The taxonomy itself—three paradigms defined by whether pretraining, instruction tuning, and LLM updates occur—is offered as the organizing principle for choosing how to add vision to an LLM. The replicated Direct Adaptation experiments are presented as evidence that parameter efficiency need not come at the cost of competitive accuracy.

Load-bearing premise

The ranking assumes that score differences between paradigms come from the training paradigm itself, not from the base LLM's size and data; in the paper's comparison tables Two-stage models mostly use 7B LLaMA or Vicuna, while Direct Adaptation mostly uses 220M T5 or BART, so if base-model capability dominates, the ranking does not follow.

Editorial extensions

If this is right

  • Two-stage tuning, especially with instruction tuning, is the recipe that currently maximizes zero-shot visual question answering and captioning performance on the benchmarks surveyed.
  • Direct Adaptation shows that a model can beat full fine-tuning on average accuracy while updating under ten percent of parameters, which makes it a practical choice for compute-limited settings.
  • Efficient MI designs, such as bottleneck adapters and convolution-based abstractors, generalize across paradigms and become more important as base LLMs grow.
  • Single-stage tuning remains a parameter-lean baseline but is dominated by the other two paradigms on downstream generalization.
  • New attempts to bring instruction tuning into Direct Adaptation are a natural next step to close the performance gap with Two-stage Tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled comparison with the same base LLM across all three paradigms would likely change the numerical ranking; the paper's comparison confounds paradigm with model scale, so the true ordering of paradigms is not yet settled.
  • The parameter-efficiency results suggest a deployment path for on-device vision assistants: combine a small or quantized LLM with a direct-adaptation integrator, avoiding the pretraining stage entirely.
  • The taxonomy could be extended to audio or video modalities and to alignment stages such as reinforcement learning from human feedback, where the same question of which integrator and which training phase is most efficient reoccurs.
  • A direct testable extension would be to evaluate the same efficient MI designs from Direct Adaptation on 7B-class LLMs to see whether the parameter-efficiency advantage persists when the backbone is scaled up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper surveys 34 vision large language models (VLLMs) and organizes them into three training paradigms: Single-stage Tuning, Two-stage Tuning, and Direct Adaptation, with a focus on parameter-efficient adaptation. It introduces transformer and parameter-efficient fine-tuning preliminaries, reviews vision encoders and modality integrators, summarizes benchmarks, and compares the performance of representative models. As part of the comparison, the authors replicate experiments for the Direct Adaptation paradigm using official public repositories. The paper concludes that, among the evaluated paradigms, Two-stage Tuning achieves the highest performance, followed by Direct Adaptation, with Single-stage Tuning showing the lowest performance, and that Direct Adaptation primarily prioritizes parameter efficiency.

Significance. The paper has several genuine strengths: it applies explicit inclusion criteria for survey selection, offers a reasonably comprehensive taxonomy of training paradigms and modality integrators, reports parameter counts and trainable-parameter percentages in detail, and adds value by replicating Direct Adaptation methods rather than only citing their published numbers. The survey is also honest in stating that the reproduced results are single-run outcomes. If the headline ranking were established, the three-paradigm taxonomy would be a useful organizing principle for practitioners choosing a training pipeline. However, as explained in the major comments, the central performance ranking is not currently supported by the evidence presented, because the compared models differ systematically in base LLM scale, training data, and evaluation protocol. The paper is therefore best viewed at present as a descriptive taxonomy and repository of parameter-efficiency facts, with the comparative claims in need of substantial qualification.

major comments (2)
  1. [Section 6; Tables 7, 9, 11] The claim that 'Two-stage Tuning demonstrated the highest performance, followed by Direct Adaptation, with Single-stage Tuning showing the lowest performance' compares paradigms that are evaluated under very different base models, training data, and protocols. In Table 7, Single-stage rows are mostly zero-shot or few-shot evaluations of frozen models, Two-stage rows are predominantly instruction-tuned or per-task-finetuned 7B LLaMA/Vicuna models, and Direct Adaptation rows are mostly multi-task-finetuned T5-base/BART models. The underlined paradigm averages therefore conflate training paradigm with base-model scale, data scale, and evaluation protocol, so the ordering does not follow from the table. The table also contains an internal inconsistency: Frozen is listed with a 1.5B GPT2-like LLM in Table 7 but with a 7B LLM in Table 1. Moreover, the advanced benchmarks in Table 9 are reported only for Two-stage models, so they cannot support a three-way ranking. The conclusion should be rephrased as a descriptive observation about representative systems, or supported by evaluations in which base LLM, training data, compute budget, and evaluation protocol are held fixed across paradigms.
  2. [Section 5.4; Table 11] The statements that 'most models outperform full fine-tuning in terms of average performance' and that Direct Adaptation designs 'achieve an excellent balance between parameter efficiency and modality fusion performance' are not established by the replicated evidence. Section 5.4 explicitly states that the reproduced results are a single-run outcome, and Table 11 shows differences of the same scale as expected run-to-run variation: VL-Adapter averages 77.07 versus 77.40 for full fine-tuning, LST averages 77.92, and the VL-PET variants average 77.74 to 79.60. In addition, the MemVP figure is taken from the original paper as an average of three runs, so the comparison mixes one-run and multi-run protocols. Without multiple seeds or error bars, the paper can report point estimates but cannot support claims of outperformance over full fine-tuning. The parameter-efficiency comparison in the same table is also only partially consistent with the text: VL-Adapter trains 7.98% of parameters while LoRA trains 7.54%, so the statement that most MIs train fewer parameters than LoRA holds only for a subset of the row entries.
minor comments (4)
  1. [Section 5.1] The sentence that advanced benchmarks are widely used in instruction-tuned models 'within the Single-stage Tuning paradigm' is inconsistent with the cited models, which are all Two-stage Tuning models; this should be corrected.
  2. [Section 2; Section 5.2; Section 5.4; Table 8] Typos and inconsistent spacing should be cleaned up: 'Large Language Models (MML)' in Section 2, 'pertaining' for 'pretraining' in Sections 5.1 and 5.2, 'fintunes' in Section 5.4, and 'Convernsa-tion' in Table 8.
  3. [Table 7] The BLIP-2 row in Table 7 contains many unlabeled numeric entries, making it difficult to verify the underlined paradigm averages; the table should use a consistent per-benchmark column order or split configurations into separate rows.
  4. [References [15], [82]] References [15] and [82] share authors with this manuscript; the paper should include a self-citation or conflict-of-interest disclosure rather than leaving these citations unremarked.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's taxonomy and ranking are descriptive summaries of externally sourced benchmark results, with no derivation reducing to its own inputs.

full rationale

This paper is a survey rather than a derivation, and its central ranking claim in Section 6 is an empirical summary of benchmark tables whose entries are taken from the cited publications or official GitHub repositories. The paper itself states: 'the experiments for VLLMs following the Direct Adaptation paradigm are replicated, and those for other paradigms are sourced from their respective publications or GitHub repositories' (Section 5). The three-paradigm taxonomy is definitional, but the comparative conclusions are not entailed by the definitions; they depend on the collected results. The two self-citations ([15] and [82]) appear only as background support for generic statements about parameter efficiency and multi-task learning, and neither is load-bearing for any prediction or ranking. The unmatched base-model scales and the single-run nature of some replicated results are threats to the validity or strength of the empirical ranking, but they are not circularity: no fitted parameter is renamed as a prediction, and no claim reduces by construction to its own input. Therefore no circular step can be exhibited under the required standard.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey introduces no fitted constants, no new entities, and no original mathematical derivations. Its working assumptions are: parameter count measures efficiency, benchmark scores across different base models can be compared, and citation and venue screening selects a representative literature sample.

assumptions (3)
  • domain assumption Trainable parameter count is a valid proxy for adaptation efficiency and cost
    The survey compares paradigms mainly by percentage of trainable parameters and peak memory; inference compute, data collection, and pretraining cost are not included in the efficiency scores (Sections 5.2 to 5.4).
  • domain assumption Cross-model benchmark comparisons attribute score differences to training paradigm
    Tables 7, 9, and 11 compare models with different base LLM sizes and data scales; the ranking of paradigms in Section 6 assumes these differences do not swamp the paradigm effect.
  • domain assumption Citation and venue based inclusion criteria yield a representative sample
    Selection is by CCF rank or annual citation count, which is a popularity filter rather than a methodological coverage criterion (Section 1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective." pith.science (2026). https://pith.science/paper/STFWCON6

@misc{pith2026250201524,
  author       = {Pith},
  title        = {Pith review of: Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/STFWCON6}},
  note         = {Machine review of arXiv:2502.01524}
}
read the original abstract

The integration of vision-language modalities has been a significant focus in multimodal learning, traditionally relying on Vision-Language Pretrained Models. However, with the advent of Large Language Models (LLMs), there has been a notable shift towards incorporating LLMs with vision modalities. Following this, the training paradigms for incorporating vision modalities into LLMs have evolved. Initially, the approach was to integrate the modalities through pretraining the modality integrator, named Single-stage Tuning. It has since branched out into methods focusing on performance enhancement, denoted as Two-stage Tuning, and those prioritizing parameter efficiency, referred to as Direct Adaptation. However, existing surveys primarily address the latest Vision Large Language Models (VLLMs) with Two-stage Tuning, leaving a gap in understanding the evolution of training paradigms and their unique parameter-efficient considerations. This paper categorizes and reviews 34 VLLMs from top conferences, journals, and highly cited Arxiv papers, focusing on parameter efficiency during adaptation from the training paradigm perspective. We first introduce the architecture of LLMs and parameter-efficient learning methods, followed by a discussion on vision encoders and a comprehensive taxonomy of modality integrators. We then review three training paradigms and their efficiency considerations, summarizing benchmarks in the VLLM field. To gain deeper insights into their effectiveness in parameter efficiency, we compare and discuss the experimental results of representative models, among which the experiment of the Direct Adaptation paradigm is replicated. Providing insights into recent developments and practical uses, this survey is a vital guide for researchers and practitioners navigating the efficient integration of vision modalities into LLMs.

Figures

Figures reproduced from arXiv: 2502.01524 by the authors.

Figure 1
Figure 1. Integrated Modules and Three Training Paradigms. MI denotes modality integrator, and VE denotes vision encoder. The trainable module and learning paradigm are the most adopted. Existing surveys [14, 2], however, mainly focus on the latest VLLMs adopting Two-stage Tuning paradigms. In this survey, the database used is Google Scholar [42], and the keywords are Multimodal, Large Language model, vision-language model, a… view at source ↗
Figure 2
Figure 2. The Taxonomy and Publishing Time. AC denotes the annual citation times. For published work, the horizontal axis shows the Published time, while for unpublished work, it shows the submission time to Arxiv. 2 Large Language Model Large Language Models (MML) are mainly transformer-based [50] models with encoder [51, 52], encoder-decoder [53, 54] and decoder-only [55, 56, 57, 58, 59, 26, 60] architectures. LLMs utilized… view at source ↗
Figure 3
Figure 3. Taxonomy of Modality Integrator. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

142 extracted references · 44 canonical work pages

  1. [15]

    Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment

    Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148, 2023

    arXiv 2023

  2. [82]

    Multi-Task Learning with LLMs for Implicit Sentiment Analysis: Data-level and Task-level Automatic Weight Learning

    Wenna Lai, Haoran Xie, Guandong Xu, and Qing Li. Multi-task learning with llms for implicit sentiment analysis: Data-level and task-level automatic weight learning. arXiv preprint arXiv:2412.09046, 2024

    work page Pith review arXiv 2024

  3. [1]

    A survey of vision-language pre-trained models

    Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. In Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI) , pages 5436–5443. AAAI Press, 2022

    2022

  4. [2]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. Natl. Sci. Rev., 11, 2024

    2024

  5. [3]

    Openflamingo: An open-source framework for training large autoregressive vision-language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arX...

    arXiv 2023

  6. [4]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 12888–12900. JMLR, 2022

    2022

  7. [5]

    CoCa: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transact. Mach. Learn. Res., 2022

    2022

  8. [6]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 200–212. Curran Associates, Inc., 2021

    2021

Show all 142 references
  1. [7]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 19730–19742. JMLR, 2023. 20 Effi...

    2023

  2. [8]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the 38th Annual Conference on Neural Information Processing System (NeurIPS), volume 36, pages 34892–34916. Curran Associates, Inc., 2024

    2024

  3. [9]

    LLaMA-Adapter v2: Parameter-efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. LLaMA-Adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023

    2023 arXiv

  4. [10]

    Enhance reasoning ability of visual-language models via large language models

    Yueting Yang, Xintong Zhang, and Wenjuan Han. Enhance reasoning ability of visual-language models via large language models. arXiv preprint arXiv:2305.13267, 2023

    2023 arXiv

  5. [11]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of the 36th Annual Conference on Neural Inform...

    2022

  6. [12]

    MMBench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  7. [13]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  8. [14]

    Efficient multimodal large language models: A survey

    Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint arXiv:2405.10739, 2024

    2024

  9. [16]

    Flamingo: A visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. In Proceedings of the 36th Annual Conference on Neural Info...

    2022

  10. [17]

    Clipcap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021

    2021 arXiv

  11. [18]

    MAPL: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting

    Oscar Mañas, Pau Rodriguez Lopez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. MAPL: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting. In Proceedings of the 17th Conference of the European Chapter of the A...

    2023

  12. [19]

    Meta learning to bridge vision and language models for multimodal few-shot learning

    Ivona Najdenkoska, Xiantong Zhen, and Marcel Worring. Meta learning to bridge vision and language models for multimodal few-shot learning. In The 11th International Conference on Learning Representations (ICLR). ICLR, 2023

    2023

  13. [20]

    Grounding language models to images for multimodal inputs and outputs

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 17283–17300. JMLR, 2023

    2023

  14. [21]

    MiniGPT-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision- language understanding with advanced large language models. In Proceedings of the 12th International Conference on Learning Representations (ICLR). ICLR, 2024

    2024

  15. [22]

    CogVLM: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. CogVLM: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  16. [23]

    Zero-shot video question answering via frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 35, pages 124–141. C...

    2022

  17. [24]

    Video-LLaMA: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), pages 543–553. ACL, 2023

    2023

  18. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 26296–26306. IEEE, 2024. 21 Efficiently Integrate Large Language Models...

    2024

  19. [26]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  20. [27]

    mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  21. [28]

    mPLUG-Owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  22. [29]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InProceedings of the 10th International Conference on Learning Representations (ICLR). ICLR, 2022

    2022

  23. [30]

    MobileVLM: A fast, strong and open vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. MobileVLM: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023

    2023 arXiv

  24. [31]

    MobileVLM v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. MobileVLM v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024

    2024 arXiv

  25. [32]

    VL-Mamba: Exploring state space models for multimodal learning

    Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. VL-Mamba: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403.13600, 2024

    2024 arXiv

  26. [33]

    VL-Adapter: Parameter-efficient transfer learning for vision-and- language tasks

    Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VL-Adapter: Parameter-efficient transfer learning for vision-and- language tasks. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5227–5237. IEEE, 2022

    2022

  27. [34]

    eP-ALM: Efficient perceptual augmentation of language models

    Mustafa Shukor, Corentin Dancette, and Matthieu Cord. eP-ALM: Efficient perceptual augmentation of language models. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22056–22069. IEEE, 2023

    2023

  28. [35]

    Modular and parameter-efficient multimodal fusion with prompting

    Sheng Liang, Mengjie Zhao, and Hinrich Schuetze. Modular and parameter-efficient multimodal fusion with prompting. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2976–2985. ACL, 2022

    2022

  29. [36]

    Memory-space visual prompting for efficient vision-language fine-tuning

    Shibo Jie, Yehui Tang, Ning Ding, Zhi-Hong Deng, Kai Han, and Yunhe Wang. Memory-space visual prompting for efficient vision-language fine-tuning. In Proceedings of the 41th International Conference on Machine Learning (ICML). JMLR, 2024

    2024

  30. [37]

    LLaMA- Adapter: Efficient fine-tuning of large language models with zero-initialized attention

    Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMA- Adapter: Efficient fine-tuning of large language models with zero-initialized attention. In Proceedings of the 12th International Conference on Learning Representations (ICLR)...

    2024

  31. [38]

    LST: Ladder side-tuning for parameter and memory efficient transfer learning

    Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: Ladder side-tuning for parameter and memory efficient transfer learning. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 35, pages 12991–13005. Curran Associates, Inc., 2022

    2022

  32. [39]

    Querying as Prompt: Parameter-efficient learning for multimodal language model

    Tian Liang, Jing Huang, Ming Kong, Luyuan Chen, and Qiang Zhu. Querying as Prompt: Parameter-efficient learning for multimodal language model. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26855–26865, 2024

    2024

  33. [40]

    VL-PET: Vision-and-language parameter-efficient tuning via granularity control

    Zi-Yuan Hu, Yanyang Li, Michael R Lyu, and Liwei Wang. VL-PET: Vision-and-language parameter-efficient tuning via granularity control. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3010–3020. IEEE, 2023

    2023

  34. [41]

    Cheap and Quick: Efficient vision-language instruction tuning for large language models

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and Quick: Efficient vision-language instruction tuning for large language models. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 36. C...

    2024

  35. [42]

    https://scholar.google.com/, accessed 3 February 2025

    Google scholar. https://scholar.google.com/, accessed 3 February 2025

    2025

  36. [43]

    https://ccf.atom.im/, accessed 3 February 2025

    CCF recommended list of international conferences and periodicals. https://ccf.atom.im/, accessed 3 February 2025

    2025

  37. [44]

    MAGMA – Multimodal augmentation of generative models through adapter-based finetuning

    Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA – Multimodal augmentation of generative models through adapter-based finetuning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2416–2428. ACL, ...

    2022

  38. [45]

    Fusing pre-trained language models with multimodal prompts through reinforcement learning

    Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Rowan Zellers, Prithviraj Ammanabrolu, Ronan Le Bras, Gunhee Kim, and Yejin Choi. Fusing pre-trained language models with multimodal prompts through reinforcement learning. In Proceedings of the 202...

    2023

  39. [46]

    Aligning large multimodal models with factually augmented RLHF

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computationa...

    2024

  40. [47]

    Honeybee: Locality-enhanced projector for multimodal LLM

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal LLM. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13817–13827. IEEE, 2024

    2024

  41. [48]

    Tuning large multimodal models for videos using reinforcement learning from AI feedback

    Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Tuning large multimodal models for videos using reinforcement learning from AI feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 923–940. Asso...

    2024

  42. [49]

    X-LLM: Bootstrapping advanced large language models by treating multi-modalities as foreign languages

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-LLM: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160, 2023

    2023 arXiv

  43. [50]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates...

    2017

  44. [51]

    RoBERTa: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  45. [52]

    DeBERTa: Decoding-enhanced BERT with disentangled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. In Proceedings of the 9th International Conference on Learning Representations (ICLR). ICLR, 2021

    2021

  46. [53]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020

    2020

  47. [54]

    BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annu...

    2020

  48. [55]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  49. [56]

    VICUNA: An open-source chatbot impressing gpt-4 with 90% chatgpt quality

    The Vicuna Team. VICUNA: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. https: //lmsys.org/blog/2023-03-30-vicuna/ , March 2023

    2023

  50. [57]

    OPT: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  51. [58]

    GPT-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020

    2020

  52. [59]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  53. [60]

    ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  54. [61]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 23 Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective

    2023 arXiv

  55. [62]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. J. Mach. Learn. Res., 25(70):1–53, 2024

    2024

  56. [63]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the 34th Annual Conference on Neural Information Processing...

    1901

  57. [64]

    GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model [software]

    Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model [software]. https://github.com/kingoflolz/mesh-transformer-jax , May 2021

    2021

  58. [65]

    An empirical analysis of compute-optimal large language model training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Sim...

    2022

  59. [66]

    Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs [software]

    Mosaic AI Research. Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs [software]. https://www.databricks.com/blog/mpt-7b, May 2023

    2023

  60. [67]

    Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models [software]

    Together.ai. Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models [software] . https://www.together.ai/blog/redpajama-models-v1 , May 2023

    2023

  61. [68]

    LLaMA 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  62. [69]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  63. [70]

    Prefix-Tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL), pages 4582–4...

    2021

  64. [71]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045–3059. ACL, 2021

    2021

  65. [72]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages ...

    2019

  66. [73]

    QLoRA: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized llms. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 36, pages 10088–10115. Curran Associates, Inc., 2023

    2023

  67. [74]

    DoRA: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  68. [75]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of the 17th European Conference on Computer Vision (ECCV), pages 709–727. Springer, 2022

    2022

  69. [76]

    Exploring versatile generative language model via parameter- efficient transfer learning

    Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter- efficient transfer learning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 441–459. ACL, 2020

    2020

  70. [77]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In The 10th International Conference on Learning Representations (ICLR). ICLR, 2022

    2022

  71. [78]

    AdapterSoup: Weight averaging to improve generalization of pretrained language models

    Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, and Jesse Dodge. AdapterSoup: Weight averaging to improve generalization of pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2054–2063. ACL, 2023

    2023

  72. [79]

    Multi-task deep neural networks for natural language understanding

    Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504, 2019

    1901 arXiv

  73. [80]

    Multitask learning

    Rich Caruana. Multitask learning. Mach. Learn., 28:41–75, 1997. 24 Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective

    1997

  74. [81]

    A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods

    Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational...

    2023

  75. [83]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022

    2022

  76. [84]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proceedings of the 36th Annual Conference on Neural Inf...

    2022

  77. [85]

    Constitutional AI: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  78. [86]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  79. [87]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  80. [88]

    RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Co...

    2024

  81. [89]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference...

    2021

  82. [90]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  83. [91]

    High-performance large-scale image recognition without normalization

    Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 1059–1071. JMLR, 2021

    2021

  84. [92]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Pro- ceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12104–12113. IEEE, 2022

    2022

  85. [93]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the 2023 IEEE/CVF Conference on Computer V...

    2023

  86. [94]

    EV A-CLIP: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  87. [95]

    OpenCLIP [software], July 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP [software], July 2021. Zenodo

    2021

  88. [96]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pttern Recognition (CVPR), pages 770–778. IEEE, 2016

    2016

  89. [97]

    Linearly mapping from image to text space

    Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In Proceedings of the 11th International Conference on Learning Representations (ICLR). ICLR, 2023. 25 Efficiently Integrate Large Language Models with Visual Percep...

    2023

  90. [98]

    Deformable DETR: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In Proceedings of the 9th International Conference on Learning Representations (ICLR). ICLR, 2021

    2021

  91. [99]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1492–1500. IEEE, 2017

    2017

  92. [100]

    Conditional positional encodings for vision transformers

    Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In Proceedings of the 11th International Conference on Learning Representations (ICLR). ICLR, 2023

    2023

  93. [101]

    VQA: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pages 2425–2433. IEEE, 2015

    2015

  94. [102]

    OK-VQA: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the 2019 Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019

    2019

  95. [103]

    GQA: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700–6709. IEEE, 2019

    2019

  96. [104]

    Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. VizWiz Grand Challenge: Answering visual questions from blind people. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)...

    2018

  97. [105]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8309–8318. IEEE, 2019

    2019

  98. [106]

    Openimages: A public dataset for large-scale multi-label and multi-class image classification [dataset], 2017

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification [dataset], 2017

    2017

  99. [107]

    OCR-VQA: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Visual question answering by reading text in images. In Proceedings of the 15th International Conference on Document Analysis and Recognition (ICDAR), pages 947–952. IEEE, 2019

    2019

  100. [108]

    Judging a book by its cover

    Brian Kenji Iwana, Syed Tahseen Raza Rizvi, Sheraz Ahmed, Andreas Dengel, and Seiichi Uchida. Judging a book by its cover. arXiv preprint arXiv:1610.09204, 2016

    2016 arXiv

  101. [109]

    Microsoft COCO Captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO Captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  102. [110]

    CIDEr: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575. IEEE, 2015

    2015

  103. [111]

    BLEU: A method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL), pages 311–318. ACL, 2002

    2002

  104. [112]

    Nocaps: Novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 89...

    2019

  105. [113]

    SPICE: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic propositional image caption evaluation. In Proceedings of the 14th European Conference on Computer Vision (ECCV), pages 382–398. Springer, 2016

    2016

  106. [114]

    Guiding the long-short term memory model for image caption generation

    Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars. Guiding the long-short term memory model for image caption generation. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), page 2407–2415. IEEE Computer Society, 2015

    2015

  107. [115]

    A corpus of natural language for visual reasoning

    Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 217–223. ACL, 2017. 26 Efficiently Integrate Large Language Models with...

    2017

  108. [116]

    SEED-Bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. SEED-Bench: Benchmarking multimodal large language models. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13299–13308. IEEE, 2024

    2024

  109. [117]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

    2023

  110. [118]

    MM-Vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR, 2024

    2024

  111. [119]

    From recognition to cognition: Visual commonsense reasoning

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6713–6724. IEEE, 2019

    2019

  112. [120]

    Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the 2017 IEEE conference...

    2017

  113. [121]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 2023

    2023

  114. [122]

    Q-Bench: A benchmark for general-purpose foundation models on low-level vision

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. Q-Bench: A benchmark for general-purpose foundation models on low-level vision. In Proceedings of the 12th International Conference on ...

    2024

  115. [123]

    LAION-5B: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. L...

    2022

  116. [124]

    LAION-400M: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021

    2021 arXiv

  117. [125]

    COYO-700M: Image-text pair dataset [dataset]

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. COYO-700M: Image-text pair dataset [dataset]. https://github.com/kakaobrain/coyo-dataset/, 2022

    2022

  118. [126]

    Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2556–2565. ACL, 2018

    2018

  119. [127]

    Conceptual 12M: Pushing web-scale image- text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image- text pre-training to recognize long-tail visual concepts. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558–3568. ...

    2021

  120. [128]

    Im2Text: Describing images using 1 million captioned photographs

    Vicente Ordonez, Girish Kulkarni, and Tamara L Berg. Im2Text: Describing images using 1 million captioned photographs. In Proceedings of the 24th International Conference on Neural Information Processing Systems (NeurIPS), page 1143–1151. Curran Associates Inc., 2011

    2011

  121. [129]

    Visual Genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yan- nis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis, 123:32–73, 2017

    2017

  122. [130]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems (NeurIPS), vol...

    2020

  123. [131]

    The prompt report: A systematic survey of prompting techniques

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, et al. The prompt report: A systematic survey of prompting techniques. arXiv preprint arXiv:2406.06608, 2024. 27 Efficiently ...

    2024 arXiv

  124. [132]

    ShareGPT4V: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4V: Improving large multi-modal models with better captions. In Proceedings of the 2024 European Conference on Computer Vision (ECCV). Springer, 2024

    2024

  125. [133]

    ShareGPT

    Dom Eccleston and Steven Tey. ShareGPT. https://sharegpt.com/, December 2023

    2023

  126. [134]

    Slimorca: An open dataset of GPT-4 augmented FLAN reasoning traces, with verification [dataset], 2023

    Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Slimorca: An open dataset of GPT-4 augmented FLAN reasoning traces, with verification [dataset], 2023

    2023

  127. [135]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford_alpaca, 2023

    2023

  128. [136]

    Baize: An open-source chat model with parameter- efficient tuning on self-chat data

    Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter- efficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6268–6278. ACL, 2023

    2023

  129. [137]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  130. [138]

    Visual dialog

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 326–335. IEEE, 2017

    2017

  131. [139]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  132. [140]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In Proceedings of the 20th International Conference on Learning Representations (ICLR), 2024

    2024

  133. [141]

    LLaV AR: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLaV AR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023

    2023 arXiv

  134. [142]

    Multimodal-GPT: A vision and language model for dialogue with humans

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-GPT: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023. 28

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.