{"work":{"id":"74a7deb6-48be-4132-9d35-882cc5870ebd","openalex_id":"https://openalex.org/W4367367040","doi":"10.48550/arxiv.2304.14178","arxiv_id":"2304.14178","raw_key":null,"title":"mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality","authors":null,"authors_text":"Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al","year":2023,"venue":"cs.CL","abstract":"Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. We carefully build a visually-related instruction evaluation set OwlEval. Experimental results show that our model outperforms existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Besides, we observe some unexpected and exciting abilities such as multi-image correlation and scene text understanding, which makes it possible to leverage it for harder real scenarios, such as vision-only document comprehension. Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github.com/X-PLUG/mPLUG-Owl. The online demo is available at https://www.modelscope.cn/studios/damo/mPLUG-Owl.","external_url":"https://arxiv.org/abs/2304.14178","cited_by_count":168,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2304.14178","created_at":"2026-05-09T23:09:26.729790+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality","render_title":"mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality"},"hub":{"state":{"work_id":"74a7deb6-48be-4132-9d35-882cc5870ebd","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":84,"external_cited_by_count":168,"distinct_field_count":7,"first_pith_cited_at":"2023-03-28T17:59:12+00:00","last_pith_cited_at":"2026-07-02T17:50:57+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-22T17:59:26.132645+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":17},{"context_role":"baseline","n":3},{"context_role":"dataset","n":1},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"background","n":15},{"context_polarity":"baseline","n":3},{"context_polarity":"support","n":1},{"context_polarity":"unclear","n":1},{"context_polarity":"use_dataset","n":1},{"context_polarity":"use_method","n":1}],"runs":{"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-17T10:19:47.622021+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models","work_id":"a7e3a737-e007-42bc-be89-c4d34c5ee071","shared_citers":35},{"title":"Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond","work_id":"cbc2bb21-b6bb-46c0-80bf-107e195ffe10","shared_citers":25},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":24},{"title":"MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models","work_id":"806d2e73-71b3-4d56-87e0-39d571cc15d6","shared_citers":24},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":22},{"title":"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models","work_id":"63d03f4d-15f4-4583-8286-913c19f02294","shared_citers":20},{"title":"Otter: A Multi-Modal Model with In-Context Instruction Tuning","work_id":"33cb3a7a-6091-48db-a246-802bbb055f43","shared_citers":20},{"title":"Visual Instruction Tuning","work_id":"68be622d-a6dc-4a13-82de-e3054a3dc509","shared_citers":20},{"title":"MMBench: Is Your Multi-modal Model an All-around Player?","work_id":"3b44943d-0f15-4228-9ac3-0e376f4f9ada","shared_citers":19},{"title":"InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning","work_id":"f3aac728-ded0-4e55-aa9e-4a1635d4313d","shared_citers":17},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":16},{"title":"Improved Baselines with Visual Instruction Tuning","work_id":"5baeaa33-5986-44a3-85a4-fcabd6fc1e8d","shared_citers":15},{"title":"Kosmos-2: Grounding Multimodal Large Language Models to the World","work_id":"46e7f9e9-24c6-49af-b7d5-96159fa6f443","shared_citers":15},{"title":"Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic","work_id":"44525076-312a-4259-b79c-134cd7eeb297","shared_citers":14},{"title":"Gemini: A Family of Highly Capable Multimodal Models","work_id":"83f7c85b-3f11-450f-ac0c-64d9745220b2","shared_citers":13},{"title":"SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension","work_id":"23881ff0-b851-474c-8712-90744cc07a3a","shared_citers":13},{"title":"Evaluating Object Hallucination in Large Vision-Language Models","work_id":"66d8ac3e-c134-4995-b528-550afa17586f","shared_citers":12},{"title":"MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities","work_id":"7f3bac41-a0a5-4a7a-bfd2-526b616db745","shared_citers":12},{"title":"LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model","work_id":"0fe2cfd8-d442-4ceb-b1a9-a465704f39b2","shared_citers":11},{"title":"Qwen Technical Report","work_id":"bb1fd52f-6b2f-437c-9516-37bdf6eb9be8","shared_citers":11},{"title":"LoRA: Low-Rank Adaptation of Large Language Models","work_id":"0426219a-789e-4964-adc8-a04538510818","shared_citers":10},{"title":"Multimodal-gpt: A vision and language model for dialogue with humans","work_id":"e5fb1f2e-4ed2-454f-87a3-9e9c40f8fa31","shared_citers":10},{"title":"CogVLM: Visual Expert for Pretrained Language Models","work_id":"0d81fb99-dae6-46d2-8bed-c01dcbd7d7cf","shared_citers":9},{"title":"LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention","work_id":"5c44e6f5-82ca-4461-9fc1-d630f3bfa3e1","shared_citers":9}],"time_series":[{"n":19,"year":2023},{"n":14,"year":2024},{"n":2,"year":2025},{"n":12,"year":2026}],"dependency_candidates":[{"n":1,"role":"method","polarity":"use_method","paper_title":"Hallucination of Multimodal Large Language Models: A Survey","primary_cat":"cs.CV","context_text":"Contrastive Decodinge.g.VCD [94], IBD [226], ICD [166] Guided Decoding e.g.MARINE [212], GCD [38], DeCo [158] Visual Amplificatione.g.M3ID [41], IBD [226], AGLA [1] Others e.g.OPERA [66], Skip'\\n' [56] Visual Prompting e.g.SoM-LLaVA [179] RAG e.g.ARA [133], FilterRAG [141] Ensembling e.g.RITUAL [169], MAD [107], MVP [134] Post-hoc Correction e.g.Woodpecker [188], Volcano [93], LURE [224], VFC [45] Fig. 1. The main content flow and categorization of this survey. Preprint, Vol. 1, No. 1, Article . Publication date: April 2025. Hallucination of Multimodal Large Language Models: A Survey 5 Vision InputVision ModelLLMImageVideo…CLIP DINO-v2Linear…LLaMAVicunaChatGLMFuyu Decoding GreedyBeam SearchSamplingText InputInstruction…TokenizerBPE SentencePiece…","citing_arxiv_id":"2404.18930"},{"n":1,"role":"baseline","polarity":"baseline","paper_title":"SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension","primary_cat":"cs.CL","context_text":"MiniGPT-4 [7] Flan-T5-XL 47.40 4 29.89 7 42.84 4 VPGTrans [40] LLaMA-7B 41.81 5 31.40 5 39.10 5 MultiModal-GPT [12] LLaMA-7B 34.54 12 29.21 10 33.15 11 Otter [11] LLaMA-7B 35.16 11 30.35 6 33.91 8 OpenFlamingo [41] LLaMA-7B 34.51 13 29.25 9 33.14 12 LLaMA-Adapter V2 [42] LLaMA-7B 35.19 10 25.75 14 32.73 13 GVT [33] Vicuna-7B 35.49 9 27.77 12 33.48 10 mPLUG-Owl [9] LLaMA-7B 37.88 7 23.02 18 34.01 7 VideoLLM VideoChat [15] Vicuna-7B 39.02 6 33.68 4 37.63 6 Video-ChatGPT [16] LLaMA-7B 33.88 14 23.46 17 31.17 14 Valley [17] LLaMA-13B 32.04 15 25.41 15 30.32 15 3.4 Evaluation Strategy Different from MMBench [ 26] that employs ChatGPT to match a model's prediction to one of the choices in a multiple-choice question (achieves only 87.","citing_arxiv_id":"2307.16125"}]},"error":null,"updated_at":"2026-05-17T10:19:52.678862+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-17T10:19:52.293046+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"mplug-owl: Modularization empowers large lan- guage models with multimodality","claims":[{"claim_text":"Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-sta","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"MiniGPT-4 [7] Flan-T5-XL 47.40 4 29.89 7 42.84 4 VPGTrans [40] LLaMA-7B 41.81 5 31.40 5 39.10 5 MultiModal-GPT [12] LLaMA-7B 34.54 12 29.21 10 33.15 11 Otter [11] LLaMA-7B 35.16 11 30.35 6 33.91 8 OpenFlamingo [41] LLaMA-7B 34.51 13 29.25 9 33.14 12 LLaMA-Adapter V2 [42] LLaMA-7B 35.19 10 25.75 14 32.73 13 GVT [33] Vicuna-7B 35.49 9 27.77 12 33.48 10 mPLUG-Owl [9] LLaMA-7B 37.88 7 23.02 18 34.01 7 VideoLLM VideoChat [15] Vicuna-7B 39.02 6 33.68 4 37.63 6 Video-ChatGPT [16] LLaMA-7B 33.88 14 23.4","claim_type":"baseline","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Contrastive Decodinge.g.VCD [94], IBD [226], ICD [166] Guided Decoding e.g.MARINE [212], GCD [38], DeCo [158] Visual Amplificatione.g.M3ID [41], IBD [226], AGLA [1] Others e.g.OPERA [66], Skip'\\n' [56] Visual Prompting e.g.SoM-LLaVA [179] RAG e.g.ARA [133], FilterRAG [141] Ensembling e.g.RITUAL [169], MAD [107], MVP [134] Post-hoc Correction e.g.Woodpecker [188], Volcano [93], LURE [224], VFC [45] Fig. 1. The main content flow and categorization of this survey. Preprint, Vol. 1, No. 1, Article .","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"[46] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. [47] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. [48] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu,","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"weights, training data, code, and hyperparam- eters are proprietary. This limits the academic community's ability to conduct research on au- toregressive vision-language models, e.g., to un- derstand how web-scraped image-text data affects models' performance and safety. Open-source al- ternatives, such as LLaVA [25], LLaMA-Adapter [41], BLIP-2 [ 23], and mPLUG-Owl [ 39], only take in single images, and they often directly train on curated datasets like COCO [ 24] rather than web data. In this t","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"proficiency in image-text dialogues through pre-training alignment and instruction fine-tuning. Subsequent research [3, 6, 10, 25, 31, 43, 57] has further enhanced LMMs by fo- cusing on the quality and diversity of pretraining and fine- tuning data. For instance, LLaV A [31] and InstructBLIP [10], with improved instruction fine-tuning, have advanced the understanding of complex prompts. mPLUG-Owl [57], Shikra [6], and KOSMOS-2 [43] have introduced new data types and training techniques, like gro","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 2023. 6 [53] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, and Fei Huang. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model, 2023. 2 [54] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks mplug-owl: Modularization empowers large lan- guage models with multimodality because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (6 contexts).","role_counts":[{"n":6,"context_role":"background"},{"n":1,"context_role":"baseline"},{"n":1,"context_role":"method"}]},"error":null,"updated_at":"2026-05-17T10:19:52.682539+00:00"}},"summary":{"title":"mplug-owl: Modularization empowers large lan- guage models with multimodality","claims":[{"claim_text":"Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-sta","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"MiniGPT-4 [7] Flan-T5-XL 47.40 4 29.89 7 42.84 4 VPGTrans [40] LLaMA-7B 41.81 5 31.40 5 39.10 5 MultiModal-GPT [12] LLaMA-7B 34.54 12 29.21 10 33.15 11 Otter [11] LLaMA-7B 35.16 11 30.35 6 33.91 8 OpenFlamingo [41] LLaMA-7B 34.51 13 29.25 9 33.14 12 LLaMA-Adapter V2 [42] LLaMA-7B 35.19 10 25.75 14 32.73 13 GVT [33] Vicuna-7B 35.49 9 27.77 12 33.48 10 mPLUG-Owl [9] LLaMA-7B 37.88 7 23.02 18 34.01 7 VideoLLM VideoChat [15] Vicuna-7B 39.02 6 33.68 4 37.63 6 Video-ChatGPT [16] LLaMA-7B 33.88 14 23.4","claim_type":"baseline","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Contrastive Decodinge.g.VCD [94], IBD [226], ICD [166] Guided Decoding e.g.MARINE [212], GCD [38], DeCo [158] Visual Amplificatione.g.M3ID [41], IBD [226], AGLA [1] Others e.g.OPERA [66], Skip'\\n' [56] Visual Prompting e.g.SoM-LLaVA [179] RAG e.g.ARA [133], FilterRAG [141] Ensembling e.g.RITUAL [169], MAD [107], MVP [134] Post-hoc Correction e.g.Woodpecker [188], Volcano [93], LURE [224], VFC [45] Fig. 1. The main content flow and categorization of this survey. Preprint, Vol. 1, No. 1, Article .","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"[46] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. [47] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. [48] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu,","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"weights, training data, code, and hyperparam- eters are proprietary. This limits the academic community's ability to conduct research on au- toregressive vision-language models, e.g., to un- derstand how web-scraped image-text data affects models' performance and safety. Open-source al- ternatives, such as LLaVA [25], LLaMA-Adapter [41], BLIP-2 [ 23], and mPLUG-Owl [ 39], only take in single images, and they often directly train on curated datasets like COCO [ 24] rather than web data. In this t","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"proficiency in image-text dialogues through pre-training alignment and instruction fine-tuning. Subsequent research [3, 6, 10, 25, 31, 43, 57] has further enhanced LMMs by fo- cusing on the quality and diversity of pretraining and fine- tuning data. For instance, LLaV A [31] and InstructBLIP [10], with improved instruction fine-tuning, have advanced the understanding of complex prompts. mPLUG-Owl [57], Shikra [6], and KOSMOS-2 [43] have introduced new data types and training techniques, like gro","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 2023. 6 [53] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, and Fei Huang. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model, 2023. 2 [54] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks mplug-owl: Modularization empowers large lan- guage models with multimodality because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (6 contexts).","role_counts":[{"n":6,"context_role":"background"},{"n":1,"context_role":"baseline"},{"n":1,"context_role":"method"}]},"graph":{"co_cited":[{"title":"MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models","work_id":"a7e3a737-e007-42bc-be89-c4d34c5ee071","shared_citers":35},{"title":"Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond","work_id":"cbc2bb21-b6bb-46c0-80bf-107e195ffe10","shared_citers":25},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":24},{"title":"MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models","work_id":"806d2e73-71b3-4d56-87e0-39d571cc15d6","shared_citers":24},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":22},{"title":"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models","work_id":"63d03f4d-15f4-4583-8286-913c19f02294","shared_citers":20},{"title":"Otter: A Multi-Modal Model with In-Context Instruction Tuning","work_id":"33cb3a7a-6091-48db-a246-802bbb055f43","shared_citers":20},{"title":"Visual Instruction Tuning","work_id":"68be622d-a6dc-4a13-82de-e3054a3dc509","shared_citers":20},{"title":"MMBench: Is Your Multi-modal Model an All-around Player?","work_id":"3b44943d-0f15-4228-9ac3-0e376f4f9ada","shared_citers":19},{"title":"InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning","work_id":"f3aac728-ded0-4e55-aa9e-4a1635d4313d","shared_citers":17},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":16},{"title":"Improved Baselines with Visual Instruction Tuning","work_id":"5baeaa33-5986-44a3-85a4-fcabd6fc1e8d","shared_citers":15},{"title":"Kosmos-2: Grounding Multimodal Large Language Models to the World","work_id":"46e7f9e9-24c6-49af-b7d5-96159fa6f443","shared_citers":15},{"title":"Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic","work_id":"44525076-312a-4259-b79c-134cd7eeb297","shared_citers":14},{"title":"Gemini: A Family of Highly Capable Multimodal Models","work_id":"83f7c85b-3f11-450f-ac0c-64d9745220b2","shared_citers":13},{"title":"SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension","work_id":"23881ff0-b851-474c-8712-90744cc07a3a","shared_citers":13},{"title":"Evaluating Object Hallucination in Large Vision-Language Models","work_id":"66d8ac3e-c134-4995-b528-550afa17586f","shared_citers":12},{"title":"MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities","work_id":"7f3bac41-a0a5-4a7a-bfd2-526b616db745","shared_citers":12},{"title":"LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model","work_id":"0fe2cfd8-d442-4ceb-b1a9-a465704f39b2","shared_citers":11},{"title":"Qwen Technical Report","work_id":"bb1fd52f-6b2f-437c-9516-37bdf6eb9be8","shared_citers":11},{"title":"LoRA: Low-Rank Adaptation of Large Language Models","work_id":"0426219a-789e-4964-adc8-a04538510818","shared_citers":10},{"title":"Multimodal-gpt: A vision and language model for dialogue with humans","work_id":"e5fb1f2e-4ed2-454f-87a3-9e9c40f8fa31","shared_citers":10},{"title":"CogVLM: Visual Expert for Pretrained Language Models","work_id":"0d81fb99-dae6-46d2-8bed-c01dcbd7d7cf","shared_citers":9},{"title":"LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention","work_id":"5c44e6f5-82ca-4461-9fc1-d630f3bfa3e1","shared_citers":9}],"time_series":[{"n":19,"year":2023},{"n":14,"year":2024},{"n":2,"year":2025},{"n":12,"year":2026}],"dependency_candidates":[{"n":1,"role":"method","polarity":"use_method","paper_title":"Hallucination of Multimodal Large Language Models: A Survey","primary_cat":"cs.CV","context_text":"Contrastive Decodinge.g.VCD [94], IBD [226], ICD [166] Guided Decoding e.g.MARINE [212], GCD [38], DeCo [158] Visual Amplificatione.g.M3ID [41], IBD [226], AGLA [1] Others e.g.OPERA [66], Skip'\\n' [56] Visual Prompting e.g.SoM-LLaVA [179] RAG e.g.ARA [133], FilterRAG [141] Ensembling e.g.RITUAL [169], MAD [107], MVP [134] Post-hoc Correction e.g.Woodpecker [188], Volcano [93], LURE [224], VFC [45] Fig. 1. The main content flow and categorization of this survey. Preprint, Vol. 1, No. 1, Article . Publication date: April 2025. Hallucination of Multimodal Large Language Models: A Survey 5 Vision InputVision ModelLLMImageVideo…CLIP DINO-v2Linear…LLaMAVicunaChatGLMFuyu Decoding GreedyBeam SearchSamplingText InputInstruction…TokenizerBPE SentencePiece…","citing_arxiv_id":"2404.18930"},{"n":1,"role":"baseline","polarity":"baseline","paper_title":"SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension","primary_cat":"cs.CL","context_text":"MiniGPT-4 [7] Flan-T5-XL 47.40 4 29.89 7 42.84 4 VPGTrans [40] LLaMA-7B 41.81 5 31.40 5 39.10 5 MultiModal-GPT [12] LLaMA-7B 34.54 12 29.21 10 33.15 11 Otter [11] LLaMA-7B 35.16 11 30.35 6 33.91 8 OpenFlamingo [41] LLaMA-7B 34.51 13 29.25 9 33.14 12 LLaMA-Adapter V2 [42] LLaMA-7B 35.19 10 25.75 14 32.73 13 GVT [33] Vicuna-7B 35.49 9 27.77 12 33.48 10 mPLUG-Owl [9] LLaMA-7B 37.88 7 23.02 18 34.01 7 VideoLLM VideoChat [15] Vicuna-7B 39.02 6 33.68 4 37.63 6 Video-ChatGPT [16] LLaMA-7B 33.88 14 23.46 17 31.17 14 Valley [17] LLaMA-13B 32.04 15 25.41 15 30.32 15 3.4 Evaluation Strategy Different from MMBench [ 26] that employs ChatGPT to match a model's prediction to one of the choices in a multiple-choice question (achieves only 87.","citing_arxiv_id":"2307.16125"}]},"authors":[]}}