Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques

Jeoungeun Lee; Jisu An; Junseok Lee; Yongseok Son

REVIEW 2 major objections 5 minor 13 references

Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the full variety of multimodal-LLM integration can be organized by four contextual fusion mechanisms, three fusion levels, three representation paradigms, and three training paradigms.

desk verdict Useful survey of MLLM fusion with a four-mechanism taxonomy, but the categories are under-specified: the same Q-former module lands in three different categories with no stated decision rule, so the 'complete framework' claim is stronger than the evidence. read the letter →

arxiv 2506.04788 v1 pith:WMMCU4Y6 submitted 2025-06-05 cs.CL cs.AIcs.LG

Jisu An , Junseok Lee , Jeoungeun Lee , Yongseok Son This is my paper

classification cs.CLcs.AIcs.LG

keywords multimodallargelanguagemodelsmodalityfusiontaxonomyintegrationmechanismsprojectionlayercross-attentionQ-formertrainingparadigms

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal large language models connect pre-trained encoders to an LLM backbone, and this paper tries to put every integration strategy on a single map. It claims that the diversity of architectures built between 2021 and 2025 can be fully described by four contextual fusion mechanisms—projection, abstraction, semantic embedding, and cross-attention—combined with fusion level (early, intermediate, hybrid), representation paradigm (joint, coordinated, hybrid), and training paradigm (single-, two-, multi-stage). The paper reads 125 MLLM papers and classifies 50 of them in a comparison table, arguing that the same component (for example a Q-former) can play different functional roles depending on the designer's intent. The payoff for a reader is a vocabulary for saying what an integration component is for, not just what it is built from, plus a claim that no earlier survey covers all of these dimensions together.

What carries the argument

The carrying object is the four-way taxonomy of contextual fusion mechanisms, defined by what a module does between a modality encoder and the language embedding space. Projection maps features into the LLM's embedding space using linear layers, MLPs, transformers, or Q-formers; abstraction compresses variable or numerous tokens into a fixed, smaller set using Perceiver Resamplers, Q-formers, C-Abstractors, or convolutions; semantic embedding injects high-level meaning, often through instruction-conditioned learnable queries such as Q-former variants; cross-attention lets the LLM attend to non-text features inside its own transformer blocks or in added layers. These mechanisms are crossed with fusion level—early (before the LLM), intermediate (inside LLM layers), or hybrid—and the resulting grid organizes the survey's at-a-glance comparison of 50 architectures.

What would settle it

Inspect Table 3's mechanism labels against the original papers: if any listed model's authors explicitly state that their Q-former or Perceiver Resampler is designed purely to cut token count with no semantic or instructional role while Table 3 labels it Semantic Embedding, or if the table labels the same component type inconsistently for models whose papers describe identical purposes, the contextual-mechanism claim fails. A smaller falsifier is one published MLLM from 2021 to 2025 whose integration module serves none of the four mechanisms, such as a purely generative-modality bridge with no projection, abstraction, semantic injection, or attention.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that modality integration in MLLMs is best understood LLM-centrically: every non-text modality enters the language model through one of four contextual mechanisms, and which mechanism a component implements can only be decided by its intended function. The survey therefore classifies 125 MLLM papers from 2021 to 2025 along architectural strategies (mechanism plus fusion level), representation learning (joint, coordinated, or hybrid), and training methodology (number of stages and objective function), and it tabulates 50 representative architectures. The paper further claims that previous surveys either merged these dimensions, used a rigid component-name classification that misdescribes components with multiple roles, or omitted representation and training dimensions entirely, and that its four-dimension framework supplies the missing integrated view.

Load-bearing premise

The taxonomy stands on the assumption that a model's integration mechanism can be read reliably from its original paper, meaning that the authors' classification of, say, a Q-former as semantic embedding rather than abstraction matches what the designers actually built and intended; the paper's own limitations section admits that many papers never state these design motivations and that models with unclear details were excluded.

Editorial extensions

If this is right

Component names stop being reliable guides: a Q-former can serve as an abstraction layer, a projection layer, or a semantic embedding layer depending on design intent, so future papers should state that intent explicitly.
A model's position in the mechanism-by-fusion-level grid makes its efficiency and grounding trade-off visible: early fusion shortens sequence length, intermediate fusion adds dynamic token-level grounding, and hybrid fusion attempts both.
The training-paradigm axis, combined with objective type (language modeling, contrastive, or reconstruction), gives a recipe for replicating design choices, such as aligning a frozen LLM first and then instruction tuning.
The comparison with earlier surveys implies that a complete account of MLLM design should treat semantic mechanisms, LLM-relative fusion level, representation learning, and training paradigms as four separate axes rather than collapsing them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: have independent readers label the same 50 architectures from the original papers using the four mechanism categories; low agreement would confirm the paper's own caveat that design intent is often unstated and that classification is interpretive.
The completeness claim carries a prediction: if the taxonomy is truly exhaustive, then any new MLLM integration module should fit into one of the four mechanism cells, and the appearance of a genuinely fifth mechanism would refute that completeness.
The survey's scope excludes indirect integration through intermediate generative models with shared loss functions, so the completeness claim applies to direct LLM-modality integration even though the framework is presented as general.
A practical consequence the authors leave implicit is that the taxonomy doubles as a design checklist: a builder can choose a mechanism by the function they want, such as token-count control versus dynamic attention, rather than by reusing a familiar component name.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my take on arXiv:2506.04788. It's a survey of MLLM integration proposing four contextual fusion mechanisms (projection, abstraction, semantic embedding, cross-attention) plus fusion level, representation, and training paradigms. The best parts: clear definitions, a 50-model comparison table, and an honest limitations section. The survey covers 125 papers from 2021-2025 and gives a useful organizer for newcomers. The training-section and timeline are helpful.

The main problem is the taxonomy's reproducibility. The paper itself says the same components can serve different functional roles depending on intent, and it classifies Q-former as abstraction layer in Section 3.1.1, semantic embedding in Section 3.1.3, and projection in Table 3. No decision rule is given—what makes a Q-former abstraction vs semantic embedding vs projection? The authors acknowledge in Section 8 that many papers don't state design motivations and they excluded unclear models. That's honest, but it undercuts the claim that the four mechanisms give a complete and useful description. A different annotator could produce different Table 3 rows. The stress-test note makes this point, and I think it holds up.

Several smaller inconsistencies: the abstract lists only joint and coordinate representations while Section 4 introduces three (adds hybrid). Section 3.1.4 says modern LLMs already contain cross-attention layers, which is wrong for decoder-only backbones like LLaMA—they have self-attention. LLaMA-Adapter V2 is listed under both projection and cross-attention, which is fine if multiple mechanisms can co-occur, but the paper doesn't say that explicitly.

Novelty is modest. The four-mechanism taxonomy is a refinement of Song et al.'s convertor/perceiver split and other earlier surveys, as the authors acknowledge. The combination of four dimensions in one survey is new, and that's the contribution.

Bottom line: this is a genuinely useful survey for practitioners who want a structured overview and a quick reference table. The flaws are real but reparable—add an operational decision rule for each mechanism, fix the abstract, correct the cross-attention claim. I'd send it to peer review; a good referee could push these fixes. I would cite it if I needed a survey reference on MLLM integration, but I'd be careful about relying on any specific row in Table 3.

Referee Report

2 major / 5 minor

Summary. The paper proposes an LLM-centric taxonomy for multimodal large language model (MLLM) integration, organizing architectural strategies into four fusion mechanisms (projection, abstraction, semantic embedding, cross-attention), three fusion levels (early, intermediate, hybrid), three representation paradigms (joint, coordinated, hybrid), and three training paradigms (single-stage, two-stage, multi-stage). The survey reviews 125 MLLMs published between 2021 and 2025, provides a detailed comparison table (Table 3), a timeline of model developments (Figure 3), and a comparison with prior surveys (Table 2). The central claim is that this framework comprehensively and usefully describes MLLM integration and fills a gap in the existing survey literature.

Significance. If the taxonomy were made operational and reproducible, the survey would provide a valuable structured resource for researchers selecting integration strategies for MLLMs. The paper's strengths include its broad coverage of 125 models, the multi-dimensional comparison table, the explicit discussion of training objectives, and the attempt to explain why the same architectural component can serve different functions across models. The principal weakness is that the four mechanism categories lack operational definitions: the same module (Q-former) is classified under projection, abstraction, and semantic embedding in different parts of the paper, and no decision rule is given. This undermines the central claim of a complete and reproducible taxonomy, although the issue is addressable with added formal criteria.

major comments (2)

[§3.1.1, §3.1.3, Table 3] The Q-former module is assigned to three different categories without a decision rule. In §3.1.1 it is listed as an abstraction layer; in §3.1.3 it is listed as a semantic embedding layer; Table 3 classifies BLIP-2's Q-former as Semantic Embedding, Video-LLaMA's as Abstractor, and X-InstructBLIP's Q-former+linear as Projection. No operational criterion (e.g., output token count, presence of instruction tokens, use of contrastive pretraining, or whether the output is subsequently projected) is given to determine which category applies to a given model. Section 8 concedes that many original papers do not state design motivations, so the assignments rest on the authors' reading of intent. As a result, Table 3 is not reproducible: a different annotator could produce different classifications, and the claimed exhaustive and mutually exclusive mechanism taxonomy is not yet well-defined. Please specify explicit, externally checkable criteria for each mechanism and re-apply them consistently across the text and tables.
[§3.1.4, Table 3] The claim that "most modern LLMs have transformer structures, they already contain cross-attention layers" is incorrect for decoder-only backbones such as LLaMA and Vicuna, which contain self-attention blocks but not cross-attention. The table's "Within Model" entries also mix distinct architectural interventions: Flamingo inserts new gated cross-attention layers, CogVLM expands internal QKV matrices, LLaMA-Adapter V2 modifies self-attention with zero-initialized adapters, and mPLUG-Owl3 uses hyper-attention blocks. Without an operational criterion for what counts as a "cross-attention layer," this category is not mutually exclusive with the others (e.g., LLaMA-Adapter V2 appears under both Projection and Cross-attention in Table 3). Please clarify whether the category refers to any attention-based fusion where queries come from one modality and keys/values from another, and specify how inserted, modified, and pre-existing variants are distinguished under that definition.

minor comments (5)

[Abstract, §1, §4] The abstract and Section 1 describe representation learning as a binary choice ("joint or coordinate"), while Section 4 defines three classes (Joint, Coordinated, Hybrid). Please align the abstract and introduction with the three-way taxonomy used in the body.
[Table 3] Table 3 contains a duplicate row for RoboFlamingo with inconsistent entries: one row lists "Cross-attention(within Model)" under the cross-attention column, while the other lists "Within Model" under the fusion-level column. Please remove the duplicate and verify the intended classification.
[Table 2/Appendix A] The coverage comparison in Table 2 assigns "✓", "×", and "∼" based on the authors' own reading of the nine surveys. It would be helpful to state how "partial/different" (∼) was judged and to make the comparison reproducible by citing the specific sections of each prior survey that were examined.
[§5.4] The text refers to "DICE loss"; the conventional spelling is "Dice loss". Please correct this and any related occurrences.
[Throughout] There are several typographical errors, including "Limiation" (Section heading), "full finetuing" (Section 2), and inconsistent capitalization of "Q-former"/"Q-Former" across the text. A careful proofreading pass is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's taxonomy is an interpretive literature organization with no derived predictions or fitted inputs.

full rationale

This paper is a survey and taxonomy rather than a derived result. It makes no quantitative predictions, fits no parameters, and cites no prior work by its own authors as load-bearing evidence. The central claim—that four fusion mechanisms, three fusion levels, three representation paradigms, and three training paradigms organize 125 MLLMs—is the paper's input framework, not a result derived from itself. The comparison with prior surveys in Appendix A is a judgment call: Table 2 marks coverage of 'our proposed classification dimensions,' so the perceived coverage gap is partly self-referential in framing, but the underlying factual assertions (e.g., that prior surveys do not explicitly classify semantic integration mechanisms or representation learning approaches) are checkable against those cited surveys and are not forced by the taxonomy's definitions alone. The skeptic's objection that the Q-former is placed in multiple categories without a decision rule, and that Section 8 concedes many papers do not state design motivations, is a reproducibility or validity concern about the annotation scheme, not circularity: the categories were not defined in terms of the table entries, and the table entries do not entail the framework. No quoted passage exhibits an equation or definition that reduces a claimed result to its own inputs. Under the hard rule requiring a specific exhibited reduction, this paper receives no circularity deductions.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numerical parameters or invented entities are involved. The taxonomy's axioms are the assumed exhaustiveness and orthogonality of its categories, together with the methodological assumption that excluding ambiguous models does not bias the conclusions.

assumptions (3)

ad hoc to paper The four fusion mechanisms (projection, abstraction, semantic embedding, cross-attention) are exhaustive and mutually exclusive.
Section 3.1 presents these as the complete set of contextual fusion mechanisms without a formal proof or a systematic search argument for other mechanisms.
domain assumption Models with ambiguous architectural descriptions can be excluded without affecting the validity of the proposed taxonomy.
Section 8 excludes such models, assuming the remainder is representative.
domain assumption The evaluation of prior surveys in Table 2 accurately reflects their coverage.
The claim that no prior survey covers all four dimensions depends on the authors' own coding of the nine surveys in Table 2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques." pith.science (2026). https://pith.science/paper/WMMCU4Y6

@misc{pith2026250604788,
  author       = {Pith},
  title        = {Pith review of: Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WMMCU4Y6}},
  note         = {Machine review of arXiv:2506.04788}
}

read the original abstract

The rapid progress of Multimodal Large Language Models(MLLMs) has transformed the AI landscape. These models combine pre-trained LLMs with various modality encoders. This integration requires a systematic understanding of how different modalities connect to the language backbone. Our survey presents an LLM-centric analysis of current approaches. We examine methods for transforming and aligning diverse modal inputs into the language embedding space. This addresses a significant gap in existing literature. We propose a classification framework for MLLMs based on three key dimensions. First, we examine architectural strategies for modality integration. This includes both the specific integration mechanisms and the fusion level. Second, we categorize representation learning techniques as either joint or coordinate representations. Third, we analyze training paradigms, including training strategies and objective functions. By examining 125 MLLMs developed between 2021 and 2025, we identify emerging patterns in the field. Our taxonomy provides researchers with a structured overview of current integration techniques. These insights aim to guide the development of more robust multimodal integration strategies for future models built on pre-trained foundations.

Figures

Figures reproduced from arXiv: 2506.04788 by the authors.

**Figure 1.** Proposed contextual fusion mechanisms full finetuing, making multimodal system development more efficient (Touvron et al., 2023; Peng et al., 2023a; Shukang Yin et al., 2023). Feature Encoder MLLMs use specialized encoders to convert modality inputs into vector representations. For images, Vision Transformers(ViT) or convolutional networks extract visual features (Krizhevsky et al., 2012; He et al., 2016; Dosovi… view at source ↗

**Figure 2.** Proposed a taxonomy of LLM-centric fusion level(Early, Intermediate, and Hybrid) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Timeline of major developments in multimodal architectures [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages

[2]

InMachine Learning for Health, ML4H@NeurIPS 2023, 10 December 2023, New Or- leans, Louisiana, USA, volume 225 ofProceedings of Machine Learning Research, pages 353–367

Med-Flamingo: A Multimodal Medical Few- shot Learner. InMachine Learning for Health, ML4H@NeurIPS 2023, 10 December 2023, New Or- leans, Louisiana, USA, volume 225 ofProceedings of Machine Learning Research, pages 353–367. PMLR. Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. 2023. Embod...

arXiv 2023
[4]

In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13052– 13062, Seattle, W A, USA

Sniffer: Multimodal Large Language Model for Explainable Out-of-Context Misinformation De- tection. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13052– 13062, Seattle, W A, USA. IEEE. Alec Radford. 2018. Improving language understanding with unsupervised learning.OpenAI Res. Alec Radford, Jong Wook Kim, Chris Hallacy...

arXiv 2018
[5]

InIEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 15120–15130

LMDrive: Closed-Loop End-to-End Driving with Large Language Models. InIEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 15120–15130. IEEE. Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh

work page 2024
[6]

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen

Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models.Preprint, arXiv:2307.14539. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models.National Science Review. Gowthami Somepalli, Arkabandhu Chowdhury, Jonas Geiping, Ronen Basri, Tom Goldstein, an...

arXiv 2023
[8]

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang

SEAL: Speech Embedding Alignment Learn- ing for Speech Large Language Model with Retrieval- Augmented Generation.Preprint, arXiv:2502.02603. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. SALMONN: Towards Generic Hear- ing Abilities for Large Language Models. InThe Twelfth International Confer...

arXiv 2024
[9]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M

OpenReview.net. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, and 1332 others. 2025. Gemini: A Fa...

arXiv 2025
[10]

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Sys- tems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. arXiv. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée ...

arXiv 2024
[11]

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai

arXiv. Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. 2023. VisionLLM: Large Language Model is also an Open-Ended De- coder for Vision-Centric Tasks. InAdvances in Neu- ral Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, Ne...

arXiv 2023

Show all 13 references

[12]

DeeR-VLA: Dynamic Inference of Multi- modal Large Language Models for Efficient Robot Execution. InAdvances in Neural Information Pro- cessing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 20...

2024 arXiv
[13]

On Evaluating Adversarial Robustness of Large Vision-Language Models. InAdvances in Neu- ral Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10 - 16, 2023. arXiv. Zixiang Zhao,...

2023
[2023]

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. arXiv. ...

2023 arXiv
[2024]

Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles

OpenReview.net. Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. 2024. X-InstructBLIP: A Framework for aligning X- Modal instruction-aware representations to LLMs and Emergent Cross-modal...

2024 arXiv
[2025]

Preprint, arXiv:2311.07594

How to Bridge the Gap between Modali- ties: Survey on Multimodal Large Language Model. Preprint, arXiv:2311.07594. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. PandaGPT: One Model To Instruction-Follow Them All.Preprint, arXiv:2305.16355. Chunyu Sun...

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2]

InMachine Learning for Health, ML4H@NeurIPS 2023, 10 December 2023, New Or- leans, Louisiana, USA, volume 225 ofProceedings of Machine Learning Research, pages 353–367

Med-Flamingo: A Multimodal Medical Few- shot Learner. InMachine Learning for Health, ML4H@NeurIPS 2023, 10 December 2023, New Or- leans, Louisiana, USA, volume 225 ofProceedings of Machine Learning Research, pages 353–367. PMLR. Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. 2023. Embod...

arXiv 2023

[2] [4]

In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13052– 13062, Seattle, W A, USA

Sniffer: Multimodal Large Language Model for Explainable Out-of-Context Misinformation De- tection. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13052– 13062, Seattle, W A, USA. IEEE. Alec Radford. 2018. Improving language understanding with unsupervised learning.OpenAI Res. Alec Radford, Jong Wook Kim, Chris Hallacy...

arXiv 2018

[3] [5]

InIEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 15120–15130

LMDrive: Closed-Loop End-to-End Driving with Large Language Models. InIEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 15120–15130. IEEE. Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh

work page 2024

[4] [6]

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen

Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models.Preprint, arXiv:2307.14539. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models.National Science Review. Gowthami Somepalli, Arkabandhu Chowdhury, Jonas Geiping, Ronen Basri, Tom Goldstein, an...

arXiv 2023

[5] [8]

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang

SEAL: Speech Embedding Alignment Learn- ing for Speech Large Language Model with Retrieval- Augmented Generation.Preprint, arXiv:2502.02603. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. SALMONN: Towards Generic Hear- ing Abilities for Large Language Models. InThe Twelfth International Confer...

arXiv 2024

[6] [9]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M

OpenReview.net. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, and 1332 others. 2025. Gemini: A Fa...

arXiv 2025

[7] [10]

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Sys- tems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. arXiv. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée ...

arXiv 2024

[8] [11]

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai

arXiv. Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. 2023. VisionLLM: Large Language Model is also an Open-Ended De- coder for Vision-Centric Tasks. InAdvances in Neu- ral Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, Ne...

arXiv 2023