Pith. sign in

REVIEW 17 cited by

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.04160 v3 pith:EUPBRWCQ submitted 2023-05-07 cs.CL cs.AIcs.CVeess.AS

Feilong Chen , Minglun Han , Haozhi Zhao , Qingyang Zhang , Jing Shi , Shuang Xu , Bo Xu This is my paper · ORCID
classification cs.CLcs.AIcs.CVeess.AS
keywords multimodallanguagex-llmgpt-4interfaceslanguagesllmsmodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) have demonstrated remarkable language abilities. GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities beyond previous visual language models. We attribute this to the use of more advanced LLMs compared with previous multimodal models. Unfortunately, the model architecture and training strategies of GPT-4 are unknown. To endow LLMs with multimodal capabilities, we propose X-LLM, which converts Multi-modalities (images, speech, videos) into foreign languages using X2L interfaces and inputs them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X'' denotes multi-modalities such as image, speech, and videos, and ``L'' denotes languages. X-LLM's training consists of three stages: (1) Converting Multimodal Information: The first stage trains each X2L interface to align with its respective single-modal encoder separately to convert multimodal information into languages. (2) Aligning X2L representations with the LLM: single-modal encoders are aligned with the LLM through X2L interfaces independently. (3) Integrating multiple modalities: all single-modal encoders are aligned with the LLM through X2L interfaces to integrate multimodal capabilities into the LLM. Our experiments show that X-LLM demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 84.5\% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. And we also conduct quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote the era of LLM-based speech recognition.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Structured intermediate probing that separates modality-specific from modality-general signals lets privileged training modalities improve single-modality MLLM inference by large margins over naive multimodal training.

  2. DriveQA: Passing the Driving Knowledge Test

    cs.CV 2025-08 conditional novelty 6.0 of 10

    DriveQA is a new multimodal driving-knowledge benchmark showing that LLMs and MLLMs struggle with right-of-way, numerical traffic rules, and sign variations, with modest transfer gains to nuScenes and BDD.

  3. Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MMER merges multiple multimodal LLMs by averaging task vectors and applying per-modality binary masks, retaining about 99% of original task performance without additional training.

  4. Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models

    cs.SD 2025-05 conditional novelty 6.0 of 10

    AJailBench is an open benchmark showing that large audio-language models can be jailbroken through TTS-converted text attacks and through subtle acoustic perturbations that preserve speech semantics.

  5. AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A 600K-question audio-visual trustworthiness benchmark reveals that current AVLLMs are brittle under mismatches and missing modalities, and CAVPref, a calibrated preference-optimization method, improves their accuracy.

  6. Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases

    cs.CV 2024-12 conditional novelty 6.0 of 10

    By fine-tuning LLaVA on a knowledge-infused agricultural dataset, Agri-LLaVA improves agricultural conversation and VQA over general LMMs, with gains of about 5 points over LLaVA on the new benchmark.

  7. Aligning Pre-trained Models for Spoken Language Translation

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Frozen speech recognition and machine translation models can be aligned by a small connector network to perform end-to-end speech translation, and the connector also serves as a domain adapter.

  8. Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Audio-Language Models

    cs.SD 2024-11 conditional novelty 6.0 of 10

    The CAA benchmark applies content, emotional, explicit noise, and implicit noise attacks to six audio-language models and finds GPT-4o the most robust.

  9. ALAS: An Automatic Latent Alignment Score for Audio Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    ALAS is a reference-based score for audio-text alignment in speech LLMs, computed from frozen hidden states and a Whisper-derived alignment path, with no training or fitted classifier.

  10. Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Instructify converts image metadata into visual instruction-tuning conversations with open LLMs, matching or exceeding GPT-4-generated data quality on LMM benchmarks.

  11. Ola: Pushing the Frontiers of Omni-Modal Language Model

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Ola, a 7B omni-modal language model, achieves competitive image, video, and audio understanding with progressive modality alignment, though it does not beat all specialized models on every benchmark.

  12. Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Across controlled ASR and speech translation experiments, dense feature prepending does not outperform cross-attention in quality and is slightly slower and more memory hungry.

  13. Multimodal Representation Alignment for Cross-modal Information Retrieval

    cs.IR 2025-06 conditional novelty 4.0 of 10

    Across CLIP, BLIP, Meta-Transformer, and three combined unimodal models on IMDB, Flickr30K, and MS-COCO, cosine similarity gives the best cross-modal retrieval for contrastively trained models, while learned MLP align...

  14. Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective

    cs.CV 2025-02 conditional novelty 4.0 of 10

    A taxonomy and benchmark comparison of 34 vision large language models, showing two-stage instruction tuning gives top scores while direct adapter tuning is most parameter-efficient, though the comparison is confounde...

  15. A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future

    cs.CV 2024-12 conditional novelty 4.0 of 10

    A historical review that organizes multimodal explainability methods into four chronological eras and three explainability types, extending coverage to generative LLMs.

  16. Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.

  17. Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

    cs.CL 2024-11 conditional novelty 1.0 of 10

    A survey that organizes VQA methods from feature extraction through MLLM reasoning, datasets, and metrics, without introducing new experimental results.

Pith tools