Pith. sign in

REVIEW 10 cited by

ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.16103 v1 pith:QK64XXCU submitted 2023-05-25 cs.CV cs.AIcs.CLcs.MM

Zijia Zhao , Longteng Guo , Tongtian Yue , Sihan Chen , Shuai Shao , Xinxin Zhu , Zehuan Yuan , Jing Liu This is my paper
classification cs.CVcs.AIcs.CLcs.MM
keywords chatbridgelanguagemodalitiesmultimodalmodelstasksaudiocapabilities
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Building general-purpose models that can perceive diverse real-world modalities and solve various tasks is an appealing target in artificial intelligence. In this paper, we present ChatBridge, a novel multimodal language model that leverages the expressive capabilities of language as the catalyst to bridge the gap between various modalities. We show that only language-paired two-modality data is sufficient to connect all modalities. ChatBridge leverages recent large language models (LLM) and extends their zero-shot capabilities to incorporate diverse multimodal inputs. ChatBridge undergoes a two-stage training. The first stage aligns each modality with language, which brings emergent multimodal correlation and collaboration abilities. The second stage instruction-finetunes ChatBridge to align it with user intent with our newly proposed multimodal instruction tuning dataset, named MULTIS, which covers a wide range of 16 multimodal tasks of text, image, video, and audio modalities. We show strong quantitative and qualitative results on zero-shot multimodal tasks covering text, image, video, and audio modalities. All codes, data, and models of ChatBridge will be open-sourced.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sample-efficient Integration of New Modalities into Large Language Models

    cs.CL 2025-09 conditional novelty 7.0 of 10

    A hypernetwork trained on image, audio, and video adapts a shared projector to new, low-resource modalities from as few as 32 examples.

  2. HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks

    cs.CV 2024-12 unverdicted novelty 7.0 of 10

    HumanVBench provides a 16-task benchmark for human-centric video understanding in MLLMs, created through automated annotation and distractor synthesis pipelines, and shows top models lag human performance on emotion p...

  3. Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Structured intermediate probing that separates modality-specific from modality-general signals lets privileged training modalities improve single-modality MLLM inference by large margins over naive multimodal training.

  4. Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    ContextGuard prunes 55% of tokens in Qwen2.5-Omni 7B while matching full performance on five of six audio-visual benchmarks by preserving audio-irrecoverable visual context.

  5. RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    RaTA-Tool retrieves suitable external tools for multimodal queries by matching generated task descriptions against tool metadata, supported by a new Hugging Face-derived dataset and DPO optimization.

  6. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    cs.CV 2023-06 unverdicted novelty 6.0 of 10

    MME is a manually annotated benchmark evaluating MLLMs on perception and cognition across 14 subtasks to avoid data leakage and support fair model comparisons.

  7. AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering

    cs.CV 2025-10 conditional novelty 5.0 of 10

    AV-Master reports state-of-the-art accuracy on four audio-visual question answering benchmarks by combining sequential question-guided focus sampling with modality-preference activation.

  8. AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering

    cs.CV 2025-10 unverdicted novelty 5.0 of 10

    AV-Master introduces dynamic adaptive focus sampling, modality preference modeling, and dual-path contrastive loss to outperform prior methods on audio-visual question answering benchmarks.

  9. A Survey on Multimodal Large Language Models

    cs.CV 2023-06 accept novelty 3.0 of 10

    This survey organizes the architectures, training strategies, data, evaluation methods, extensions, and challenges of Multimodal Large Language Models.

  10. A Comprehensive Overview of Large Language Models

    cs.CL 2023-07 unverdicted novelty 2.0 of 10

    A survey paper providing an overview of Large Language Models, their background, and recent advances in the field.

Pith tools