Pith. sign in

REVIEW 5 cited by

Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.06199 v4 pith:OZ3R4MNL submitted 2024-03-10 cs.CV cs.CL

Minjie Zhu , Yichen Zhu , Xin Liu , Ning Liu , Zhiyuan Xu , Chaomin Shen , Yaxin Peng , Zhicai Ou
show 2 more authors
Feifei Feng Jian Tang
This is my paper · ORCID
classification cs.CVcs.CL
keywords languagemodelsmultimodalmllmsaspectsassistantlargemipha
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs. Our code is available at https://github.com/zhuyiche/llava-phi.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  2. InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A single 3B-parameter end-to-end model with object-aware video perceiving and multi-granularity text fusion reports SOTA results across four instructed visual segmentation tasks.

  3. LinVT: Empower Your Image-level Large Language Model to Understand Videos

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A plug-and-play linear video tokenizer converts existing image-based LLMs into video-understanding LLMs by condensing frames into weighted-average tokens while preserving image capabilities.

  4. FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression

    cs.CV 2024-12 conditional novelty 5.0 of 10

    FlashSloth compresses visual input to 90 tokens with attention pooling and an embedded cross-attention query module, achieving 2-5x faster response than tiny MLLM baselines with competitive benchmark scores.

  5. HyperSeg: Towards Universal Visual Segmentation with Large Language Model

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A single VLLM-based model, HyperSeg, unifies image and video segmentation, including reasoning tasks, and reports SOTA on RefCOCO, ReasonSeg, ReVOS, and panoptic segmentation.

Pith tools