REVIEW 13 cited by
TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In recent years, multimodal large language models (MLLMs) such as GPT-4V have demonstrated remarkable advancements, excelling in a variety of vision-language tasks. Despite their prowess, the closed-source nature and computational demands of such models limit their accessibility and applicability. This study introduces TinyGPT-V, a novel open-source MLLM, designed for efficient training and inference across various vision-language tasks, including image captioning (IC) and visual question answering (VQA). Leveraging a compact yet powerful architecture, TinyGPT-V integrates the Phi-2 language model with pre-trained vision encoders, utilizing a unique mapping module for visual and linguistic information fusion. With a training regimen optimized for small backbones and employing a diverse dataset amalgam, TinyGPT-V requires significantly lower computational resources 24GB for training and as little as 8GB for inference without compromising on performance. Our experiments demonstrate that TinyGPT-V, with its language model 2.8 billion parameters, achieves comparable results in VQA and image inference tasks to its larger counterparts while being uniquely suited for deployment on resource-constrained devices through innovative quantization techniques. This work not only paves the way for more accessible and efficient MLLMs but also underscores the potential of smaller, optimized models in bridging the gap between high performance and computational efficiency in real-world applications. Additionally, this paper introduces a new approach to multimodal large language models using smaller backbones. Our code and training weights are available in the supplementary material.
Forward citations
Cited by 13 Pith papers
-
Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs
Jointly optimizing visual-token retention and LLM depth/width under a shared FLOPs budget yields better accuracy-efficiency Pareto frontiers than optimizing either dimension alone.
-
EdgeMM: Multi-Core CPU with Heterogeneous AI-Extension and Activation-aware Weight Pruning for Multimodal LLMs at Edge
EdgeMM is a 22nm multi-core CPU that combines systolic-array and compute-in-memory coprocessors with adaptive activation-aware pruning, reported as 2.84x faster than a laptop RTX 3060 for multimodal LLMs.
-
Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
A tuning-free pipeline using frozen LLaMA-3, MiniGPT-v2, and Video-ChatGPT reports state-of-the-art zero-shot video moment retrieval on three benchmarks.
-
Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation
Simignore improves multimodal LLM complex question answering on ScienceQA by masking image tokens whose embeddings have low cosine similarity to the text prompt.
-
[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs
Using the CLIP [CLS] token's attention to prune image tokens before the LLM runs outperforms prior training-free pruning methods on LLaVA-1.5-7B and speeds up inference.
-
Efficient Multi-modal Large Language Models via Visual Token Grouping
A new grouping mechanism, VisToG, uses semantic tokens and isolated attention inside a pretrained CLIP vision encoder to compress visual tokens for MLLMs, retaining roughly 98% performance while cutting inference time...
-
DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
A training-free two-stage token compression method speeds up video language model inference by about 1.5x and cuts memory use, while keeping accuracy close to the uncompressed model.
-
TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
TinyAlign improves lightweight VLMs by retrieving compressed image-text memory items to augment visual inputs, claiming large loss and data-efficiency gains, but the memory bank is sampled from the training captions, ...
-
Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks
A task-oriented semantic communication framework for LLaVA-based vehicle assistants that crops image slices matched to the user's question and allocates transmission power by fused objective and subjective attention, ...
-
NanoVLMs: How small can we go and still make coherent Vision Language Models?
NanoVLMs, 5M to 25M parameter vision-language models trained on simplified GPT-4o captions, are judged by GPT-4o as nearly as coherent as the 50x larger Kosmos-2 on a 25-sample test.
-
Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts
A 1.8B-parameter vision-language model, Eve, uses elastic visual experts and type-aware token routing to reach a 68.87% average on six VLM benchmarks while preserving language performance.
-
EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models
EvoMoE creates MoE experts as decaying averages of a single trained FFN and routes tokens with hypernetwork-generated weights, yielding small benchmark gains over MoE-LLaVA.
-
Multimodal Large Language Models for Medicine: A Comprehensive Survey
A comprehensive review cataloging medical MLLMs, their uses in report generation, diagnosis, and treatment, and the challenges of accuracy, hallucination, fairness, privacy, and deployment.
Discussion (0). Continue with ORCID to comment.