REVIEW 8 cited by
A Review of Multi-Modal Large Language and Vision Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) have recently emerged as a focal point of research and application, driven by their unprecedented ability to understand and generate text with human-like quality. Even more recently, LLMs have been extended into multi-modal large language models (MM-LLMs) which extends their capabilities to deal with image, video and audio information, in addition to text. This opens up applications like text-to-video generation, image captioning, text-to-speech, and more and is achieved either by retro-fitting an LLM with multi-modal capabilities, or building a MM-LLM from scratch. This paper provides an extensive review of the current state of those LLMs with multi-modal capabilities as well as the very recent MM-LLMs. It covers the historical development of LLMs especially the advances enabled by transformer-based architectures like OpenAI's GPT series and Google's BERT, as well as the role of attention mechanisms in enhancing model performance. The paper includes coverage of the major and most important of the LLMs and MM-LLMs and also covers the techniques of model tuning, including fine-tuning and prompt engineering, which tailor pre-trained models to specific tasks or domains. Ethical considerations and challenges, such as data bias and model misuse, are also analysed to underscore the importance of responsible AI development and deployment. Finally, we discuss the implications of open-source versus proprietary models in AI research. Through this review, we provide insights into the transformative potential of MM-LLMs in various applications.
Forward citations
Cited by 8 Pith papers
-
TOKON: TOKenization-Optimized Normalization for time series analysis with a large language model
TOKON rounds normalized time series values into integer tokens and adds a 'forecast with care' prompt, reporting RMSE improvements of 7 to 28 percent on two datasets with GPT-4o-mini.
-
Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering
A training-free pipeline constructs a spatiotemporal knowledge graph from egocentric video, enabling low-latency, explainable embodied question answering.
-
LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents
On CUAD legal contracts, a prompt-engineered QWEN-2 pipeline with chunking and two answer-selection heuristics reportedly outperforms the fine-tuned DeBERTa-large baseline by about 9%, reaching claimed state-of-the-ar...
-
Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models
Uncertainty-o estimates uncertainty in large multimodal models by perturbing prompts and computing entropy over semantically clustered answers, improving hallucination detection across five modalities.
-
HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMs
A progressive two-stage knowledge distillation framework (HKD4VLM) reports first-place F1 scores of 98.2% and 98.4% on multimodal hallucination and factuality detection, but its ablation lacks a directly fine-tuned baseline.
-
From Screens to Scenes: A Survey of Embodied AI in Healthcare
A survey of embodied AI in healthcare, organizing 35 tasks into four application domains and proposing a five-level intelligence scale.
-
TSPE: Task-Specific Prompt Ensemble for Improved Zero-Shot Audio Classification
Task-specific prompt ensembling with GPT-4-generated attributes and sources improves some zero-shot audio classification datasets while degrading others.
-
A Comprehensive Survey of Large AI Models for Future Communications: Foundations, Applications and Challenges
A survey organizing the growing literature on large AI models for 6G communications, with a classification of model types, training and evaluation methods, and a list of challenges.
Discussion (0). Continue with ORCID to comment.