REVIEW 12 cited by
Enhancing Large Vision Language Models with Self-Training on Image Comprehension
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large vision language models (LVLMs) integrate large language models (LLMs) with pre-trained vision encoders, thereby activating the perception capability of the model to understand image inputs for different queries and conduct subsequent reasoning. Improving this capability requires high-quality vision-language data, which is costly and labor-intensive to acquire. Self-training approaches have been effective in single-modal settings to alleviate the need for labeled data by leveraging model's own generation. However, effective self-training remains a challenge regarding the unique visual perception and reasoning capability of LVLMs. To address this, we introduce Self-Training on Image Comprehension (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference dataset for image descriptions using unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses are generated from either corrupted images or misleading prompts. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data and append its self-generated image descriptions to the prompts. We validate the effectiveness of STIC across seven different benchmarks, demonstrating substantial performance gains of 4.0% on average while using 70% less supervised fine-tuning data than the current method. Further studies investigate various components of STIC and highlight its potential to leverage vast quantities of unlabeled images for self-training. Code and data are made publicly available.
Forward citations
Cited by 12 Pith papers
-
LPOI: Listwise Preference Optimization for Vision Language Models
LPOI reduces VLM hallucination by training the model to prefer the original image over progressively masked versions of the same image, using a listwise ranking loss built from pairwise preference data.
-
Probing Visual Language Priors in VLMs
ViLP shows that vision-language models often answer from text priors instead of image content, and an image-corruption DPO method partially fixes this.
-
Improving Large Vision and Language Models by Learning from a Panel of Peers
A panel of LVLMs that generate, evaluate, and learn from each other's outputs improves average benchmark scores by 9 points across 15 tasks.
-
Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilization
Mol-LLM combines SELFIES text, 2D molecular graphs, and preference optimization to build a generalist chemistry LLM that outperforms prior generalist models on most property, reaction, and translation benchmarks.
-
CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs
CHiP adds image-level and phrase/token-level preference signals to DPO for multimodal LLMs, and the authors report large hallucination-rate reductions on Object HalBench, AMBER, MMHal, and HallusionBench.
-
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
IXC-2.5-Reward is an open-source multimodal reward model that achieves 70.0% macro accuracy on VL-RewardBench and improves LVLM chat via PPO.
-
AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs
A 600K-question audio-visual trustworthiness benchmark reveals that current AVLLMs are brittle under mismatches and missing modalities, and CAVPref, a calibrated preference-optimization method, improves their accuracy.
-
Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution
A multimodal LLM can improve itself using only unlabeled images by self-generating questions, self-enhancing answers, and adding a description-alignment loss to DPO.
-
From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought
Answer-oriented chain-of-thought prompts that generate both positive and negative reasoning data, combined with iterative DPO, improve multimodal LLM reasoning on several benchmarks.
-
Diving into Self-Evolving Training for Multimodal Reasoning
M-STAR, a self-evolving training recipe combining continuous updates, a process-reward-model reranker, and adaptive sampling temperature, improves multimodal reasoning on several benchmarks across three vision-languag...
-
LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs
LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.
-
Optimizing Vision-Language Interactions Through Decoder-Only Models
An encoder-free vision-language model that claims state-of-the-art scores; the paper lacks any reproducible evidence.
Discussion (0). Continue with ORCID to comment.