REVIEW 30 cited by
LLaVA-Critic: Learning to Evaluate Multimodal Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We introduce LLaVA-Critic, the first open-source large multimodal model (LMM) designed as a generalist evaluator to assess performance across a wide range of multimodal tasks. LLaVA-Critic is trained using a high-quality critic instruction-following dataset that incorporates diverse evaluation criteria and scenarios. Our experiments demonstrate the model's effectiveness in two key areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation scores, performing on par with or surpassing GPT models on multiple evaluation benchmarks; and (2) Preference Learning, where it generates reward signals for preference learning, enhancing model alignment capabilities. This work underscores the potential of open-source LMMs in self-critique and evaluation, setting the stage for future research into scalable, superhuman alignment feedback mechanisms for LMMs.
Forward citations
Cited by 30 Pith papers
-
Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention
Gazette is the first generative gaze-to-text model: it decodes a single gaze scanpath into free-form natural-language descriptions of the viewer's goal, using GPT-4-generated 'think-aloud' transcripts as auxiliary tra...
-
MentalThink: Shaping Thoughts in Mental SVG World
MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.
-
AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation
With prompt engineering (audio concatenation plus in-context examples), large audio models rank speech synthesis systems in line with human preferences, reaching up to 0.91 Spearman correlation.
-
Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
MMRB is the first benchmark combining multi-image inputs with chain-of-thought reasoning annotations, and its evaluation shows open-source MLLMs trail commercial models while multi-image reward models are unstable.
-
Understanding Generative AI Capabilities in Everyday Image Editing Tasks
On real Reddit photo-editing requests, human judges prefer human edits over AI edits 66% of the time, and AI editors can satisfactorily handle about 33% of requests.
-
VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning
A new benchmark with human-annotated step-level critiques shows vision-language models can correct errors when given human feedback, but their self-generated critiques are weak and sometimes harmful; reverifying image...
-
VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models
VL-RewardBench curates 1,250 hard preference pairs for vision-language reward models; GPT-4o reaches only 62 to 65 percent accuracy, and benchmark scores correlate strongly with Best-of-N gains on MMMU-Pro.
-
Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences
LVLM judges are near chance when asked to pick the correctly ordered version of an image sequence, and this temporal blindness persists after fine-tuning and at larger scale.
-
Poplar: A Scalable Pipeline for Human-Centric Image Dataset Synthesis
Poplar-9K is a curated human-centric image dataset generated through a reproducible attribute sampling, diffusion rendering, and vision-language inspection pipeline, with 9,401 accepted pairs from 11,765 candidates.
-
Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles
A cluster-conditioned Bayesian prompt ensemble improves the calibration and accuracy of multimodal LLM judges for text-to-image preference evaluation.
-
Improving Large Vision and Language Models by Learning from a Panel of Peers
A panel of LVLMs that generate, evaluate, and learn from each other's outputs improves average benchmark scores by 9 points across 15 tasks.
-
LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...
-
VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering
A person-anchored tree plus multi-agent LLM pipeline lets a system answer cross-video queries about the same person, and it beats single-video models on the authors' new CrossVideoQA benchmark.
-
VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
A vision-expert-filtered, chain-of-thought-guided, iteratively fine-tuned reward model boosts a compact 7B model's ability to judge vision-language responses, especially detecting hallucinations.
-
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.
-
AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time
A stochastic scheduling framework that modulates slow-to-fast reasoning in large reasoning models at test time, improving accuracy while reducing token usage.
-
UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation
UniGen shows a 1.5B model trained on open data can beat larger systems on image understanding and generation once it verifies its own outputs with chain-of-thought and Best-of-N selection.
-
MINERVA: Evaluating Complex Video Reasoning
MINERVA provides 1,515 multi-step video QA questions with human reasoning traces; frontier models score far below humans and fail mainly on temporal localization and perception.
-
MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.
-
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
IXC-2.5-Reward is an open-source multimodal reward model that achieves 70.0% macro accuracy on VL-RewardBench and improves LVLM chat via PPO.
-
Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
Eagle2-9B matches or outperforms much larger vision-language models on many benchmarks through a carefully constructed post-training data strategy.
-
LLM-as-a-Verifier: A General-Purpose Verification Framework
Expecting over scoring-token logits yields continuous, scalable verification that improves agent trajectory selection and dense RL rewards across coding, robotics, and medical benchmarks.
-
Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning
A two-stage, value-guided decoding strategy with a margin-based reward adjustment is claimed to yield more faithful, detailed VLM captions at about a quarter of VisVM's inference cost.
-
Generative RLHF-V: Learning Principles from Multi-modal Human Preference
A reinforcement-learned multimodal judge with grouped pairwise scoring improves vision-language model alignment on seven benchmarks.
-
Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?
Some small open-source LVLMs, especially LLaVA-Critic-7B, can serve as cheap automatic judges for chart tasks, agreeing with GPT-4o about 80% of the time, though position and length biases persist.
-
Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
SPORT builds a multimodal tool agent by generating synthetic tasks, sampling candidate tool calls, and tuning the agent on AI-verified step preferences, improving GTA and GAIA accuracy over the MAT baseline.
-
YINYANG-ALIGN: Benchmarking Contradictory Objectives and Proposing Multi-Objective Optimization based DPO for Text-to-Image Alignment
Introduces a six-axis contradictory-objective benchmark and a weighted DPO variant (CAO), but the claimed balanced alignment rests on evaluations that reuse the training objectives.
-
MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models
MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.
-
LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs
LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.
-
EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation
A critic trained on filtered GPT-4V scoring data selects the best and worst self-generated responses, and DPO with extra SFT improves hallucination and reasoning benchmarks in multimodal LLMs.
Discussion (0). Continue with ORCID to comment.