REVIEW 8 cited by
Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision
Forward citations
Cited by 8 Pith papers
-
Autoregressive Modeling of Film with Applications in Video Montage
An autoregressive transformer with an explicit cut token and footage-constrained decoding edits raw video into sequences that people rate as better than two prior automated editing methods.
-
Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences
LVLM judges are near chance when asked to pick the correctly ordered version of an image sequence, and this temporal blindness persists after fine-tuning and at larger scale.
-
Improving Large Vision and Language Models by Learning from a Panel of Peers
A panel of LVLMs that generate, evaluate, and learn from each other's outputs improves average benchmark scores by 9 points across 15 tasks.
-
Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts
Multimodal LLMs underperform humans at directly rating charts' experiential impact, but they are substantially better at pairwise comparisons, especially when the human ratings differ clearly.
-
CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
CHIRP, a 104-question pairwise benchmark, exposes scaling trends in vision-language models that existing benchmarks miss, and its evaluations correlate better with training loss.
-
ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts
A new guidance reweighting, derived from a contrastive loss, makes negative prompting in diffusion models remove unwanted concepts with less quality loss than standard negated CFG.
-
LLM-as-a-Verifier: A General-Purpose Verification Framework
Expecting over scoring-token logits yields continuous, scalable verification that improves agent trajectory selection and dense RL rewards across coding, robotics, and medical benchmarks.
-
Personalized Preference Fine-tuning of Diffusion Models
PPD fine-tunes a single diffusion model to follow per-user preferences by conditioning on VLM-extracted embeddings, reporting 76-81% win rates over Stable Cascade with four examples per user.
Discussion (0). Continue with ORCID to comment.