VLRM: Vision-Language Models act as Reward Models for Image Captioning

Alexander Kunitsyn; Andrei Ivaniuta; Maksim Dzabraev

VLRM: Vision-Language Models act as Reward Models for Image Captioning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.01911 v1 pith:SRMIO5AZ submitted 2024-04-02 cs.CV

Maksim Dzabraev , Alexander Kunitsyn , Andrei Ivaniuta This is my paper · ORCID

classification cs.CV

keywords modelsmodelcaptioningclipimagerewardvision-languageable

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

In this work, we present an unsupervised method for enhancing an image captioning model (in our case, BLIP2) using reinforcement learning and vision-language models like CLIP and BLIP2-ITM as reward models. The RL-tuned model is able to generate longer and more comprehensive descriptions. Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split. Weights are available at https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ZIUM: Zero-Shot Intent-Aware Adversarial Attack on Unlearned Models
cs.CV 2025-07 conditional novelty 6.0 of 10

ZIUM attacks unlearned diffusion models by optimizing an image-captioning module that turns a target image into a text embedding, then reuses that module zero-shot on unseen images of the same unlearned concept.

Pith tools