Pith. sign in

REVIEW 7 cited by

Language Models Can See: Plugging Visual Controls in Text Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.02655 v2 pith:WA3SL523 submitted 2022-05-05 cs.CV cs.CL

Yixuan Su , Tian Lan , Yahui Liu , Fangyu Liu , Dani Yogatama , Yan Wang , Lingpeng Kong , Nigel Collier This is my paper · ORCID
classification cs.CVcs.CL
keywords generationtextmagicimagedecodingframeworkcalledcaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Generative language models (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality. While they are designed for text-prompted generation, it remains an open question how the generation process could be guided by modalities beyond text such as images. In this work, we propose a training-free framework, called MAGIC (iMAge-Guided text generatIon with CLIP), for plugging in visual controls in the generation process and enabling LMs to perform multimodal tasks (e.g., image captioning) in a zero-shot manner. MAGIC is a simple yet efficient plug-and-play framework, which directly combines an off-the-shelf LM (i.e., GPT-2) and an image-text matching model (i.e., CLIP) for image-grounded text generation. During decoding, MAGIC influences the generation of the LM by introducing a CLIP-induced score, called magic score, which regularizes the generated result to be semantically related to a given image while being coherent to the previously generated context. Notably, the proposed decoding scheme does not involve any gradient update operation, therefore being computationally efficient. On the challenging task of zero-shot image captioning, MAGIC outperforms the state-of-the-art method by notable margins with a nearly 27 times decoding speedup. MAGIC is a flexible framework and is theoretically compatible with any text generation tasks that incorporate image grounding. In the experiments, we showcase that it is also capable of performing visually grounded story generation given both an image and a text prompt.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning

    cs.CV 2026-08 conditional novelty 6.0 of 10

    WSV trains a zero-shot video captioner on synthetic video latents generated from text, then uses a prompter plus GPT-2 at inference, reaching 52.0 BLEU@4 and 95.7 CIDEr on MSVD without seeing real video during training.

  2. Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A multi-checkpoint alignment pipeline lifts zero-shot COCO captioning CIDEr from 108.0 to 117.6 by adding a cross-attention verifier and self-supervised beam rerankers to an unchanged IFCap captioner.

  3. Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    ReAlign corrects the modality gap in unpaired data to let MLLMs learn visual distributions from text alone before instruction tuning, reducing dependence on expensive paired corpora.

  4. Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Video-LLMs can be trained, via SFT or DPO on a new synthetic dataset UVQA, to refuse questions that cannot be answered from the video content, with modest cost to answerable QA performance.

  5. InsTALL: Context-aware Instructional Task Assistance with Multi-modal Large Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    InsTALL feeds a procedural graph mined from training videos into a multimodal LLM, improving task recognition, action recognition, next-action and plan prediction, and error detection on instructional videos.

  6. Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning

    cs.CV 2024-12 conditional novelty 6.0 of 10

    PCM-Net improves zero-shot image captioning by training on synthetic image-caption pairs and replacing defective visual patches with concept text features, achieving new SOTA on MSCOCO and Flickr30k.

  7. How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey

    cs.CV 2024-12 conditional novelty 4.0 of 10

    A survey that categorizes pre-trained-model-based vision-language methods into four challenge-driven paradigms, with performance tables and a discussion of risks.

Pith tools