Pith. sign in

REVIEW 5 cited by

VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.07398 v3 pith:XQYCD3ZN submitted 2024-02-12 cs.AI

Dongsheng Zhu , Xunzhu Tang , Weidong Han , Jinghui Lu , Yukun Zhao , Guoliang Xing , Junfeng Wang , Dawei Yin This is my paper
classification cs.AI
keywords mmlmsvislinginstructmulti-modalzero-shotlearningvisualinstructionallanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper presents VisLingInstruct, a novel approach to advancing Multi-Modal Language Models (MMLMs) in zero-shot learning. Current MMLMs show impressive zero-shot abilities in multi-modal tasks, but their performance depends heavily on the quality of instructions. VisLingInstruct tackles this by autonomously evaluating and optimizing instructional texts through In-Context Learning, improving the synergy between visual perception and linguistic expression in MMLMs. Alongside this instructional advancement, we have also optimized the visual feature extraction modules in MMLMs, further augmenting their responsiveness to textual content. Our comprehensive experiments on MMLMs, based on FlanT5 and Vicuna, show that VisLingInstruct significantly improves zero-shot performance in visual multi-modal tasks. Notably, it achieves a 13.1% and 9% increase in accuracy over the prior state-of-the-art on the TextVQA and HatefulMemes datasets. Our main code is available at https://github.com/Zhudongsheng75/VisLingInstruct.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation

    cs.LG 2025-08 reject novelty 4.0 of 10

    An LVLM-driven iterative text-to-image framework whose claimed performance scores are explicitly labeled fictitious, so no empirical result is established.

  2. LVLM-Composer's Explicit Planning for Image Generation

    cs.CV 2025-07 reject novelty 4.0 of 10

    An image generation model that explicitly plans objects, attributes, locations, and relations before synthesizing the image, with reported gains on LongBench-T2I that cannot be verified from the paper.

  3. MM-FusionNet: Context-Aware Dynamic Fusion for Multi-modal Fake News Detection with Large Vision-Language Models

    cs.CR 2025-08 reject novelty 3.0 of 10

    MM-FusionNet uses bi-directional cross-modal attention and a dynamic gating network to weight text and image features for fake news detection, reporting 0.938 F1 on the private LMFND dataset.

  4. VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extraction

    cs.CV 2025-08 reject novelty 3.0 of 10

    A prompt-wrapper framework for vision-language models reports improved story and poetry generation on a private benchmark, without releasing code, data, or models.

  5. Large Language Models for Zero-Shot Multicultural Name Recognition

    cs.CL 2025-07 reject novelty 3.0 of 10

    A prompt-tuned LLM with data augmentation and cultural context prompts reportedly recognizes multicultural names at 93.1% accuracy and unseen names at 89.5%, but the evidence is not reproducible.

Pith tools