Pith. sign in

REVIEW 10 cited by

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.20271 v3 pith:WBOCGK55 submitted 2024-03-29 cs.CV

Weifeng Lin , Xinyu Wei , Ruichuan An , Peng Gao , Bocheng Zou , Yulin Luo , Siyuan Huang , Shanghang Zhang
show 1 more author
Hongsheng Li
This is my paper
classification cs.CV
keywords visualimagesmllmsframeworkmodelspromptsunderstandingcapabilities
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we present the Draw-and-Understand framework, exploring how to integrate visual prompting understanding capabilities into Multimodal Large Language Models (MLLMs). Visual prompts allow users to interact through multi-modal instructions, enhancing the models' interactivity and fine-grained image comprehension. In this framework, we propose a general architecture adaptable to different pre-trained MLLMs, enabling it to recognize various types of visual prompts (such as points, bounding boxes, and free-form shapes) alongside language understanding. Additionally, we introduce MDVP-Instruct-Data, a multi-domain dataset featuring 1.2 million image-visual prompt-text triplets, including natural images, document images, scene text images, mobile/web screenshots, and remote sensing images. Building on this dataset, we introduce MDVP-Bench, a challenging benchmark designed to evaluate a model's ability to understand visual prompting instructions. The experimental results demonstrate that our framework can be easily and effectively applied to various MLLMs, such as SPHINX-X and LLaVA. After training with MDVP-Instruct-Data and image-level instruction datasets, our models exhibit impressive multimodal interaction capabilities and pixel-level understanding, while maintaining their image-level visual perception performance.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO

    cs.CV 2026-07 accept novelty 7.0 of 10

    A single MLLM jointly improves region captioning and localization by rewarding captions that let it reconstruct the original mask, needing only region inputs.

  2. LPOI: Listwise Preference Optimization for Vision Language Models

    cs.CV 2025-05 conditional novelty 7.0 of 10

    LPOI reduces VLM hallucination by training the model to prefer the original image over progressively masked versions of the same image, using a listwise ranking loss built from pairwise preference data.

  3. E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes

    cs.CV 2026-04 conditional novelty 6.0 of 10

    E-VLA integrates event streams directly into VLA models via lightweight fusion, raising Pick-Place success from 0% to 60-90% at 20 lux and from 0% to 20-25% under severe motion blur.

  4. VoCap: Video Object Captioning and Segmentation from Any Prompt

    cs.CV 2025-08 conditional novelty 6.0 of 10

    VoCap jointly performs promptable video object segmentation and object captioning, and introduces a 50k-video pseudo-caption dataset that improves both tasks.

  5. DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DenseWorld-1M provides one million images with detailed object captions, pixel masks, and spatial relations by chaining SAM, APE, RAM++, and VLMs through a three-stage labeling pipeline.

  6. Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Context-to-Cue Direct Preference Optimization (CcDPO) reduces multi-image hallucinations in 7B multimodal LLMs by training on perturbed full-sequence captions and region-focused visual prompts, improving average multi...

  7. Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A new benchmark and method (Spot-IT) aim to improve multimodal LLMs' ability to locate fine details in documents, with reported significant gains.

  8. Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

    cs.CV 2025-06 reject novelty 5.0 of 10

    PAM extends SAM 2 with a frozen LLM and a Semantic Perceiver to jointly segment and describe regions in images, videos, and streaming video, and contributes a 0.6M-sample region-level streaming video caption dataset.

  9. RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought

    cs.CV 2025-06 conditional novelty 5.0 of 10

    RSVP couples region-grid visual prompting and multimodal chain-of-thought reasoning with a BEiT-3/SAM segmentation module, achieving state-of-the-art zero-shot results on ReasonSeg and SegInW.

  10. The high-speed X-ray camera on AXIS: design and performance updates

    astro-ph.IM 2025-08 unverdicted novelty 4.0 of 10

    An X-ray camera design-update whose supporting full text is a different paper (RynnEC, an embodied AI model), leaving all camera performance claims unverified.

Pith tools