REVIEW 9 cited by
Visual Prompting in Multimodal Large Language Models: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.
Forward citations
Cited by 9 Pith papers
-
Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification
Rule-VLN is the first large-scale benchmark injecting 177 regulatory categories into an urban environment, and the proposed SNRM module equips pre-trained VLN agents with zero-shot semantic reasoning and detour planni...
-
Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
A new counterfactual benchmark, PriVE-Bench, plus a controlled tool-based extension, PriVE-Tools, shows that VLMs often answer from priors and that tool-derived visual evidence helps some models but does not reliably ...
-
Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Design
Water4MU tunes an invisible watermark on data so that machine unlearning algorithms can remove requested images more effectively, beating prior methods on 'challenging forgets'.
-
PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior
Introduces PDB-Eval, a dual-view benchmark for fine-grained driver behavior description and explanation, and shows fine-tuning on it boosts performance on driving QA and downstream intention and recognition tasks.
-
AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs
AutoV selects instance- and query-specific visual prompts via loss-based pairwise ranking, consistently improving LVLMs across many benchmarks with no backbone fine-tuning.
-
Plover: Steering GUI Agents through Plan-Centric Interaction
An expert repairing visible plans rescued 23 of 26 failed GUI automation runs, turning 17 into full and 6 into partial successes.
-
Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?
A new benchmark and method (Spot-IT) aim to improve multimodal LLMs' ability to locate fine details in documents, with reported significant gains.
-
Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models
Uncertainty-o estimates uncertainty in large multimodal models by perturbing prompts and computing entropy over semantically clustered answers, improving hallucination detection across five modalities.
-
A Survey on Video Temporal Grounding with Multimodal Large Language Model
A taxonomized review of video temporal grounding with multimodal large language models, covering model roles, training paradigms, feature processing, benchmarks, and open problems.
Discussion (0). Sign in to comment.