REVIEW 45 cited by
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. This opens the door to richer interaction with the world, for example robotic control. However, VLMs produce only textual outputs, while robotic control and other spatial tasks require outputting continuous coordinates, actions, or trajectories. How can we enable VLMs to handle such settings without fine-tuning on task-specific data? In this paper, we propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as iterative visual question answering. In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e.g., candidate robot actions, localizations, or trajectories). The VLM then selects the best ones for the task. These proposals are iteratively refined, allowing the VLM to eventually zero in on the best available answer. We investigate PIVOT on real-world robotic navigation, real-world manipulation from images, instruction following in simulation, and additional spatial inference tasks such as localization. We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities. Although current performance is far from perfect, our work highlights potentials and limitations of this new regime and shows a promising approach for Internet-Scale VLMs in robotic and spatial reasoning domains. Website: pivot-prompt.github.io and HuggingFace: https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
Forward citations
Cited by 45 Pith papers
-
Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
A VLM-driven model predictive controller that evaluates simulated future outcomes rendered from a physics-based digital twin.
-
DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models
DINO-R1 trains visual-prompt detectors with group-relative query rewards and KL regularization, improving zero-shot and fine-tuned detection over supervised fine-tuning.
-
Vision Language Models Cannot Reason About Physical Transformation
Current VLMs cannot maintain transformation-invariant representations of number, length, volume or size and instead rely on textual invariance priors that reverse on matched non-conserving controls.
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
Sparse Meets Dense: Correspondence Guided Robotic Manipulation with Rigid-Deformable Interactions
A hybrid sparse-plus-dense correspondence representation improves robotic manipulation success rates for rigid-deformable interaction tasks such as hanging clothes and packing bags.
-
World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models
Pairing VLM-generated action proposals with rollouts from a pose-image-conditioned video world model yields high success rates in novel simulated manipulation tasks without end-to-end policy retraining.
-
RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation
Dense per-frame intermediate representations (traces, masks, grasp poses, subtasks) improve embodied VQA, VLA action generation, and world-model video prediction in the new 230k-episode RoboInter-Data suite.
-
Visual-Language-Guided Task Planning for Horticultural Robots
A vision-language model drives a simulated greenhouse robot through simple crop-inspection tasks with ~87% success, but long multi-target tasks collapse to under 10% success.
-
EVE: A Generator-Verifier System for Generative Policies
Zero-shot VLM verifiers, ensembled and fused via guided diffusion, improve frozen generative robot policies' success rates by 1-2 percentage points on simulated manipulation tasks.
-
SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
Dense scene-graph-grounded rewards let a 7B multimodal LLM trained on 7K synthetic questions beat SFT and sparse-RL baselines and outscore GPT-4o on average across 12 spatial/real-world benchmarks.
-
SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation
SocialNav-SUB introduces a VQA benchmark for social robot navigation and shows current VLMs underperform rule-based and human-agreement baselines on spatial, spatiotemporal, and social reasoning questions.
-
TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goals
A navigation pipeline that bridges object-level global planning with traversability-aware local control, using only RGB images and pretrained models, improves success over prior zero-shot and learned baselines in simulation.
-
CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models
Counterfactual language-action relabeling raises instruction-following success from about 26% to 53% in real-world navigation tests.
-
EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos
EgoLoc localizes hand-object contact and separation timestamps in egocentric videos in a zero-shot manner using hand-dynamics-guided sampling, a VLM localizer, and closed-loop feedback.
-
AimBot: A Simple Auxiliary Visual Cue to Enhance Spatial Awareness of Visuomotor Policies
Overlaying end-effector-derived shooting lines and reticles on RGB images consistently raises success rates of visuomotor policies, especially on long-horizon manipulation tasks.
-
RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping
A multi-domain affordance benchmark with 273k images and 26k reasoning instructions is introduced, together with a VLM-based grasping pipeline that shows strong zero-shot affordance segmentation and real-robot performance.
-
Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
TCoT uses a single VLM to select question-relevant video frames from segments, then answers from that curated context, improving video QA accuracy across four benchmarks and three VLMs.
-
T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models
A zero-training framework that adaptively selects spatial representation extractors per object and per task stage improves real-world robot manipulation success and efficiency over fixed-representation baselines.
-
VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models
In controlled experiments across 2D, 3D, simulated, and real robot tasks, VLA-OS shows visually grounded planning representations outperform language planning, and hierarchical planning-plus-action models generally ou...
-
CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
CodeDiffuser uses vision-language-model-generated code to build 3D attention maps that condition a diffusion policy, improving success on ambiguous language manipulation tasks compared with end-to-end baselines.
-
Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models
A VLM-powered assistive teleoperation system infers diverse user intents from teleoperation snippets and executes them with a skill library, outperforming baselines on real-world mobile manipulation tasks.
-
AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making
AntiGrounding lifts candidate robot trajectories into the VLM's visual space via multi-view rendering and structured VQA, and reports 57.5% average success across eight manipulation tasks, beating three intermediate-r...
-
UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
UAD distills affordance knowledge from vision-language models and DINOv2 features into a lightweight task-conditioned model that predicts pixel-level manipulation regions and improves few-shot imitation learning gener...
-
Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning
PDC trains a single egocentric-vision policy that lets a simulated humanoid search for, grasp, and place objects and open drawers without privileged state information.
-
PointArena: Probing Multimodal Grounding Through Language-Guided Pointing
A three-stage benchmark consisting of 982 pointing tasks, a live pairwise arena with 4,500 votes, and a robot manipulation study shows that pointing-supervised open models such as Molmo-72B can match proprietary model...
-
Visual Test-time Scaling for GUI Agent Grounding
RegionFocus improves GUI agent grounding by error-triggered zooming into sub-regions and aggregating candidate actions with visual landmarks on the screenshot.
-
A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards
IKER uses VLM-generated keypoint rewards to train manipulation policies in simulation that transfer to a real robot, enabling multi-step tasks and replanning.
-
OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
OmniManip represents manipulation as canonical-space interaction points and directions, lets a VLM select them under closed-loop verification, and tracks object poses during execution to achieve zero-shot open-vocabul...
-
Green for Go, Red for No: Visual Grounding via Semantic Segmentation for VLA Navigation Policies
Real-time SegFormer green/red overlays reduce OmniVLA far-waypoint error 27-44% on Grand Tour language goals mainly by shortening trajectories, with little help for image goals.
-
IMBench: A Benchmark for Intuitive Robotic Manipulation
IMBench is a 35-task robosuite benchmark with a three-stage evaluation showing current VLMs and robot policies fail to convert physical reasoning into executable manipulation.
-
Sem-NaVAE: Semantically-Guided Outdoor Mapless Navigation via Generative Trajectory Priors
A lightweight CLIPSeg semantic scorer selects among 200 CVAE-generated trajectories, giving 90% success on 120-240 m mapless outdoor routes.
-
DyNaVLM: Zero-Shot Vision-Language Navigation System with Dynamic Viewpoints and Self-Refining Graph Memory
A zero-shot VLM navigation policy with dynamic waypoint selection and graph memory reports state-of-the-art results among VLM-based methods on ObjectNav and GOAT-Bench, plus real-world tests on a quadruped.
-
OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
A vision-language model fine-tuned on 572K synthetic simulation examples improves open-world mobile manipulation action decisions and object grounding over GPT-4o, with 21.9% full-task success in simulation and 90% ac...
-
Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare
The authors propose a four-category taxonomy of healthcare vision-language model studies with category-specific reporting standards and a peer-review checklist, arguing existing ML reporting guidelines are unfit for VLMs.
-
Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospects
A structured review and pilot workshop that organizes VLM trust research into a new taxonomy and finds a shortage of direct user studies.
-
CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation
CrayonRobo trains a vision-language-action model to read colored 2D prompt overlays (contact point, end-effector axes, movement direction) and output SE(3) contact poses, enabling step-by-step and long-horizon robotic...
-
Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models
CoM sequentially prompts a VLM over video, force/audio, and hand pose, yielding roughly threefold better extraction of task plans and control parameters, with real robots succeeding in 73% of trials.
-
VLM-driven Behavior Tree for Context-aware Task Planning
A VLM-generated behavior tree with self-prompted visual conditions lets a real robot branch on what it sees, clearing cups correctly in 8/10 cafe trials.
-
MOSU: Autonomous Long-range Robot Navigation with Multi-modal Scene Understanding
MOSU improves outdoor robot traversability by about 10% over prior state-of-the-art on the GND benchmark by fusing geometric, semantic, and vision-language-model scores for candidate trajectories.
-
Grounding Language Models with Semantic Digital Twins for Robotic Planning
The system grounds an LLM's action plans in hand-built semantic rules about a simulated home and reports success on all 14 selected ALFRED tasks.
-
LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks
A unified vision-language-action model that emits a sub-task description followed by a discrete action token outperforms modular and action-only baselines on simulated long-horizon tabletop tasks.
-
Efficient Sensorimotor Learning for Open-world Robot Manipulation
A PhD dissertation argues that object, spatial, and behavioral regularities, extracted with foundation models, enable data-efficient, generalizable robot manipulation, and presents seven systems and a benchmark built ...
-
Zero-Shot Scene Understanding for Automatic Target Recognition Using Large Vision-Language Models
A YOLO-world detector plus a vision-language model can label military vehicle crops zero-shot, but post-hoc label alignment and missing baselines make the headline numbers unreliable.
-
Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACT
A pick-and-place system overlays bounding boxes on camera images, trains an ACT transformer on human demonstrations, and reports 80% to 100% success rates across three retail scenarios.
-
Foundation Model Driven Robotics: A Comprehensive Review
A review of foundation-model-driven robotics that synthesizes recent work across perception, planning, control, HRI, simulation, and sim-to-real transfer, and highlights open challenges.
Discussion (0). Continue with ORCID to comment.