REVIEW 24 cited by
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. Sketchpad can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment with a wide range of math tasks (including geometry, functions, graphs, and chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in https://visualsketchpad.github.io/.
Forward citations
Cited by 24 Pith papers
-
Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
Point-RFT uses point-grounded chain-of-thought with GRPO reinforcement to improve chart reasoning, reporting 90.04% on ChartQA, though internal tables and the OOD setup weaken the claim.
-
VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning
A new benchmark with human-annotated step-level critiques shows vision-language models can correct errors when given human feedback, but their self-generated critiques are weak and sometimes harmful; reverifying image...
-
SketchAgent: Language-Driven Sequential Sketch Generation
SketchAgent uses a multimodal LLM prompted with a numbered-grid sketching language to generate, edit, and collaboratively draw sequential vector sketches without any training.
-
Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning
Replacing returned images with a fixed text placeholder in tool-augmented visual reasoning preserves benchmark accuracy, suggesting the tool-call text, not the returned pixels, carries the gain.
-
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
Multimodal models usually pick relevant visual actions, but faithful rendering is the bottleneck, and corrupted visual feedback drops accuracy over 10 points in 3D tasks.
-
SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning
Integrating binary self-verification into multi-turn GRPO rollouts raises VLM multimodal reasoning accuracy over matched GRPO baselines while the model learns to need fewer rethinks.
-
Beyond the Textual: Generating Coherent Visual Options for MCQs
A four-stage framework (convertibility check, question/reason generation, optimal pair selection, and template-based image generation) produces MCQs with image options from ScienceQA content.
-
SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning
SyncLoop jointly evolves multimodal training data and model capability through alternating SFT and RL, selecting error-prone samples to improve geometry reasoning.
-
MATP-BENCH: Can MLLM Be a Good Automated Theorem Prover for Multimodal Problems?
MATP-BENCH pairs 1,056 multimodal math problems with formal theorem statements in Lean 4, Coq, and Isabelle; the strongest tested model solves only 5.68% of Lean 4 end-to-end proving tasks at pass@10.
-
MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
A new 1,188-question multimodal benchmark covering deductive, inductive, and abductive reasoning shows that leading MLLMs score around 60% and are especially weak at abductive reasoning.
-
VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection
VisTA uses GRPO reinforcement learning to train a vision-language agent to select external visual tools for a frozen reasoning model, improving accuracy on ChartQA, Geometry3K, BlindTest, and MathVerse.
-
From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualization
EduVisAgent, a five-agent framework, outperforms all baseline AI models at generating pedagogically effective interactive visualizations for STEM problems, according to the new EduVisBench benchmark and its GPT-4o-bas...
-
PointArena: Probing Multimodal Grounding Through Language-Guided Pointing
A three-stage benchmark consisting of 982 pointing tasks, a live pairwise arena with 4,500 votes, and a robot manipulation study shows that pointing-supervised open models such as Molmo-72B can match proprietary model...
-
ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding
Visual editing of input images as a chain of thought improves multimodal LLM accuracy on structured image tasks by 3 to 12 points.
-
MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
MMFactory automatically generates and benchmarks a pool of reusable programmatic vision-language solutions from a few examples, letting users pick one that fits their accuracy and speed constraints.
-
Reinforced Visual Perception with Tools
ReVPT uses GRPO reinforcement learning with a cold-start SFT phase to make Qwen2.5-VL models call visual tools, improving perception benchmarks over SFT and text-only RL baselines.
-
Augmented Vision-Language Models: A Systematic Review
A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.
-
MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning
MINT-CoT-7B interleaves fine-grained visual tokens into each math reasoning step and reports 73.70 on MathVista-Math, 64.72 on GeoQA, and 69.6 on MMStar-Math.
-
Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start
Using a two-stage SFT-plus-GRPO pipeline, the authors train 3B and 7B multimodal models that beat prior open-source models on four math reasoning benchmarks, while showing that reflective 'aha moment' text is not a re...
-
ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuning
Fine-tuning a vision-language-action robot model on teacher-generated reasoning rationales raises average simulated manipulation success by up to 8.6 percentage points over the SpatialVLA baseline.
-
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
AURORA augments LLaVA with tokenized depth maps and bounding boxes as intermediate reasoning steps, improving counting accuracy on some benchmarks, while the strongest depth numbers rest on a benchmark the authors con...
-
Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models
A survey organizes multimodal reasoning research into a staged roadmap and proposes native large multimodal reasoning models that unify perception, generation, and agentic planning.
-
From Simple to Professional: A Combinatorial Controllable Image Captioning Agent
CapAgent converts simple captioning requests into professional multi-constraint instructions and uses a tool-using agent to generate captions, but the report provides no experiments.
-
YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls
A video-guided sound effects model with a learnable audio-visual aggregator and multi-modal chain-of-thought module reports strong VGGSound benchmark scores, but its few-shot claim rests on three qualitative samples.
Discussion (0). Continue with ORCID to comment.