REVIEW 6 cited by
Multimodal Procedural Planning via Dual Text-Image Prompting
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Embodied agents have achieved prominent performance in following human instructions to complete tasks. However, the potential of providing instructions informed by texts and images to assist humans in completing tasks remains underexplored. To uncover this capability, we present the multimodal procedural planning (MPP) task, in which models are given a high-level goal and generate plans of paired text-image steps, providing more complementary and informative guidance than unimodal plans. The key challenges of MPP are to ensure the informativeness, temporal coherence,and accuracy of plans across modalities. To tackle this, we propose Text-Image Prompting (TIP), a dual-modality prompting method that jointly leverages zero-shot reasoning ability in large language models (LLMs) and compelling text-to-image generation ability from diffusion-based models. TIP improves the interaction in the dual modalities using Text-to-Image Bridge and Image-to-Text Bridge, allowing LLMs to guide the textual-grounded image plan generation and leveraging the descriptions of image plans to ground the textual plan reversely. To address the lack of relevant datasets, we collect WIKIPLAN and RECIPEPLAN as a testbed for MPP. Our results show compelling human preferences and automatic scores against unimodal and multimodal baselines on WIKIPLAN and RECIPEPLAN in terms of informativeness, temporal coherence, and plan accuracy. Our code and data: https://github.com/YujieLu10/MPP.
Forward citations
Cited by 6 Pith papers
-
LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning
A VLM-based planner with task-oriented segmentation reranking and a clause-level condition retriever reports state-of-the-art scores on ActPlan-1K and ALFRED, though the reported ablation numbers are internally inconsistent.
-
Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
A self-critique, revision, and verification loop makes small vision-language models produce more detailed and more executable robot plans, beating their own baselines and, on the paper's judge-based evaluation, plans ...
-
CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion
CookingDiffusion generates step-by-step cooking images by feeding previous step texts and images as memory into Stable Diffusion, beating baselines on FID and a new CLIP-based consistency score.
-
VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting
VG-TVP enriches LLM-generated text plans with captions from instructional videos and produces a short video per step; human raters prefer it over text-only baselines.
-
PDDLFuse: A Tool for Generating Diverse Planning Domains
A tool that fuses two PDDL domains with random action mutations to produce new, guaranteed-solvable planning problems.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.