{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:2RCKNUMH66EZWGGSE6X4BKL32A","short_pith_number":"pith:2RCKNUMH","schema_version":"1.0","canonical_sha256":"d444a6d187f7899b18d227afc0a97bd0068e1f401790682db06cd5a4f03c3c19","source":{"kind":"arxiv","id":"2310.10639","version":1},"attestation_state":"computed","paper":{"title":"Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"A finetuned image-editing diffusion model generates subgoal images that let a low-level policy complete manipulation tasks on objects and instructions absent from robot training data.","cross_cats":[],"primary_cat":"cs.RO","authors_text":"Aviral Kumar, Chelsea Finn, Homer Walke, Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Sergey Levine","submitted_at":"2023-10-16T17:57:23Z","abstract_excerpt":"If generalist robots are to operate in truly unstructured environments, they need to be able to recognize and reason about novel objects and scenarios. Such objects and scenarios might not be present in the robot's own training data. We propose SuSIE, a method that leverages an image-editing diffusion model to act as a high-level planner by proposing intermediate subgoals that a low-level controller can accomplish. Specifically, we finetune InstructPix2Pix on video data, consisting of both human videos and robot rollouts, such that it outputs hypothetical future \"subgoal\" observations given th"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":false},"canonical_record":{"source":{"id":"2310.10639","kind":"arxiv","version":1},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.RO","submitted_at":"2023-10-16T17:57:23Z","cross_cats_sorted":[],"title_canon_sha256":"1afa1eb71c2abc380fef589399b6b9ce5c674f28e4c2d67bdf69760ca03cf0eb","abstract_canon_sha256":"3a5b18f67532489bed26bb37ff3209f173b78c96f1e8dc820dd2ab374d3cfa83"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:48.917523Z","signature_b64":"nqn+u3XR1cV6GQY/kmoXqPfIyTLmMlxQMfXC5W4WVvndviuo89mKTeN+nwyvTY7B1drUrKxKVVXRjgJr+DsNAg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"d444a6d187f7899b18d227afc0a97bd0068e1f401790682db06cd5a4f03c3c19","last_reissued_at":"2026-05-17T23:38:48.917021Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:48.917021Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"A finetuned image-editing diffusion model generates subgoal images that let a low-level policy complete manipulation tasks on objects and instructions absent from robot training data.","cross_cats":[],"primary_cat":"cs.RO","authors_text":"Aviral Kumar, Chelsea Finn, Homer Walke, Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Sergey Levine","submitted_at":"2023-10-16T17:57:23Z","abstract_excerpt":"If generalist robots are to operate in truly unstructured environments, they need to be able to recognize and reason about novel objects and scenarios. Such objects and scenarios might not be present in the robot's own training data. We propose SuSIE, a method that leverages an image-editing diffusion model to act as a high-level planner by proposing intermediate subgoals that a low-level controller can accomplish. Specifically, we finetune InstructPix2Pix on video data, consisting of both human videos and robot rollouts, such that it outputs hypothetical future \"subgoal\" observations given th"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We achieve state-of-the-art results on the CALVIN benchmark, and also demonstrate robust generalization on real-world manipulation tasks, beating strong baselines that have access to privileged information or that utilize orders of magnitude more compute and training data.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That subgoal images generated by the finetuned diffusion model remain sufficiently accurate and executable for the low-level policy when the robot encounters objects, lighting, or instructions outside the finetuning distribution.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"SuSIE uses a finetuned InstructPix2Pix diffusion model to propose subgoal images that guide a low-level goal-conditioned policy, achieving SOTA zero-shot performance on CALVIN and real-world manipulation.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A finetuned image-editing diffusion model generates subgoal images that let a low-level policy complete manipulation tasks on objects and instructions absent from robot training data.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"253f64bfecedb08354612fff5d71577a46cdc5be8de0d115200c18ebd5862462"},"source":{"id":"2310.10639","kind":"arxiv","version":1},"verdict":{"id":"489a6c1b-4720-498b-a71d-957d779ff58c","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T05:50:34.186645Z","strongest_claim":"We achieve state-of-the-art results on the CALVIN benchmark, and also demonstrate robust generalization on real-world manipulation tasks, beating strong baselines that have access to privileged information or that utilize orders of magnitude more compute and training data.","one_line_summary":"SuSIE uses a finetuned InstructPix2Pix diffusion model to propose subgoal images that guide a low-level goal-conditioned policy, achieving SOTA zero-shot performance on CALVIN and real-world manipulation.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That subgoal images generated by the finetuned diffusion model remain sufficiently accurate and executable for the low-level policy when the robot encounters objects, lighting, or instructions outside the finetuning distribution.","pith_extraction_headline":"A finetuned image-editing diffusion model generates subgoal images that let a low-level policy complete manipulation tasks on objects and instructions absent from robot training data."},"references":{"count":66,"sample":[{"doi":"","year":2023,"title":"Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Confer","work_id":"773059c2-43a3-4b27-89bc-0f2c9d64cff5","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"Compositional founda- tion models for hierarchical planning","work_id":"31122d0b-b1dc-41ad-8818-e05c05898071","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2020,"title":"Fitvid: Overfitting in pixel-level video prediction","work_id":"98b75ffa-1d61-4641-a59f-5967267b7d2c","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"Robotic offline rl from internet videos via value-function pre-training","work_id":"51083cdf-e320-4588-923c-a475af2728ba","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"Introducing ChatGPT and Whis- per APIs","work_id":"749457d5-cf67-4ff2-893e-3b8b82dcff9b","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":66,"snapshot_sha256":"fb7e009c4ae4a695ff90c22c427ee6aa1089b29b0a973bf0e7abd1ca1aba7c38","internal_anchors":17},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2310.10639","created_at":"2026-05-17T23:38:48.917100+00:00"},{"alias_kind":"arxiv_version","alias_value":"2310.10639v1","created_at":"2026-05-17T23:38:48.917100+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2310.10639","created_at":"2026-05-17T23:38:48.917100+00:00"},{"alias_kind":"pith_short_12","alias_value":"2RCKNUMH66EZ","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_16","alias_value":"2RCKNUMH66EZWGGS","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_8","alias_value":"2RCKNUMH","created_at":"2026-05-18T12:33:33.725879+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":34,"internal_anchor_count":34,"sample":[{"citing_arxiv_id":"2506.14135","citing_title":"GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulation","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2411.02385","citing_title":"How Far is Video Generation from World Model: A Physical Law Perspective","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19319","citing_title":"SWEET: Sparse World Modeling with Image Editing for Embodied Task Execution","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2507.12768","citing_title":"AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulation","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2401.03568","citing_title":"Agent AI: Surveying the Horizons of Multimodal Interaction","ref_index":162,"is_internal_anchor":true},{"citing_arxiv_id":"2402.10885","citing_title":"3D Diffuser Actor: Policy Diffusion with 3D Scene Representations","ref_index":47,"is_internal_anchor":true},{"citing_arxiv_id":"2506.15799","citing_title":"Steering Your Diffusion Policy with Latent Space Reinforcement Learning","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2409.12514","citing_title":"TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2507.15493","citing_title":"GR-3 Technical Report","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2512.00961","citing_title":"Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2401.00025","citing_title":"Any-point Trajectory Modeling for Policy Learning","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2507.04447","citing_title":"DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge","ref_index":118,"is_internal_anchor":true},{"citing_arxiv_id":"2601.07060","citing_title":"PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2509.06951","citing_title":"F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2507.12898","citing_title":"Vidar: Embodied Video Diffusion Model for Generalist Manipulation","ref_index":42,"is_internal_anchor":true},{"citing_arxiv_id":"2503.22020","citing_title":"CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2510.10125","citing_title":"Ctrl-World: A Controllable Generative World Model for Robot Manipulation","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2508.00795","citing_title":"Video Generators are Robot Policies","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2602.20231","citing_title":"UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models","ref_index":52,"is_internal_anchor":true},{"citing_arxiv_id":"2409.16283","citing_title":"Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2502.05855","citing_title":"DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control","ref_index":57,"is_internal_anchor":true},{"citing_arxiv_id":"2604.04974","citing_title":"From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12167","citing_title":"From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2512.13030","citing_title":"Motus: A Unified Latent Action World Model","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2412.14803","citing_title":"Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations","ref_index":81,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A","json":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A.json","graph_json":"https://pith.science/api/pith-number/2RCKNUMH66EZWGGSE6X4BKL32A/graph.json","events_json":"https://pith.science/api/pith-number/2RCKNUMH66EZWGGSE6X4BKL32A/events.json","paper":"https://pith.science/paper/2RCKNUMH"},"agent_actions":{"view_html":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A","download_json":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A.json","view_paper":"https://pith.science/paper/2RCKNUMH","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2310.10639&json=true","fetch_graph":"https://pith.science/api/pith-number/2RCKNUMH66EZWGGSE6X4BKL32A/graph.json","fetch_events":"https://pith.science/api/pith-number/2RCKNUMH66EZWGGSE6X4BKL32A/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A/action/timestamp_anchor","attest_storage":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A/action/storage_attestation","attest_author":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A/action/author_attestation","sign_citation":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A/action/citation_signature","submit_replication":"https://pith.science/pith/2RCKNUMH66EZWGGSE6X4BKL32A/action/replication_record"}},"created_at":"2026-05-17T23:38:48.917100+00:00","updated_at":"2026-05-17T23:38:48.917100+00:00"}