{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:TOPMBZWI6RZ5XX2DT5CY6BIJNU","short_pith_number":"pith:TOPMBZWI","schema_version":"1.0","canonical_sha256":"9b9ec0e6c8f473dbdf439f458f05096d27563615659ffcb6b3014e1b29554d09","source":{"kind":"arxiv","id":"2410.11758","version":2},"attestation_state":"computed","paper":{"title":"Latent Action Pretraining from Videos","license":"http://creativecommons.org/licenses/by/4.0/","headline":"LAPA learns discrete latent actions from unlabeled videos with VQ-VAE, pretrains a VLA model to predict them, and finetunes on small robot datasets to outperform both video-only baselines and labeled SOTA VLA models on language-conditioned manipulation tasks.","cross_cats":["cs.CL","cs.CV","cs.LG"],"primary_cat":"cs.RO","authors_text":"Ajay Mandlekar, Baolin Peng, Bill Yuchen Lin, Byeongguk Jeon, Dieter Fox, Jianfeng Gao, Jianwei Yang, Joel Jang, Kimin Lee, Lars Liden, Luke Zettlemoyer, Minjoon Seo, Reuben Tan, Sejune Joo, Seonghyeon Ye, Yu-Wei Chao","submitted_at":"2024-10-15T16:28:09Z","abstract_excerpt":"We introduce Latent Action Pretraining for general Action models (LAPA), an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":true},"canonical_record":{"source":{"id":"2410.11758","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.RO","submitted_at":"2024-10-15T16:28:09Z","cross_cats_sorted":["cs.CL","cs.CV","cs.LG"],"title_canon_sha256":"697b959e671bb5612b90d4d518b6f82f95585f0178d02046d14fbe7347b291a6","abstract_canon_sha256":"c6fad7a3cbbc7d753ab0f1d39139b2cafc029ca7da297eb88a9da486880d7cbb"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-18T02:36:37.771795Z","signature_b64":"Th7JDnaJ5voq/B+SAjifz65TIOvfk0nczGb/uap3ui6Yf/XZHjrdrZyQtzTWZQ4u9URGt+hIjEMKnvJSxyqvAg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"9b9ec0e6c8f473dbdf439f458f05096d27563615659ffcb6b3014e1b29554d09","last_reissued_at":"2026-05-18T02:36:37.771191Z","signature_status":"signed_v1","first_computed_at":"2026-05-18T02:36:37.771191Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Latent Action Pretraining from Videos","license":"http://creativecommons.org/licenses/by/4.0/","headline":"LAPA learns discrete latent actions from unlabeled videos with VQ-VAE, pretrains a VLA model to predict them, and finetunes on small robot datasets to outperform both video-only baselines and labeled SOTA VLA models on language-conditioned manipulation tasks.","cross_cats":["cs.CL","cs.CV","cs.LG"],"primary_cat":"cs.RO","authors_text":"Ajay Mandlekar, Baolin Peng, Bill Yuchen Lin, Byeongguk Jeon, Dieter Fox, Jianfeng Gao, Jianwei Yang, Joel Jang, Kimin Lee, Lars Liden, Luke Zettlemoyer, Minjoon Seo, Reuben Tan, Sejune Joo, Seonghyeon Ye, Yu-Wei Chao","submitted_at":"2024-10-15T16:28:09Z","abstract_excerpt":"We introduce Latent Action Pretraining for general Action models (LAPA), an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between"},"claims":{"count":3,"items":[{"kind":"strongest_claim","text":"Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That discrete latent actions extracted from human manipulation videos contain sufficient transferable information to map effectively to robot actions during finetuning and yield better generalization than direct supervised training on labeled robot data.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"LAPA learns discrete latent actions from unlabeled videos with VQ-VAE, pretrains a VLA model to predict them, and finetunes on small robot datasets to outperform both video-only baselines and labeled SOTA VLA models on language-conditioned manipulation tasks.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"}],"snapshot_sha256":"1dd1973c2177d350f72c677c66595154b23809b747fb41d0ae158dbb0bcf80d5"},"source":{"id":"2410.11758","kind":"arxiv","version":2},"verdict":{"id":"f5188e97-42fa-4359-bebc-1948939438a6","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T02:23:37.058623Z","strongest_claim":"Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions.","one_line_summary":"LAPA learns discrete latent actions from unlabeled videos with VQ-VAE, pretrains a VLA model to predict them, and finetunes on small robot datasets to outperform both video-only baselines and labeled SOTA VLA models on language-conditioned manipulation tasks.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That discrete latent actions extracted from human manipulation videos contain sufficient transferable information to map effectively to robot actions during finetuning and yield better generalization than direct supervised training on labeled robot data.","pith_extraction_headline":""},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":2,"snapshot_sha256":"c5bc0f3d18b0aa144e8d64d0cbcc47c58090d38db979e35ae20bd97ea25f93f6"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2410.11758","created_at":"2026-05-18T02:36:37.771322+00:00"},{"alias_kind":"arxiv_version","alias_value":"2410.11758v2","created_at":"2026-05-18T02:36:37.771322+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2410.11758","created_at":"2026-05-18T02:36:37.771322+00:00"},{"alias_kind":"pith_short_12","alias_value":"TOPMBZWI6RZ5","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"TOPMBZWI6RZ5XX2D","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"TOPMBZWI","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":38,"internal_anchor_count":16,"sample":[{"citing_arxiv_id":"2505.07813","citing_title":"DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies","ref_index":57,"is_internal_anchor":true},{"citing_arxiv_id":"2512.24497","citing_title":"What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?","ref_index":74,"is_internal_anchor":true},{"citing_arxiv_id":"2507.12440","citing_title":"EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos","ref_index":66,"is_internal_anchor":true},{"citing_arxiv_id":"2605.15725","citing_title":"DiLA: Disentangled Latent Action World Models","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2507.00990","citing_title":"Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations","ref_index":128,"is_internal_anchor":true},{"citing_arxiv_id":"2412.14058","citing_title":"What Matters in Building Vision-Language-Action Models for Generalist Robots","ref_index":48,"is_internal_anchor":true},{"citing_arxiv_id":"2505.03233","citing_title":"GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2507.15493","citing_title":"GR-3 Technical Report","ref_index":77,"is_internal_anchor":true},{"citing_arxiv_id":"2602.11075","citing_title":"RISE: Self-Improving Robot Policy with Compositional World Model","ref_index":89,"is_internal_anchor":true},{"citing_arxiv_id":"2507.23682","citing_title":"villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models","ref_index":69,"is_internal_anchor":true},{"citing_arxiv_id":"2602.20231","citing_title":"UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2512.15692","citing_title":"mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs","ref_index":58,"is_internal_anchor":true},{"citing_arxiv_id":"2604.19092","citing_title":"RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation","ref_index":58,"is_internal_anchor":true},{"citing_arxiv_id":"2605.10819","citing_title":"ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models","ref_index":55,"is_internal_anchor":true},{"citing_arxiv_id":"2605.13452","citing_title":"CUBic: Coordinated Unified Bimanual Perception and Control Framework","ref_index":57,"is_internal_anchor":true},{"citing_arxiv_id":"2605.13403","citing_title":"RotVLA: Rotational Latent Action for Vision-Language-Action Model","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2604.04974","citing_title":"From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data","ref_index":104,"is_internal_anchor":false},{"citing_arxiv_id":"2504.02792","citing_title":"Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets","ref_index":51,"is_internal_anchor":false},{"citing_arxiv_id":"2412.14803","citing_title":"Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations","ref_index":132,"is_internal_anchor":false},{"citing_arxiv_id":"2605.03637","citing_title":"Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing","ref_index":26,"is_internal_anchor":false},{"citing_arxiv_id":"2604.27448","citing_title":"LA-Pose: Latent Action Pretraining Meets Pose Estimation","ref_index":39,"is_internal_anchor":false},{"citing_arxiv_id":"2605.06222","citing_title":"When to Trust Imagination: Adaptive Action Execution for World Action Models","ref_index":29,"is_internal_anchor":false},{"citing_arxiv_id":"2605.09613","citing_title":"SABER: A Scalable Action-Based Embodied Dataset for Real-World VLA Adaptation","ref_index":30,"is_internal_anchor":false},{"citing_arxiv_id":"2605.10819","citing_title":"ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models","ref_index":55,"is_internal_anchor":false},{"citing_arxiv_id":"2605.06222","citing_title":"When to Trust Imagination: Adaptive Action Execution for World Action Models","ref_index":28,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU","json":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU.json","graph_json":"https://pith.science/api/pith-number/TOPMBZWI6RZ5XX2DT5CY6BIJNU/graph.json","events_json":"https://pith.science/api/pith-number/TOPMBZWI6RZ5XX2DT5CY6BIJNU/events.json","paper":"https://pith.science/paper/TOPMBZWI"},"agent_actions":{"view_html":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU","download_json":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU.json","view_paper":"https://pith.science/paper/TOPMBZWI","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2410.11758&json=true","fetch_graph":"https://pith.science/api/pith-number/TOPMBZWI6RZ5XX2DT5CY6BIJNU/graph.json","fetch_events":"https://pith.science/api/pith-number/TOPMBZWI6RZ5XX2DT5CY6BIJNU/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU/action/timestamp_anchor","attest_storage":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU/action/storage_attestation","attest_author":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU/action/author_attestation","sign_citation":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU/action/citation_signature","submit_replication":"https://pith.science/pith/TOPMBZWI6RZ5XX2DT5CY6BIJNU/action/replication_record"}},"created_at":"2026-05-18T02:36:37.771322+00:00","updated_at":"2026-05-18T02:36:37.771322+00:00"}