{"paper":{"title":"A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"A two-level vector quantizer that clusters robot actions while also reconstructing their timestamps improves in-context imitation learning.","cross_cats":[],"primary_cat":"cs.RO","authors_text":"Ali Shah Ali, Andrey Konin, Fawad Javed Fateh, Murad Popattia, M. Zeeshan Zia, Quoc-Huy Tran, Usman Nizamani","submitted_at":"2026-04-16T16:47:08Z","abstract_excerpt":"We present a novel hierarchical spatiotemporal action tokenizer for in-context imitation learning. We first propose a hierarchical approach, which consists of two successive levels of vector quantization. In particular, the lower level assigns input actions to fine-grained subclusters, while the higher level further maps fine-grained subclusters to clusters. Our hierarchical approach outperforms the non-hierarchical counterpart, while mainly exploiting spatial information by reconstructing input actions. Furthermore, we extend our approach by utilizing both spatial and temporal cues, forming a"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Extensive evaluations on multiple simulation and real robotic manipulation benchmarks show that our approach establishes a new state-of-the-art performance in in-context imitation learning.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that the hierarchical spatiotemporal clustering and joint reconstruction of actions plus timestamps will produce tokens that generalize beyond the specific benchmarks and robot platforms used in the evaluations.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"A two-level vector quantization tokenizer that jointly reconstructs robot actions and timestamps to improve in-context imitation learning performance on manipulation benchmarks.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A two-level vector quantizer that clusters robot actions while also reconstructing their timestamps improves in-context imitation learning.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"facd1aa0e5d8555b11319ab38dc526825097a692ba02164df460fddbc075c352"},"source":{"id":"2604.15215","kind":"arxiv","version":2},"verdict":{"id":"5e7c0969-ddef-4b02-9f30-3539fd8e1081","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-19T17:20:03.419167Z","strongest_claim":"Extensive evaluations on multiple simulation and real robotic manipulation benchmarks show that our approach establishes a new state-of-the-art performance in in-context imitation learning.","one_line_summary":"A two-level vector quantization tokenizer that jointly reconstructs robot actions and timestamps to improve in-context imitation learning performance on manipulation benchmarks.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that the hierarchical spatiotemporal clustering and joint reconstruction of actions plus timestamps will produce tokens that generalize beyond the specific benchmarks and robot platforms used in the evaluations.","pith_extraction_headline":"A two-level vector quantizer that clusters robot actions while also reconstructing their timestamps improves in-context imitation learning."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2604.15215/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":52,"sample":[{"doi":"","year":2021,"title":"A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipul","work_id":"70db99d4-b7bf-4ca3-8395-8c071b286702","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Octo: An Open-Source Generalist Robot Policy","work_id":"f9ca0722-8855-48c3-a27a-0eefb7e19253","ref_index":2,"cited_arxiv_id":"2405.12213","is_internal_anchor":true},{"doi":"","year":2024,"title":"Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration 0","work_id":"c8fe1d83-8e88-41b8-ad0b-feb61a748902","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Khazatsky et al","work_id":"63a82a94-47f5-4f81-803e-9ec693af5607","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2020,"title":"T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.NeurIPS, 2020","work_id":"c3601e58-863b-4181-a8b4-c4f8474aab09","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":52,"snapshot_sha256":"c2241be740254df9ac51dbe81b68079d3c74542c081dfe96ff4e162c5002d2dd","internal_anchors":4},"formal_canon":{"evidence_count":2,"snapshot_sha256":"b1a6ef2c4b0222dbc2df4a792d1e447bc64b4414a7679bb7d4d9bce29ad374f1"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}