REVIEW 7 cited by
Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We show that off-the-shelf text-based Transformers, with no additional training, can perform few-shot in-context visual imitation learning, mapping visual observations to action sequences that emulate the demonstrator's behaviour. We achieve this by transforming visual observations (inputs) and trajectories of actions (outputs) into sequences of tokens that a text-pretrained Transformer (GPT-4 Turbo) can ingest and generate, via a framework we call Keypoint Action Tokens (KAT). Despite being trained only on language, we show that these Transformers excel at translating tokenised visual keypoint observations into action trajectories, performing on par or better than state-of-the-art imitation learning (diffusion policies) in the low-data regime on a suite of real-world, everyday tasks. Rather than operating in the language domain as is typical, KAT leverages text-based Transformers to operate in the vision and action domains to learn general patterns in demonstration data for highly efficient imitation learning, indicating promising new avenues for repurposing natural language models for embodied tasks. Videos are available at https://www.robot-learning.uk/keypoint-action-tokens.
Forward citations
Cited by 7 Pith papers
-
MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.
-
Try Once, Then Optimal: De-Redundified Procedure Memory for Cross-Episode Exploration Amortization
Object-centric procedure memory amortizes hidden-state exploration across encounters, cutting robot manipulation operations 16–30% at non-regressing success.
-
RoboSSM: Scalable In-context Imitation Learning via State-Space Models
RoboSSM shows that a state-space model backbone can extend in-context imitation learning to prompts much longer than those seen in training, where a Transformer-based baseline degrades.
-
Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
A semantic keypoint graph matched to novel objects lets imitation-learned manipulation policies generalize with a quarter of the demonstrations.
-
VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers
A convolutional residual VQ-VAE action tokenizer trained on over 100x more data than prior work improves OpenVLA success rates and inference speed on several manipulation tasks.
-
Reducing Latency in LLM-Based Natural Language Commands Processing for Robot Navigation
A direct ChatGPT-to-ROS2 voice command pipeline for robot navigation is reported to cut latency by 7.01%, but the paper's own data do not reproduce that number.
- CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
Discussion (0). Sign in to comment.