REVIEW 8 cited by
In-Context Imitation Learning via Next-Token Prediction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We explore how to enhance next-token prediction models to perform in-context imitation learning on a real robot, where the robot executes new tasks by interpreting contextual information provided during the input phase, without updating its underlying policy parameters. We propose In-Context Robot Transformer (ICRT), a causal transformer that performs autoregressive prediction on sensorimotor trajectories without relying on any linguistic data or reward function. This formulation enables flexible and training-free execution of new tasks at test time, achieved by prompting the model with sensorimotor trajectories of the new task composing of image observations, actions and states tuples, collected through human teleoperation. Experiments with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks specified by prompts, even in environment configurations that differ from both the prompt and the training data. In a multitask environment setup, ICRT significantly outperforms current state-of-the-art next-token prediction models in robotics on generalizing to unseen tasks. Code, checkpoints and data are available on https://icrt.dev/
Forward citations
Cited by 8 Pith papers
-
RoboTTT: Context Scaling for Robot Policies
A robot policy that updates its own weights during deployment can use 8,000 steps of history, steadily improving as context grows and enabling one-shot imitation from human videos.
-
WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation
WARP trains a reward model on time-warped successful demonstrations to produce frame-level progress estimates that upweight high-advantage chunks during behavior cloning, maintaining high success rates on suboptimal d...
-
MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.
-
Try Once, Then Optimal: De-Redundified Procedure Memory for Cross-Episode Exploration Amortization
Object-centric procedure memory amortizes hidden-state exploration across encounters, cutting robot manipulation operations 16–30% at non-regressing success.
-
VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory
A transformer-based compressor that turns old observations into fixed-size memory tokens improves non-Markovian imitation-learning robot policies, with large gains on memory-intensive simulated tasks.
-
RoboSSM: Scalable In-context Imitation Learning via State-Space Models
RoboSSM shows that a state-space model backbone can extend in-context imitation learning to prompts much longer than those seen in training, where a Transformer-based baseline degrades.
-
Behavioral Exploration: Learning to Explore via In-Context Adaptation
A coverage-conditioned behavioral cloning policy adapts in-context to its own history, making robots explore new expert-like behaviors online without online reinforcement learning.
-
CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion
Causal Diffusion Policy adds historical action conditioning and attention cache sharing to diffusion-based robot policies, improving success rates on most tested manipulation tasks under degraded observations.
Discussion (0). Sign in to comment.