Pith. sign in

REVIEW 4 cited by

In-Context Learning Enables Robot Action Prediction in LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.12782 v2 pith:OOMNIGUS submitted 2024-10-16 cs.RO cs.CL

Yida Yin , Zekai Wang , Yuvan Sharma , Dantong Niu , Trevor Darrell , Roei Herzig This is my paper · ORCID
classification cs.ROcs.CL
keywords actionsllmsrobotdirectlyenablespredictrobopromptdescriptions
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recently, Large Language Models (LLMs) have achieved remarkable success using in-context learning (ICL) in the language domain. However, leveraging the ICL capabilities within LLMs to directly predict robot actions remains largely unexplored. In this paper, we introduce RoboPrompt, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions through ICL without training. Our approach first heuristically identifies keyframes that capture important moments from an episode. Next, we extract end-effector actions from these keyframes as well as the estimated initial object poses, and both are converted into textual descriptions. Finally, we construct a structured template to form ICL demonstrations from these textual descriptions and a task instruction. This enables an LLM to directly predict robot actions at test time. Through extensive experiments and analysis, RoboPrompt shows stronger performance over zero-shot and ICL baselines in simulated and real-world settings. Our project page is available at https://davidyyd.github.io/roboprompt.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

    cs.AI 2024-12 conditional novelty 7.0 of 10

    LMAct is an open benchmark showing that frontier AI models mostly fail to improve at interactive tasks even when given up to 512 expert demonstrations in a million-token context.

  2. Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Auras, a perception-generation disaggregation framework with a public context buffer and asynchronous pipeline executor, raises embodied-agent throughput by 2.54x on average without losing accuracy (102.7%).

  3. In-Context Learning Strategies Emerge Rationally

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Transformer in-context learning is modeled as a posterior-weighted mixture of memorizing and generalizing Bayesian predictors, with a loss-complexity tradeoff governed by three fitted parameters.

  4. Robot Operation of Home Appliances by Reading User Manuals

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A robot system that constructs a symbolic appliance model from a user manual and uses it to reliably execute natural language appliance operation tasks, outperforming direct VLM-based policies.

Pith tools