Pith.

Robotics · arXiv:2610.19495 v1 · submitted 14 Oct 2026 · 39 references

Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution

Yunze Liu, Jing Wang, Shenrui Wu, Wenkun He, Guanqi Zhan, Xianzheng Ma, Junyu Chen, Wenhang Ge

show 11 more authors

Haozhan Tang, Yuchao Gu, Zhuoyang Zhang, Zekai Wang, Dantong Niu, Zhen Dong, Yuke Zhu, Trevor Darrell, Linxi "Jim" Fan, Song Han, Han Cai

The paper says

“Retrieving useful context for a robot requires matching its task, execution stage, and the continuation that should follow.”

more of the abstract

We introduce Robota-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. Predictive retrieval training aligns these endpoints, enabling historical continuations to serve as context at inference time. We construct a multi-source corpus of 1,695,684 cleaned training anchors and introduce Robota-Embed-Bench, with 4,000 queries and a shared 8,000-entry gallery. Robota-Embed achieves 47.33% clip-level and 81.77% episode-level R@1, exceeding action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. Controlled retrieval studies show that removing native action degrades retrieval more than removing vision. We study two complementary applications. Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset, while improving task progress on real robots. Embed2Evolve uses execution feedback to grow task-local memory and refine context use around a frozen GPT-6 Astra policy. Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout, counting both successes and failures. These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 15 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: agrees with the reader. Referee: major revision.

Robota-Embed is a solid, well-engineered VLA retrieval model with thorough ablations and honest reporting; the core novelty—jointly encoding all three modalities in both query and target—holds up. This line is wrong

2 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 39 references say.

The paper says, at Section A.2 and Section 3.3

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The training pipeline relies heavily on weak supervision from Qwen3-VL parsing to generate semantic conflict labels (task, object, action, joint). The paper acknowledges these are 'rule-driven weak supervision' but provides no quantification of parser accuracy on the four-source corpus. Even modest error rates (e.g., 5-10%) in labeling false negatives could corrupt tens of thousands of training pairs in the 1.7M-anchor dataset, potentially biasing the contrastive objective.

more of objection 1

The authors should include an analysis—e.g., human validation of a random sample, or a comparison with hand-authored conflict labels—to demonstrate that the noise level does not undermine the learned embedding quality. Without such analysis, a reader cannot assess the robustness of the central retrieval result.

The paper says, at Section 3.1 and Section6

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The paper fixes the future window to exactly 4 seconds for all retrieval targets across all four datasets. This is a design assumption that may not be optimal for tasks or stages with different temporal scales (e.g., rapid motions vs. slow manipulation).

more of objection 2

While the authors note this as future work, a brief sensitivity study (e.g., evaluating window lengths of 2, 4, 6 seconds on a subset) would help establish whether the results are robust to this choice. Without it, the reported 47.33% clip-level R@1 may be dependent on this specific window length.

5 minor objections

The paper says, at Abstract and Section 1

Pith says: minor objection 1 of 5

The paper repeatedly claims 'to our knowledge the first VLA embedding model to jointly encode all three modalities in both endpoints.' The use of 'first' is assertive; given prior work like VLA-R (2025) and RADAR (2026) that also use action in retrieval spaces, the authors should clarify how their work specifically differs in scope (e.g., unified endpoint embedding, cross-source gallery, predictive alignment).

The paper says, at Table 6 title and the surrounding text

Pith says: minor objection 2 of 5

The token savings statement 'approximately 79.5% fewer policy tokens than this baseline' refers to the Omni-Embed-Nemotron-3B evolved system. The baseline uses only 20% success and the tokens per rollout include failures. This is clearly stated, but the prominence of the number might mislead a casual reader into thinking it is a direct ablation.

more of objection 2

A please add a cautionary note in the main text (not just the appendix) that the comparison is between independently evolved systems with different success rates and memories. The appendix already has this nuance, but a brief sentence in Section 5.2 would help.

show 3 more

The paper says, at Figure 2

Pith says: minor objection 3 of 5

The diagram is informative but the text font for loss terms and modules is very small and may be illegible in print. Consider increasing font size or simplifying the figure.

The paper says, at Section B.4.6 (Table 20) and Section B.5 (Table 21)

Pith says: minor objection 4 of 5

The two sets of masking experiments (four-source final model vs. DROID controlled model) are carefully distinguished. However, the text states 'these interventions answer different questions' but does not explicitly reconcile the differences in effect sizes (e.g., retraining without vision drops 1.9 points while masking vision drops 24.33 points). A brief explanation that the retraining study measures re-learnability while masking study measures reliance under distribution shift would help.

The paper says, at Appendix A.2

Pith says: minor objection 5 of 5

The description of semantic conflict rules is quite detailed, but the pseudocode or a small example would improve clarity. The current text is dense and hard to follow.

What would settle it

Sample 500 conflict pairs from the training data (100 per conflict type), have two independent annotators judge whether the pair is genuinely incompatible according to the stated semantic rule. Compute inter-annotator agreement (Cohen's κ) and per-type accuracy relative to the rule label. If κ < 0.80 or any conflict type has < 90% agreement with the rule, then parser noise is a material concern and a human-validated subset should be used to re-train and re-evaluate.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 15 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 46 s · 32,192 tokens read, 4,867 written
  3. Editor wrote the verdict above 22 s · 33,094 tokens read, 1,597 written
  4. Referee major revision: 2 major objections, 5 minor 41 s · 33,979 tokens read, 1,645 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should know about Robota-Embed. It's a well-executed piece of robot experience retrieval. The core idea is that retrieval works better when you encode vision, language, and native action jointly in both the current-state query and the recorded future continuation. That's genuinely new—prior work either left action out of the retrieval key or encoded it on only one side. The paper delivers on its claims with careful experiments.

The benchmark (Robota-Embed-Bench) is thoughtfully designed, with held-out episodes and separate clip-level and episode-level metrics. The ablation studies are a highlight: retraining and masking both show that action is the second most important modality after language, and the same-instruction gallery controls confirm the model isn't just memorizing task strings. The two downstream applications (Embed2ICL and Embed2Evolve) demonstrate that better retrieval actually helps policy conditioning and memory evolution, with reasonable baselines including random references and an independently evolved Omni-Embed system. The authors are refreshingly honest about confounds—they note that the time-savings numbers are observational, that execution time includes API waits, and that fewer policy calls don't imply fewer physical control ticks.

Soft spots: the weak supervision from Qwen3-VL parsing plus deterministic rules could introduce false negatives in the contrastive loss. The authors acknowledge it's weak, and the retrieval numbers (especially the same-instruction galleries) suggest the signal is adequate, but I'd want to see a parser accuracy estimate on the specific corpus. The token-efficiency comparison against the Omni-Embed baseline uses independently evolved memories, so the improvement isn't entirely attributable to the embedding model alone—but the paper reports this transparently and the overall comparison is still useful.

Neither issue is load-bearing. The core claim—that joint VLA encoding improves retrieval and downstream performance—is well supported. This paper is for anyone working on robot experience retrieval, in-context learning for policies, or memory-augmented control. It deserves a serious referee.

The strongest objection

The training pipeline (Section A.2) relies on Qwen3-VL to parse task instructions into operation names, object identities, and categories, from which deterministic rules generate four types of semantic conflict (task, object, action, joint). These conflicts define negative pairs for the contrastive loss, including hard negatives. If the parser misidentifies objects, operations, or relations, a pair that is actually compatible can be labelled as a conflict (false negative), or an incompatible pair can be excluded (false positive in the validity mask). Because the system uses rule-driven weak supervision rather than human annotations or validated automatic labels, systematic parser errors—especially on rare objects, ambiguous phrasing, or long-tailed tasks—could corrupt a significant fraction of the training signal. The paper acknowledges the labels are 'weak' but does not quantify parser accuracy on its specific four-source corpus. Given the large training set (1.7M anchors), even a 5–10% error rate on conflict classification could introduce tens of thousands of mislabelled negatives, potentially degrading the embedding's ability to genuinely discriminate trajectories rather than exploit artefacts of the parsing pipeline.

The reader identified the same weak-supervision assumption as the weakest link. I agree that it is a genuine vulnerability, but the paper provides strong downstream evidence (ablations, cross-baseline improvements, real-robot gains) that the embedding works despite possible label noise. The concern is load-bearing because if the parser is systematically wrong, the claimed retrieval numbers could be artifacts of flawed negatives rather than genuine semantic understanding. However, the paper’s controlled abalations show that predictive alignment alone (without multimodal grounding) already achieves 75.6% R@1 on DROID (Table 17), indicating the core predictive signal does not depend on the conflict rules alone. The concrete test would settle whether the noise level is tolerable; until then the verdict stands.

The referee's report

The work is significant for its unified treatment of action, vision, and language in a predictive retrieval setting. The benchmark, controlled ablations, and two downstream applications provide strong evidence that including native action in both query and candidate improves task- and stage-sensitive retrieval beyond vision-and-language-only approaches. If the findings hold, this could influence how robot foundation models index and use prior experience, potentially enabling more efficient in-context learning and memory management. The scale of the experiments and the matched comparisons with multiple baselines lend credibility to the claimed advances.

Pith's reading of the paper (a machine's summary, not the authors')

The paper aims to establish that a unified embedding space for vision, language, and native action (VLA) can retrieve useful robotic experience—matching not just the task but also the current execution stage and the continuation that should follow. This matters because current retrieval methods ignore native action, so they conflate different moments in the same task. The authors show that including action in both the query and the retrieved clip raises retrieval accuracy sharply (47% vs 25% clip-level R@1 over action-free baselines) and, in two downstream applications, improves robot task success while reducing policy tokens. If correct, the core claim is that action is not just an output of the policy but a necessary part of the retrieval key for any system that reuses past experience.

What carries the argument. Robota-Embed is a vision-language-action (VLA) embedding model built on a large Transformer backbone (Cosmos3-Edge 4B). Its key novelty is that both endpoints—the current-state query (instruction, interaction crop, synchronized native action) and the future-clip target (instruction, four-second video, aligned native actions)—are jointly encoded into one normalized vector each. Native actions from different sources are mapped through per-source projectors into the shared backbone. The training objective combines predictive retrieval alignment with multimodal grounding and representation regularization, using weak supervision from structured parsing of task instructions and rule-based action-phase comparisons. This architecture produces a single cosine-similarity space where retrieving a recorded continuation requires matching task, visual state, and ongoing motion simultaneously.

Core claim. The paper discovers that jointly encoding vision, language, and native action—especially in both the current-state query and the recorded future continuation—produced a retrieval space that is far more temporally precise than action-free embeddings. On the Robota-Embed-Bench benchmark, the unified VLA model achieves 47.33% clip-level and 81.77% episode-level Recall@1, compared to 25.24% and 48.30% for the best action-free baseline (Gemini Embedding 2). Controlled studies confirm that removing native action degrades retrieval more than removing vision, and that the same VLA embedding can be used to warm-up a policy's visual cache (training-free in-context learning) or to guide the evolution of task-local memory around a frozen policy, increasing success rates on multiple manipulation tasks whil using fewer policy tokens per rollout.

The premise it rests on. The weak supervision labels from structured parsing and rule-based action-phase comparison are accurate enough to define valid negative pairs and modality swaps; if the parser frequently mislabels task relations or the action-phase comparator does not reliably distinguish different trajectories, the contrastive objective could learn from noisy signals.

If the paper is right (Pith's inferences, not the authors')

If Robota-Embed is correct, then any robot that reuses past episodes—whether for few-shot demonstration conditioning, retrieval-augmented generation, or offline memory management—should include native action in the retrieval key to avoid retrieving a clip from the wrong execution phase.

The high clip-level recall (47.33% on a four-source benchmark) suggests that VLA retrieval can support reranker-free or minimal-reranker pipelines, reducing latency for real-time context fetching.

Because the embedding is trained across human, simulated, and real-robot data, it may enable cross-embodiment retrieval: a human video can serve as context for a robot with different kinematics, which the paper demonstrates in the Fast-WAM application where human EgoDex references condition a robot policy.

The Embed2Evolve results imply that retrieval quality directly shapes the efficacy of context evolution around a frozen policy: better embeddings allow smaller memory banks and less frequent refreshes while improving success, reducing policy tokens per rollout.

The paper's controlled studies show that removing action during retraining degrades retrieval (by 18.9 percentage points) more than removing vision (by 1.9 points). An editorial inference is that action may serve as a effective signal for execution-phase discrimination that vision alone cannot provide under domain shift, since vision often changes appearance while action captures the ongoing dynam

The benchmark Robota-Embed-Bench contains 4,000 queries from four sources but does not include out-of-distribution tasks (e.g., insertion vs assembly). One might infer that the model's cross-embodiment retrieval quality on entirely novel tasks has not been measured, and the reported gains may hinge on the presence of semantically similar instructions across sources.

The Embed2Evolve development searches over retrieval hyperparameters independently for each embedding model. An inference is that the final performance gap between Robota-Embed and Omni-Embed-Nemotron-3B (68% vs 33%) may partly reflect different discovered memory/configurations, not just the embedding quality, though the paper's protocol correctly varies both together as a 'herness'.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Retrieving useful context for a robot requires matching its task, execution stage, and the continuation that should follow. We introduce Robota-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. Predictive retrieval training aligns these endpoints, enabling historical continuations to serve as context at inference time. We construct a multi-source corpus of 1,695,684 cleaned training anchors and introduce Robota-Embed-Bench, with 4,000 queries and a shared 8,000-entry gallery. Robota-Embed achieves 47.33% clip-level and 81.77% episode-level R@1, exceeding action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. Controlled retrieval studies show that removing native action degrades retrieval more than removing vision. We study two complementary applications. Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset, while improving task progress on real robots. Embed2Evolve uses execution feedback to grow task-local memory and refine context use around a frozen GPT-6 Astra policy. Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout, counting both successes and failures. These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-15

The authors' main claim, as the classifier picked it out 1.00

“These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution.”

How the main claim is stated: asserted 0.94

What the abstract itself states, as probabilities: claims first 0.95 · states limitation 0.10 · human participants 0.05 · controlled comparison 0.97 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Retrieving useful context for a robot requires matching its task, execution stage, and the continuation that should follow. background 0.99 0.07 0.03 asserted 1.00
We introduce Robota-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. contribution 0.99 0.93 0.07 asserted 0.98
Predictive retrieval training aligns these endpoints, enabling historical continuations to serve as context at inference time. method 0.94 0.69 0.09 asserted 0.99
We construct a multi-source corpus of 1,695,684 cleaned training anchors and introduce Robota-Embed-Bench, with 4,000 queries and a shared 8,000-entry gallery. method 0.68 0.97 0.03 asserted 1.00
Robota-Embed achieves 47.33% clip-level and 81.77% episode-level R@1, exceeding action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. result 1.00 0.96 0.09 asserted 1.00
Controlled retrieval studies show that removing native action degrades retrieval more than removing vision. result 0.97 0.93 0.04 asserted 1.00
We study two complementary applications. contribution 0.90 0.57 0.02 asserted 0.61
Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset, while improving task progress on real robots. result 0.93 0.95 0.08 asserted 1.00
Embed2Evolve uses execution feedback to grow task-local memory and refine context use around a frozen GPT-6 Astra policy. contribution 0.65 0.90 0.04 asserted 0.93
Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. result 1.00 0.94 0.06 asserted 1.00
Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout, counting both successes and failures. result 1.00 0.94 0.04 asserted 1.00
These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution. interpretation 1.00 0.83 0.92 asserted 0.94

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.94

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 9,075 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (14)
Figure 1 of the paper
Figure 1. One joint VLA embedding, two context applications. Robota-Embed jointly encodes vision, language, and native action from human, robot, and simulated experience to retrieve recorded continuations. Left: retrieval accuracy and modality contributions on Robota-Embed-Bench. Right: retrieved demonstrations condition robot policies in Embed2ICL; task-local memory and context use evolve around a fixed GPT-6 Astra policy in Embed2Evolve. Token savings include failed final tests and exclude development; decision/time savings use matched successful pairs. These efficiency comparisons use…
Figure 2 of the paper
Figure 2. Robota-Embed architecture and objectives. Each query is paired with the next four seconds of recorded video and actions. Three objective groups train a joint VLA space. Predictive retrieval alignment (ℒalign) uses bidirectional InfoNCE with exact future-clip positives and task, object, action, or joint conflicts as negatives. Compatible, uncertain, or potentially duplicate non-positives are ignored
Figure 3 of the paper
Figure 3. Multi-source training-data curation. We split episodes before sampling anchors, localize and verify interaction crops, and pair accepted anchors with video and actions from the next four seconds. Query filtering uses only the instruction and current observations. Robota-Embed-Bench. All 4,000 held-out queries search one shared 8,000-clip gallery from the four source datasets. Each source contributes 1,000 queries from 800 episodes, 1,000 paired future clips, and 1,000 nearby same-episode clips. Queries follow Equation 1. Clip-level retrieval treats only the paired clip as…
Figure 4 of the paper
Figure 4. contrasts a paired-clip hit with a temporal near-miss. The latter retrieves a nearby window from the correct episode rather than the paired continuation, illustrating why episode matching and clip localization require separate metrics. (a) Clip-level hit paired rank 1 Current query Retrieved continuation Instruction Write on paper with a pink marker. Top-1 Paired future clip (b) Temporal near-miss paired rank 7 Current query Retrieved continuation Instruction Turn off the black kettle, then open its lid. Top-1 Same-episode neighbor Paired clip
Figure 5 of the paper
Figure 5. Embed2ICL: one retriever, two policy interfaces. Reranked DROID videos warm up DreamZero’s visual cache; top-1 EgoDex matches supply Context Tokens for Fast-WAM’s native cross-attention. Retrieval vectors select experience; videos or tokens condition the policy
Figure 6 of the paper
Figure 6. Retrieved references and real-robot execution. Each task pairs a retrieved reference (top) with real-robot execution (bottom). References come from DROID for DreamZero/YAM and EgoDex for Fast-WAM. Timestamps are relative to each clip’s start. 5.2. Embed2Evolve: Embedding-Guided Context Evolution Embed2Evolve builds task-local memory and refines how it is used through repeated attempts. This memory and context recipe form a harness; GPT-6 Astra and Robota-Embed remain fixed (
Figure 7 of the paper
Figure 7. Embed2Evolve. Development grows memory and optimizes its use; testing fixes both. GPT-6 Astra proposes actions for a fixed controller. The model comparison replaces Robota-Embed in both phases. Images show the PullCubeTool task. history, prioritizing success before successful-execution efficiency. The selected memory and recipe are frozen for fresh-test initializations (
Figure 8 of the paper
Figure 8. Selected qualitative evidence across embodiments. Each query searches only the illustrated target source. The top-ranked clips share placement or removal roles with their queries. C. Embed2ICL Implementation and Supplementary Results This section covers DreamZero’s training-free video-cache warm-up and Fast-WAM’s training-based Context Tokens, followed by integration controls. C.1. DreamZero-DROID Reference Warm-Up Embed2ICL initializes frozen DreamZero-DROID’s cache with a retrieved and reranked video (
Figure 9 of the paper
Figure 9. Training-free Embed2ICL. Robota-Embed retrieves candidate VLA clips, and the Reranking Agent selects a task-relevant reference containing completion evidence. Its video initializes DreamZero-DROID’s visual key–value cache. The policy remains frozen while live observations update the cache during execution. Evaluation task 2.0 s before release Release frame 0.8 s after release Cube → Bowl DROID reference block → bowl Can → Mug DROID reference cap → container Banana → Bin DROID reference banana → bowl
Figure 10 of the paper
Figure 10. DROID references for training-free Embed2ICL. Each row shows one task’s fixed reference from a single camera, before, at, and after the detected final gripper release. DreamZero-DROID receives all three synchronized four-second video streams. C.3. Training-Free Embed2ICL on the YAM Robot Policy and reference. We adapt DreamZero-AgiBot [9] to YAM for 20 epochs on 55 play trajectories, comprising approximately 30 minutes of interaction. Both conditions share this adapted policy; Embed2ICL adds retrieved video through visual KV-cache warm-up without further policy updates. It reuses…
Figure 11 of the paper
Figure 11. Training-based Embed2ICL. Offline encoding caches each candidate’s retrieval vector, Context Tokens, and context mask. Online cosine top-1 search selects a reference and loads its tokens and mask. The prefilling adapter projects the tokens to Fast-WAM’s context width and appends them to the native text and proprioceptive conditioning for cross-attention. Offline cache and online lookup. Offline, we cache each candidate’s retrieval vector, Context Tokens, and context mask 𝑚𝑖 . Online, we encode the robot query as 𝑞, find 𝑖 * = arg max𝑖 𝑞 ⊤𝑘𝑖 , and load (𝐶𝑖 * , 𝑚𝑖 * ) without re-en
Figure 12 of the paper
Figure 12. LIBERO G3: reference and final states. Successes out of 50 per condition. C.6. Training-Based Integration Ablation We compare two complete integration settings: Embed2ICL’s native-context prefilling with policy fine-tuning, and a frozen-backbone reference side branch. Both use the Robota-Embed retriever, EgoDex index, cosine top-1 selection, and complete 32 × 1024 Context Tokens. They differ in both the conditioning path and trainable scope. Side-branch setting. A shared reference encoder projects the cached tokens to width 512. Each of the 30 video-expert and 30 action-expert…
Figure 13 of the paper
Figure 13. From Direct control to a selected harness. PullCubeTool’s R4 → R5 → R8 parent–child sequence grows memory while reducing retained history and retrieval frequency. The bottom filmstrips show the same fresh-test initialization completed successfully by both systems. Task execution time includes API waits. 29
Figure 14 of the paper
Figure 14. traces Robota-Embed’s candidate performance. Decision counts average each round’s successful trials; test-time efficiency uses matched initializations (Section D.5). Hollow diamonds mark selected harnesses, which need not come from the final round. Task success (%) 0 50 100 2 3 4 5 6 7 8 0 25 50 2 3 4 5 6 7 8 Policy decisions per successful episode Development round Development round Poke Cube Stack Cube Lift Peg Upright Pull Cube with Tool Two-Robot Stack Cube Selected harness

Figures are the authors', reproduced from the paper.

References (39)
  1. [1] Generalist Team. Gen-1.5: Embodied foundation models are one-shot learners.Generalist AI Blog, 2026. https://generalistai.com/blog/gen-1.5 on Pith
  2. [2] Skild AI. Introducing s1: In-context learning for robotics. August 2026. URLhttps://skild.ai/blogs/s1 on Pith
  3. [3] OpenAI. Gpt-6 astra: A new generation of intelligence.https://openai.com/index/gpt-6-astra/, 2026 on Pith
  4. [4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag... on Pith
  5. [5] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180–15190. IEEE, 2023 on Pith
  6. [6] Mengyao Xu, Wenfei Zhou, Yauhen Babakhin, Gabriel Moreira, Ronay Ak, Radek Osmulski, Bo Liu, Even Oldridge, and Benedikt Schifferer. Omni-embed-nemotron: A unified multimodal retrieval model for text, image, audio, and video.arXiv preprint arXiv:2510.03458, 2025 on PitharXiv
  7. [7] Jaeyong Sung, Ian Lenz, and Ashutosh Saxena. Deep multimodal embedding: Manipulating novel objects with point-clouds, language and trajectories. In2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2794–2801. IEEE, 2017 on Pith
  8. [8] Krishan Rana, Andrew Melnik, and Niko Sünderhauf. Contrastive language, action, and state pre-training for robot learning.arXiv preprint arXiv:2304.10782, 2023 on PithPith's reviewarXiv
  9. [9] Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026 on PithPith's reviewarXiv
  10. [10] Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026 on PithPith's reviewarXiv
  11. [11] Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Bangalath, et al. Perception encoder: The best visual embeddings are not at the output of the network.Advances in Neural Information Processing ... on Pith
  12. [12] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. InEuropean conference on computer vision, pages 396–416. Springer, 2024 on Pith
  13. [13] Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, et al. Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents.arXiv preprint arXiv:2507.04590, 2025 on PithPith's reviewarXiv
  14. [14] Junjie Zhou, Ke Mei, Lei Li, Tianyi Wang, Fengyun Rao, and Jing Lyu. Wemm-embedding: Wechat multi-modal embedding technical report.arXiv preprint arXiv:2608.24053, 2026 on PithPith's reviewarXiv
  15. [15] Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, Gustavo Hernández Ábrego, Shih-Cheng Huang, Aashi Jain, Daniel Salz, Sonam Goenka, Chaitra Hegde, Ji Ma, et al. Gemini embedding 2: A native multimodal embedding model from gemini.arXiv preprint arXiv:2605.27295, 2026 on PithPith's reviewarXiv
  16. [16] Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the- art multimodal retrieval and ranking.arXiv preprint arXiv:2601.04720, 2026 on PithPith's reviewarXiv
  17. [17] Hyunki Seong, Seongwoo Moon, Hojin Ahn, Jehun Kang, and David Hyunchul Shim. Vla-r: Vision-language action retrieval toward open-world end-to-end autonomous driving.arXiv preprint arXiv:2511.12405, 2025 on PitharXiv
  18. [18] Jensen Gao, Dorsa Sadigh, Sandy Huang, and Dhruv Shah. Grounding robot generalization in training data via retrieval-augmented vlms.arXiv preprint arXiv:2603.11426, 2026 on PithPith's reviewarXiv
  19. [19] Zillur Rahman, Eddison Pham, Alejandro Daniel Noel, and Cristian Meo. Roser: Few-shot robotic sequence retrieval for scalable robot learning. 2026 on Pith
  20. [20] Yichen Zhu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Retrieval-augmented embodied agents. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17985–17995. IEEE, 2024. 11 Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution on Pith
show 19 more
  1. [21] Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, and Insup Lee. Regent: A retrieval-augmented generalist agent that can act in-context in new environments. InInternational Conference on Learning Representations, volume 2025, pages 39385–39414, 2025 on Pith
  2. [22] Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, and Insup Lee. Ricl: Adding in-context adaptability to pre-trained vision-language-action models.arXiv preprint arXiv:2508.02062, 2025 on PithPith's reviewarXiv
  3. [23] Fu Chen, Xin Ding, Bingjia Huang, Xiangyu Li, Mingju Wang, Jiawei He, Kun Li, Wei Sun, Yunxin Liu, Hao Wu, et al. Zeva: In-context causal learning for generalizable embodied manipulation.arXiv preprint arXiv:2608.30880, 2026 on PithPith's reviewarXiv
  4. [24] Ajay Sridhar, Jennifer Pan, Satvik Sharma, and Chelsea Finn. Scaling up memory for robotic control via experience retrieval. InInternational Conference on Learning Representations, volume 2026, pages 97142–97166, 2026 on Pith
  5. [25] Izat Temiraliev, Diji Yang, and Yi Zhang. Retrieval-augmented robots via retrieve-reason-act.arXiv preprint arXiv:2603.02688, 2026 on PitharXiv
  6. [26] Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, et al. Zero-wam: In-context world-action modeling from human videos for open-ended task generalization.arXiv preprint arXiv:2608.26103, 2026 on PithPith's reviewarXiv
  7. [27] Guangyan Chen, Meiling Wang, Te Cui, Zichen Zhou, Qi Shao, Shalfun Li, Hang Su, Roy Gan, Hao Wang, Mengyin Fu, et al. Robots acquire manipulation skills in seconds from a single human video.arXiv preprint arXiv:2607.20033, 2026 on PithPith's reviewarXiv
  8. [28] Georgios Papagiannis, Norman Di Palo, Pietro Vitiello, and Edward Johns. R+ x: Retrieval and execution from everyday human videos. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8284–8290. IEEE, 2025 on Pith
  9. [29] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023 on Pith
  10. [30] Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026 on PithPith's reviewarXiv
  11. [31] Ryan Hoque, Peide Huang, David Yoon, Jian Zhang, et al. Egodex: Learning dexterous manipulation from large-scale egocentric video. InInternational Conference on Learning Representations, volume 2026, pages 4218–4237, 2026 on Pith
  12. [32] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12... on PithPith's reviewarXiv
  13. [33] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36:44776–44791, 2023 on Pith
  14. [34] Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation.arXiv preprint arXiv:2412.13877, 2024 on PithPith's reviewarXiv
  15. [35] Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, et al. T-rex: Tactile-reactive dexterous manipulation.arXiv preprint arXiv:2606.17055, 2026 on PithPith's reviewarXiv
  16. [36] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-Kai Chan, et al. Maniskill3: Gpu parallelized robot simulation and rendering for generalizable embodied ai. In7th Robot Learning Workshop: Towards Robots with ... on Pith
  17. [37] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris Coll-Vinent, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. InInternational conference on learning representations, vol... on Pith
  18. [38] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025 on PithPith's reviewarXiv
  19. [39] Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026. 12 Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evoluti... on PithPith's reviewarXiv
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment