Pith. sign in

REVIEW 2 major objections 5 minor 58 references

Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CONPE learns one contrastive visual prompt per domain factor and mixes the resulting CLIP embeddings with cosine-guidance attention, yielding zero-shot policy adaptation that outperforms prior state-of-the-art across navigation…

desk verdict The prompt-ensemble idea is worth a look, but the headline zero-shot results are confounded by the extra expert data used for prompt learning, and the paper omits the control that would settle it. read the letter →

arxiv 2412.11484 v1 pith:F5NIYBDR submitted 2024-12-16 cs.AI cs.CVcs.RO

Wonje Choi , Woo Kyung Kim , SeungHyun Kim , Honguk Woo This is my paper · ORCID
classification cs.AIcs.CVcs.RO
keywords zero-shotadaptationembodiedreinforcementlearningvisualpromptensemblecontrastiveCLIPvision-languagemodeldomaingeneralizationpolicyattentionmechanism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an embodied reinforcement learning agent can adapt to unseen visual domains without any retraining if the visual encoder is augmented with a pool of learned prompts, one per domain factor such as camera position, stride length, or illumination. Each prompt is trained with a contrastive loss on a few expert demonstrations so that it pulls together observations that share behavior while separating observations across domains. During policy learning, the prompted CLIP embeddings are combined by an attention module whose weights are steered by cosine similarity between the image and each prompted embedding, giving a state representation that is both domain-invariant and task-specific. The authors show this yields higher zero-shot success than prior methods in AI2THOR navigation, egocentric-Metaworld manipulation, and CARLA driving, and it needs fewer environment samples to reach comparable performance.

What carries the argument

A visual prompt is a short sequence of learnable vectors prepended to the patch embeddings of the CLIP vision encoder; the paper learns one prompt per domain factor with a contrastive loss (Equation 3) on expert demonstrations, forming a prompt pool. The ensemble module G computes, for each observation, prompted embeddings $z_i = T_\phi(o, p_i^v)$, a guidance score $g_i$ equal to the cosine similarity between the base embedding $z_0$ and $z_i$, and softmax attention weights $\omega_i$; the final state is $Z = z_0 + \sum_i \omega_i z_i$. This guided attention is the load-bearing mechanism: it lets the policy downweight prompts irrelevant to the current observation and upweight the one whose domain factor is active, which is what makes the representation adapt zero-shot.

What would settle it

Build a target domain that changes only a visual property outside the enumerated prompt pool, for example lens distortion or snow accumulation, while holding all enumerated factors at their source values; if a CONPE policy's zero-shot success drops to roughly the level of a plain CLIP baseline on that domain, then the claimed generalization is limited to the factor list and the central claim fails.

Watch

Extended reading notes

Core claim

CONPE establishes that zero-shot policy adaptation to visual domain shifts can be reduced to learning a small set of visual prompts, each encoding invariance to one named domain factor, and then letting an attention module weight those prompted embeddings for the task at hand. On the paper's own terms, the central discovery is that prompt-based contrastive learning plus cosine-guidance attention makes the CLIP embedding space reusable across unseen combinations of known visual factors, so policies trained in a few source domains keep working in target domains without adaptation. The strongest reported evidence is a 20.7 percentage point gain over EmbCLIP on unseen AI2THOR object-navigation domains, with similar gains in manipulation and driving.

Load-bearing premise

The load-bearing premise is that a short, hand-enumerated list of domain factors (camera pose, stride, illumination, weather, and a few others) together with about 10 expert episodes per factor captures every visual shift an agent will face at deployment, so that unseen domains are only unseen combinations of known factors.

Editorial extensions

If this is right

  • Zero-shot transfer: policies trained with CONPE in four source domains maintain high success in 30 seen and 10 unseen target domains, with the largest reported gap over the strongest baseline being 20.7 percentage points in AI2THOR object navigation.
  • Sample efficiency: to reach comparable zero-shot success, CONPE uses less than 60% of the timesteps EmbCLIP needs on seen target domains and 50% on unseen target domains in the same task.
  • Pretrained-policy adaptation: when only the attention module and a task prompt are updated (about 10% of the training samples), zero-shot target performance improves by 3.5 to 7.0 points in AI2THOR and by 9.0 to 57.6 points in egocentric-Metaworld.
  • Scalability: increasing the prompt pool from 2 to 10 factors raises zero-shot success by 42.8 points on seen domains and 36.7 points on unseen domains, and performance stays stable for 10 to 18 prompts.
  • Semantic regularization: adding object-level text descriptions to regularize augmented prompted embeddings improves robustness to larger augmentation noise, indicating that language information can further strengthen the ensemble.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the learned prompts are approximately orthogonal in CLIP space, the attention weights can be read as a soft estimate of which domain factors are active in the current observation, which would make CONPE usable for domain-factor identification and calibration rather than only policy execution; the paper does not claim this explicitly.
  • Because each prompt is trained from only 10 expert episodes per factor, a practical extension the paper does not test is building prompt pools from short user-collected videos instead of rule-based expert demonstrations, which could make the method applicable to real robots where hand-designed experts are unavailable.
  • The enumerated-factor assumption implies a sharp testable boundary: if deployment introduces a shift outside the prompt pool, such as lens distortion or sensor noise, zero-shot performance should degrade; the paper's own limitation statement says adaptability is constrained when domain changes extend beyond the considered factors.
  • The semantic-regularization results hint that language descriptions could eventually replace the manually enumerated domain factors, letting the prompt pool be defined by text rather than by hand; the paper lists this as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes CONPE, a framework for zero-shot policy adaptation in embodied RL that combines a frozen CLIP visual encoder with a set of contrastively learned visual prompts, each associated with a factor that affects egocentric observation (e.g., camera position, stride length, illumination). A guided-attention ensemble weights the prompted embeddings via cosine-similarity guidance, and the resulting state representation is jointly optimized with the policy. Experiments in AI2THOR navigation, egocentric-Metaworld manipulation, and CARLA driving report state-of-the-art zero-shot performance on unseen target domains, improved sample efficiency, and an extension that adapts a pretrained policy by updating only the attention module. The appendix provides implementation details, hyperparameters, dataset descriptions, and additional ablations.

Significance. If the central claim is established, the contribution is valuable: a parameter-efficient way to adapt a large vision-language model to embodied RL without fine-tuning the backbone, with a clean ensemble mechanism and evidence of interpretability. The paper is thorough in scope (three environments, multiple tasks, many ablations) and provides detailed appendices on hyperparameters and datasets. The internal ablations (Tables 3-4) convincingly show that the guided-attention ensemble outperforms uniform averaging and composition schemes. The main caveat is that the headline comparison to state-of-the-art baselines is currently confounded by unequal data access, so the claimed superiority is not yet established.

major comments (2)
  1. [Section 4, Appendix E.2] The comparison with baselines in Table 1 is confounded by unequal access to expert demonstrations. CONPE's prompt-based contrastive learning uses a dedicated expert dataset (10 episodes per domain factor; 28,464 samples for AI2THOR per Table 17; 3,840 for Metaworld; 7,394 for CARLA), while the descriptions of LUSR, CURL, ATC, ACO, and EmbCLIP in Appendix E.2-E.4 do not indicate that any of these baselines receive the same demonstrations. In particular, EmbCLIP is a frozen-CLIP baseline, so it cannot benefit from this data. The 20.7% unseen-domain gain over EmbCLIP in Table 1(a) could therefore be caused by the extra expert data rather than by the prompt ensemble mechanism. The authors should either provide the same expert demonstrations to all baselines (e.g., by fine-tuning a CLIP encoder with the same contrastive objectives on the same data and using it in EmbCLIP or as a representation for the other baselines), or include an ablation that trains the CONPE encoder on the same contrastive data without the prompt ensemble. Without such a control, the central claim of state-of-the-art zero-shot performance is not supported.
  2. [Section 4.1, Table 1(c)] The CARLA results are statistically fragile. With only three seeds and no significance tests, the reported differences often fall within the baselines' large standard deviations; for example, Map 1 seen-target performance is 1732.2±588.6 for EmbCLIP vs. 1738.0±163.5 for CONPE, and Map 2 seen-target is 2139.1±655.9 for EmbCLIP vs. 2422.5±79.6 for CONPE. The unseen-target improvements are more sizable, but the variance overlap across methods makes it unclear whether the ranking is stable. The authors should report confidence intervals, run a significance test (e.g., paired bootstrap or a nonparametric test over seeds and domains), or provide additional seeds for the CARLA experiments. This matters because the paper claims superior performance in all three environments, not only in AI2THOR.
minor comments (5)
  1. [Equation (3)] The citation key '[clr]' in the sentence 'As in [clr], for latent vectors x, y...' is undefined; this should be replaced with the intended reference (likely SimCLR, [51]).
  2. [Table 2 and Table 15] The headers contain typos: 'Souce' and 'Scoure' should be 'Source'.
  3. [Table 4] The table caption has a typo: 'Tabel' should be 'Table'.
  4. [Abstract and Section 4] The term 'zero-shot' is applied to seen target domains, but the prompt-based contrastive learning step has already consumed data from those domains; the paper should consistently distinguish zero-shot at the policy level from domain generalization to entirely unseen domains, and the abstract should be qualified accordingly.
  5. [Section 4.1 and Figure 4] The abstract reports sample-efficiency numbers for ATC ('less than 50.0% and 16.7%'), but Section 4.1 only quantifies the comparison to EmbCLIP; please clarify how the ATC numbers are derived from Figure 4 or the appendix.

Circularity Check

1 steps flagged · score 2.0 of 10

No construction-level circularity; headline unseen-domain result is held out, but prompt-count selection on the same zero-shot metric introduces mild selection circularity.

  1. fitted input called prediction [Section 4.3, Table 3 (Prompt Ensemble Scalability)]
    "Compared to the case of n = 2, for n = 10, there was a significant improvement in zero-shot performance for both seen and unseen target domains, with increases of 42.8% and 36.7%, respectively. For n ≥ 10, we observe stable performance."

    The ablation selects the number of prompts n by inspecting the unseen-target zero-shot success column of Table 3, and the same unseen-target evaluation is later reported in Table 1 as CONPE's zero-shot prediction. Choosing n=10 after seeing its score makes the reported 79.7% unseen-target success (and the 20.7% gain over EmbCLIP) a selected maximum on the evaluation set rather than an independent held-out estimate. The 'prediction' is therefore partly determined by the model-selection criterion applied to the same data, satisfying the fitted-input-called-prediction pattern.

full rationale

The central derivation is not circular. Visual prompts are contrastively learned per domain factor (Section 3.2, Eq. 3), the guided-attention ensemble and policy are jointly learned on source environments (Section 3.3, Eqs. 5-6), and the headline zero-shot numbers are evaluated on unseen target domains that, per Section 4, 'are not encountered during either learning phases.' No equation defines the target performance in terms of the fitted prompts or attention weights; at deployment the prompts, attention module, and policy are fixed and applied as a forward computation. The paper invokes no uniqueness theorem and no load-bearing self-citation; its cited prompt-tuning works (VPT, CoOp) are external and used only as implementation references. The one mild circular element is the prompt-count selection in Table 3, which uses the same unseen-target metric that later appears as the reported zero-shot result; this is a test-set selection issue rather than a construction-level equivalence and does not collapse the main comparison to a refit. The paper's own Section 6 limitation (adaptability constrained when changes extend beyond the predefined domain factors) is a scope restriction, not a circular step.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumption that a small set of hand-picked domain factors and contrastively trained prompts can span the space of visual variations. All free parameters are standard ML hyperparameters or chosen by ablation, not fitted to the target zero-shot results.

free parameters (6)
  • number of visual prompts n = 10
    Chosen based on ablation in Table 3; n=10 gives best unseen target performance.
  • visual prompt length u = 8
    Set in implementation details (Appendix E.5), not swept.
  • softmax temperature tau = not specified
    Appears in Eq. (5) for attention weights, but no value is given.
  • similarity scale lambda = not specified
    Appears in the contrastive similarity function Eq. (3), no value given.
  • timestep window k = 3
    Hyperparameter for timestep-driven contrastive learning (Appendix E.5).
  • semantic noise scale delta = 0.1 best in ablation
    Gaussian noise variance for semantic regularized augmentation, swept in Table 6.
assumptions (5)
  • domain assumption CLIP ViT-B/32 image embeddings are a suitable base representation for egocentric observations in embodied RL.
    The whole method builds on the CLIP visual encoder without fine-tuning; this is adopted from EmbCLIP and is not independently justified for the new tasks.
  • domain assumption Contrastive learning with one prompt per domain factor makes each prompt selectively invariant to that factor while preserving task-relevant information.
    This is the core premise of Section 3.2; the paper provides empirical evidence via embedding visualizations but no theoretical guarantee.
  • domain assumption Cosine similarity between base and prompted embeddings reliably indicates which domain factor is active for a given observation.
    Used to compute guidance scores in Eq. (5); the interpretability plots support it qualitatively.
  • domain assumption The predefined domain factors (camera, stride, illumination, etc.) cover the visual variation that matters for zero-shot adaptation.
    All domains are generated by combining these factors; the limitation section admits the method may fail for changes outside them.
  • domain assumption Expert demonstrations collected by rule-based policies are sufficient for contrastive prompt learning.
    Used in Section 4 and appendix; no analysis of expert suboptimality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents." pith.science (2026). https://pith.science/paper/F5NIYBDR

@misc{pith2026241211484,
  author       = {Pith},
  title        = {Pith review of: Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5NIYBDR}},
  note         = {Machine review of arXiv:2412.11484}
}
read the original abstract

For embodied reinforcement learning (RL) agents interacting with the environment, it is desirable to have rapid policy adaptation to unseen visual observations, but achieving zero-shot adaptation capability is considered as a challenging problem in the RL context. To address the problem, we present a novel contrastive prompt ensemble (ConPE) framework which utilizes a pretrained vision-language model and a set of visual prompts, thus enabling efficient policy learning and adaptation upon a wide range of environmental and physical changes encountered by embodied agents. Specifically, we devise a guided-attention-based ensemble approach with multiple visual prompts on the vision-language model to construct robust state representations. Each prompt is contrastively learned in terms of an individual domain factor that significantly affects the agent's egocentric perception and observation. For a given task, the attention-based ensemble and policy are jointly learned so that the resulting state representations not only generalize to various domains but are also optimized for learning the task. Through experiments, we show that ConPE outperforms other state-of-the-art algorithms for several embodied agent tasks including navigation in AI2THOR, manipulation in egocentric-Metaworld, and autonomous driving in CARLA, while also improving the sample efficiency of policy learning and adaptation.

Figures

Figures reproduced from arXiv: 2412.11484 by the authors.

Figure 1
Figure 1. Visual Domain Changes of Embodied Agents [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. CONPE Framework. The CLIP visual encoder is enhanced offline via (i) prompt-based contrastive learning that generates the visual prompt pool, and a policy is learned online by (ii) guided-attention-based prompt ensemble that uses the prompt pool. In (iii) zero-shot deployment, the policy is immediately evaluated upon domain changes. the policy adaptation problem as finding the optimal policy π ∗ such that π ∗ = argm… view at source ↗
Figure 3
Figure 3. Guided-Attention-based Prompt Ensemble. The cosine similarity-guided attention module [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Sample-efficiency of Prompt Ensemble-based Policy Learning for Object Navigation in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Prompt Ensemble Interpretability. In (a), the embeddings in the big circle are intra prompted [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Prompt-based Contrastive Learning with Different Contrastive Tasks Behavior-driven Contrast. Similar to [19], we exploit expert actions to obtain positive sample pairs from expert trajectories of dif￾ferent domains. With observation and ac￾tion pairs (o, a),(o ′ , a′ )…
Figure 7
Figure 7. Figure 7: Guided-Attention-based Prompt Ensemble. The cosine similarity-guided attention module [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Examples from the Source, Seen Target, and Unseen Target Domains. (a) represents the [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Intra Prompted Embeddings. Two distinct domains, represented in blue and red dots in [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Prompt Ensemble Attention Weight Matrix for Seen Domains Factors. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Sample-efficiency of Prompt Ensemble-based Policy Learning (up to 1 million timesteps). [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Sample-efficiency of Prompt Ensemble with a Pretrained Policy. The pretrained policy is [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 41 canonical work pages

  1. [1]

    Decoupling representation learning from reinforcement learning

    Adam Stooke et al. “Decoupling representation learning from reinforcement learning”. In: Proceedings of the 38th International Conference on Machine Learning. 2021, pp. 9870–9879

    work page 2021

  2. [2]

    CURL: Contrastive unsupervised repre- sentations for reinforcement learning

    Aravind Srinivas, Michael Laskin, and Pieter Abbeel. “CURL: Contrastive unsupervised repre- sentations for reinforcement learning”. In: Proceedings of the 37th International Conference on Machine Learning. 2020, pp. 5639–5650

    work page 2020

  3. [3]

    Data-efficient reinforcement learning with self-predictive representa- tions

    Max Schwarzer et al. “Data-efficient reinforcement learning with self-predictive representa- tions”. In: arXiv preprint arXiv:2007.05929 (2020)

    arXiv 2020

  4. [4]

    Unsupervised Domain Adaptation for Visual Navigation

    Shangda Li et al. “Unsupervised domain adaptation for visual navigation”. In: arXiv preprint arXiv:2010.14543 (2020). 22

    work page Pith review arXiv 2020

  5. [5]

    Zero experience required: Plug & play modular transfer learning for semantic visual navigation

    Ziad Al-Halah, Santhosh K. Ramakrishnan, and Kristen Grauman. “Zero experience required: Plug & play modular transfer learning for semantic visual navigation”. In: Proceedings of the 9th International Conference on Vision and Pattern Recognition. 2021, pp. 17010–17020

    work page 2021

  6. [6]

    Zero-shot object goal visual navigation

    Qianfan Zhao et al. “Zero-shot object goal visual navigation”. In: arXiv preprint arXiv:2206.07423 (2022)

    work page Pith review arXiv 2022

  7. [7]

    Simple but Effective: CLIP embeddings for embodied AI

    Apoorv Khandelwal et al. “Simple but Effective: CLIP embeddings for embodied AI”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 14809–14818

    work page 2022

  8. [8]

    Zson: Zero-shot object-goal navigation using multimodal goal embed- dings

    Arjun Majumdar et al. “Zson: Zero-shot object-goal navigation using multimodal goal embed- dings”. In: arXiv preprint arXiv:2206.12403 (2022)

    arXiv 2022

Show all 58 references
  1. [9]

    Ai2-thor: An interactive 3d environment for visual ai

    Eric Kolve et al. “Ai2-thor: An interactive 3d environment for visual ai”. In:arXiv preprint arXiv:1712.05474 (2017)

    2017 arXiv

  2. [10]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy et al. “CARLA: An open urban driving simulator”. In: Proceedings of the 1st Conference on Robot Learning. 2017, pp. 1–16

    2017

  3. [11]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  4. [12]

    Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining

    Qihang zhang, Zheghao Peng, and Bolei Zhou. “Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining”. In: Proceedings of the 17th European Conference on Computer Vision. 2022, pp. 111–128

    2022

  5. [13]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. “Representation learning with contrastive predictive coding”. In: arXiv preprint arXiv:1807.03748 (2018)

    2018 arXiv

  6. [14]

    Meta-World: A benchmark and evaluation for multi-task and meta reinforce- ment learning

    Tianhe Yu et al. “Meta-World: A benchmark and evaluation for multi-task and meta reinforce- ment learning”. In: Proceedings of the 3rd Conference on Robot Learning. 2019, pp. 1094– 1100

    2019

  7. [15]

    Domain adaptation in reinforcement learning via latent unified state representation

    Jinwei Xing et al. “Domain adaptation in reinforcement learning via latent unified state representation”. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, pp. 10452–10459

    2021

  8. [16]

    Visual prompt tuning

    Menglin Jia et al. “Visual prompt tuning”. In: Proceedings of the 17th European Conference on Computer Vision. 2022, pp. 709–727

    2022

  9. [17]

    Learning to prompt for vision-language models

    Kaiyang Zhou et al. “Learning to prompt for vision-language models”. In: International Journal of Computer Vision (2022)

    2022

  10. [18]

    Integrating contrastive learning with dynamic models for reinforcement learning from images

    Bang You et al. “Integrating contrastive learning with dynamic models for reinforcement learning from images”. In: Neurocomputing (2022)

    2022

  11. [19]

    Action-driven contrastive representation for reinforcement learning

    Minbeom Kim et al. “Action-driven contrastive representation for reinforcement learning”. In: PLOS ONE (2022)

    2022

  12. [20]

    Contrastive behavioral similarity embeddings for generalization in reinforcement learning

    Rishabh Agarwal et al. “Contrastive behavioral similarity embeddings for generalization in reinforcement learning”. In: arXiv preprint arXiv:2101.05265 (2021)

    2021 arXiv

  13. [21]

    STACoRe: Spatio-temporal and action-based contrastive representations for reinforcement learning in Atari

    Young Jae Lee et al. “STACoRe: Spatio-temporal and action-based contrastive representations for reinforcement learning in Atari”. In: Neural Networks (2023)

    2023

  14. [22]

    Proximal policy optimization algorithms

    John Schulman et al. “Proximal policy optimization algorithms”. In: arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  15. [23]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. “A reduction of imitation learning and structured prediction to no-regret online learning”. In: Proceedings of the 14th Interna- tional Conference on Artificial Intelligence and Statistics. 2011, pp. 627–635

    2011

  16. [24]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu et al. “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing”. In: ACM Computing Surveys (2023)

    2023

  17. [25]

    Effective approaches to attention-based neural machine translation

    Minh-Thang Luong, Hieu Pham, and Christopher D Manning. “Effective approaches to attention-based neural machine translation”. In: arXiv preprint arXiv:1508.04025 (2015)

    2015 arXiv

  18. [26]

    Attention is all you need

    Ashish Vaswani et al. “Attention is all you need”. In:Advances in neural information process- ing systems (2017)

    2017

  19. [27]

    Image Augmentation Is All You Need: Regu- larizing deep reinforcement learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. “Image Augmentation Is All You Need: Regu- larizing deep reinforcement learning from pixels”. In: Proceedings of the 9th International Conference on Learning Representations. 2021

    2021

  20. [28]

    S4RL: Surprisingly Simple Self- Supervision for Offline Reinforcement Learning in Robotics

    Samarth Sinha, Ajay Mandlekar, and Animesh Garg. “S4RL: Surprisingly Simple Self- Supervision for Offline Reinforcement Learning in Robotics”. In: Proceedings of the 5th Conference on Robotics Learning. 2021, pp. 907–917. 23

    2021

  21. [29]

    Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

    Arjun Majumdar et al. “Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?” In: arXiv preprint arXiv:2303.18240 (2023)

    2023 arXiv

  22. [30]

    R3m: A universal visual representation for robot manipulation

    Suraj Nair et al. “R3m: A universal visual representation for robot manipulation”. In: arXiv preprint arXiv:2203.12601 (2022)

    2022 arXiv

  23. [31]

    Open-world object manipulation using pre-trained vision-language mod- els

    Austin Stone et al. “Open-world object manipulation using pre-trained vision-language mod- els”. In: arXiv preprint arXiv:2303.00905 (2023)

    2023 arXiv

  24. [32]

    Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots

    Jiange Yang et al. “Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots”. In: arXiv preprint arXiv:2306.05716 (2023)

    2023 arXiv

  25. [33]

    Gnm: A general navigation model to drive any robot

    Dhruv Shah et al. “Gnm: A general navigation model to drive any robot”. In: Proceedings of the IEEE International Conference on Robotics and Automation. IEEE. 2023, pp. 7226–7233

    2023

  26. [34]

    ExAug: Robot-conditioned navigation policies via geometric experi- ence augmentation

    Noriaki Hirose et al. “ExAug: Robot-conditioned navigation policies via geometric experi- ence augmentation”. In: Proceedings of the IEEE International Conference on Robotics and Automation. IEEE. 2023, pp. 4077–4084

    2023

  27. [35]

    Speaker-follower models for vision-and-language navigation

    Daniel Fried et al. “Speaker-follower models for vision-and-language navigation”. In: Pro- ceedings of the 31th Conference on Neural Information Processing Systems. 2018, pp. 3318– 3329

    2018

  28. [36]

    Take the scenic route: Improving generalization in vision-and-language navigation

    Felix Yu et al. “Take the scenic route: Improving generalization in vision-and-language navigation”. In: arXiv preprint arXiv:2003.14269 (2020)

    2020 arXiv

  29. [37]

    Dialfred: Dialogue-enabled agents for embodied instruction following

    Xiaofeng Gao et al. “Dialfred: Dialogue-enabled agents for embodied instruction following”. In: arXiv preprint arXiv:2202.13330 (2022)

    2022 arXiv

  30. [38]

    Vision-language navigation with random environmental mixup

    Chong Liu et al. “Vision-language navigation with random environmental mixup”. In: Pro- ceedings of the International Conference on Computer Vision. 2021, pp. 1624–1634

    2021

  31. [39]

    EnvEdit: Environment Editing for Vision-and-Language Navigation

    Jialu Li, Hao Tan, and Mohit Bansal. “EnvEdit: Environment Editing for Vision-and-Language Navigation”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 15386–15396

    2022

  32. [40]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation

    Xin Wang et al. “Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 6629–6638

    2019

  33. [41]

    Explore and explain: self-supervised navigation and recounting

    Roberto Bigazzi et al. “Explore and explain: self-supervised navigation and recounting”. In: Proceeding of the 25th International Conference on Pattern Recognition. 2020, pp. 1152–1159

    2020

  34. [42]

    MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agents

    Eun Sun Lee et al. “MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agents”. In: Proceeding of the 17th European Conference on Computer Vision. 2022, pp. 338–354

    2022

  35. [43]

    CLIP-Nav: Using CLIP for zero-shot vision-and-language navigation

    Vishnu Sashank Dorbala et al. “CLIP-Nav: Using CLIP for zero-shot vision-and-language navigation”. In: arXiv preprint arXiv:2211.16649 (2022)

    2022 arXiv

  36. [44]

    CLIP on Wheels: zero-shot object navigation as object localization and exploration

    Samir Yitzhak Gadre et al. “CLIP on Wheels: zero-shot object navigation as object localization and exploration”. In: arXiv preprint arXiv:2203.10421 (2022)

    2022 arXiv

  37. [45]

    Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action

    Dhruv Shah et al. “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action”. In: Proceedings of the 6th Conference on Robot Learning. 2022, pp. 492– 504

    2022

  38. [46]

    DARLA: Improving zero-shot transfer in reinforcement learning

    Irina Higgins et al. “DARLA: Improving zero-shot transfer in reinforcement learning”. In: Proceedings of the 34th International Conference on Machine Learning. 2018, pp. 1480–1490

    2018

  39. [47]

    Exploring visual prompts for adapting large-scale models

    Hyojin Bahng et al. “Exploring visual prompts for adapting large-scale models”. In: arXiv preprint arXiv:2203.17274 (2022)

    2022 arXiv

  40. [48]

    MaPLe: Multi-modal prompt learning

    Muhammad Uzair Khattak et al. “MaPLe: Multi-modal prompt learning”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023

    2023

  41. [49]

    Unified vision and language prompt learning

    Yuhang Zang et al. “Unified vision and language prompt learning.” In: arXiv preprint arXiv:2210.07225 (2022)

    2022 arXiv

  42. [50]

    Algorithm AS 136: A k-means clustering algorithm

    John A Hartigan and Manchek A Wong. “Algorithm AS 136: A k-means clustering algorithm”. In: Journal of the royal statistical society. series c (applied statistics)(1979)

    1979

  43. [51]

    A simple framework for contrastive learning of visual representations

    Ting Chen et al. “A simple framework for contrastive learning of visual representations”. In: Proceedings of the 37th International Conference on Machine Learning. 2020, pp. 1597–1607

    2020

  44. [52]

    Generalization in reinforcement learning by soft data augmentation

    Nicklas Hansen and Xiaolong Wang. “Generalization in reinforcement learning by soft data augmentation”. In: Proceedings of the 38th International Conference on Robotics and Automa- tion. IEEE. 2021, pp. 13611–13617. 24

    2021

  45. [53]

    Data-driven color augmentation techniques for deep skin image analy- sis

    Adrian Galdran et al. “Data-driven color augmentation techniques for deep skin image analy- sis”. In: arXiv preprint arXiv:1703.03702 (2017)

    2017 arXiv

  46. [54]

    Robustnav: Towards benchmarking robustness in embodied navigation

    Prithvijit Chattopadhyay et al. “Robustnav: Towards benchmarking robustness in embodied navigation”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 15691–15700

    2021

  47. [55]

    Efficient adaptation for end-to-end vision-based robotic manipulation

    Ryan Julian et al. “Efficient adaptation for end-to-end vision-based robotic manipulation”. In: Proceedings of the 4th Lifelong Machine Learning Workshop at ICML. 2020

    2020

  48. [56]

    On evaluation of embodied navigation agents

    Peter Anderson et al. “On evaluation of embodied navigation agents”. In: arXiv preprint arXiv:1807.06757 (2018)

    2018 arXiv

  49. [57]

    Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts

    Akari Asai et al. “Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts”. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2022, pp. 6655–6672

    2022

  50. [58]

    Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning

    Xiangyu Peng et al. “Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning”. In: arXiv preprint arXiv:2210.12587 (2022). 25 Table 17: AI2THOR Expert Dataset ID Observation Environmental Difference Physical Property Expe...

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.