Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examples

Byeonghwi Kim; Jonghyun Choi; Taewoong Kim

REVIEW 3 major objections 5 minor 56 references

Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examples

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An embodied agent can plan household tasks from about 100 language–demonstration pairs when the LLM planner is grounded in visual observations and missing objects are replaced by similar ones in the scene.

desk verdict Useful few-shot ALFRED system with released code, but the headline gain over LLM-Planner is mostly from the base system, not the two proposed components. read the letter →

arxiv 2412.17288 v1 pith:HQM47SVQ submitted 2024-12-23 cs.RO cs.AI

Taewoong Kim , Byeonghwi Kim , Jonghyun Choi This is my paper · ORCID

classification cs.ROcs.AI

keywords embodiedinstructionfollowingfew-shotlearninglargelanguagemodelstaskplanningvisualgroundingreplanningALFREDbenchmarkmulti-modalretrieval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a few-shot embodied agent can plan long-horizon household tasks from about 100 language-and-demonstration pairs if the LLM's in-context examples are selected using both the instruction text and the agent's initial visual view, and if a lightweight replanner later substitutes missing objects with semantically similar ones seen in the scene. On the ALFRED benchmark, the resulting system, FLARE, reports the highest few-shot success rates among the compared methods, with up to +24.46 absolute percentage points over the previous best on the test-unseen split when using GPT-4. The authors argue this matters because collecting free-form language annotations is costly, and prior LLM planners rely mostly on linguistic common sense and ignore the environment's current state, producing plans that send the agent searching for objects that are not there.

What carries the argument

Key machinery: a multi-modal similarity retrieval that scores each candidate demonstration by the normalized sum of cosine similarities between (a) the current instruction and the demonstration's instruction, both embedded with BERT, and (b) the agent's initial panoramic view and the demonstration's view, both embedded with CLIP; the top-k demonstrations are then represented as compact (Action, Object, Receptacle) triplets for the LLM prompt. This is paired with an environment adaptive replanning rule that, upon failure to find the target object, replaces it with the detected object maximizing cosine similarity of text embeddings. The two equations are Eq. (1) for retrieval and Eq. (3) for replacement; together they convert a purely linguistic plan into one tied to objects actually present.

What would settle it

Run LLM-Planner with the same low-level action policy, same prompt format, and same LLM version as FLARE on ALFRED test unseen. If the difference in success rate drops below the reported gap, the claimed advantages of multi-modal grounding and visual replanning are not supported.

Watch

Extended reading notes

Core claim

FLARE's central claim is that grounding matters at both ends of planning: before the LLM writes a plan, and after it fails. Before, the Multi-Modal Planner retrieves the top-k training demonstrations by a normalized sum of language and visual similarity, embedding the instruction with BERT and the agent's surrounding views with CLIP, then formats them as (Action, Object, Receptacle) triplets in the prompt. After, the Environment Adaptive Replanning module tracks a list of detected objects and, when the agent cannot find the target object, replaces it with the observed object whose text embedding is maximally similar to the missing one—without another LLM call. On ALFRED with 100 examples, the full system reaches 40.88% success (and 51.72% goal-condition) on the test-unseen split with GPT-4 and sequential instructions, versus 16.42% success reported for LLM-Planner; isolating the planner, FLARE without EAR attains 67.48% unseen plan accuracy versus 30.12% for LLM-Planner static with GPT-4. Ablations with GPT-3.5 show each component contributes, and a version with neither still reaches 28.58% unseen success, so the margin over LLM-Planner is not solely due to the proposed components.

Load-bearing premise

The headline comparison treats the published LLM-Planner numbers as directly comparable to FLARE's, but the two systems differ in low-level action policy, plan representation, prompt structure, and possibly LLM version, and the paper's own ablation shows most of the headline gap comes from the base system rather than the two proposed components.

Editorial extensions

If this is right

With 100 training pairs, an agent can reach 40.88% success and 51.72% goal-condition success on ALFRED test-unseen when using GPT-4 with step-by-step instructions, compared with 16.42% for the cited LLM-Planner baseline.
The replanning step is an LLM-free lookup, so correcting a wrong subgoal costs one text-embedding computation rather than another full LLM generation.
The (Action, Object, Receptacle) representation reduces prompt token usage by about 25% relative to LLM-Planner's format, leaving more room for in-context examples.
Using a better backbone LLM produces consistent gains across all four tested models (LLaMA2, Vicuna, GPT-3.5, GPT-4), so the method is not tied to a single model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not isolate how much of the headline margin comes from multi-modal retrieval versus the base system's different prompt format and low-level policy; a clean test would compare language-only retrieval and multi-modal retrieval under the same LLM and action policy.
EAR's object replacement is an unconstrained lexical lookup; adding a similarity threshold or an action-compatibility check (e.g., a Put target must be a receptacle) would likely reduce false replacements, which the paper does not evaluate.
The same EAR mechanism could be reused for other vocabulary mismatch problems, such as zero-shot navigation with novel object names, by treating any absent target as a candidate for semantic substitution.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

FLARE is a few-shot LLM planner for ALFRED with two new pieces: MMP (multimodal retrieval for in-context examples using egocentric views plus language) and EAR (LLM-free replanning that replaces an absent object with the most semantically similar detected object). The empirical work is genuine: FLARE with GPT-4 gets 40.88% unseen SR, up from the cited LLM-Planner 16.42%, and the code is public.

What the paper does well: the ablations are internally consistent and show each component helps; the qualitative examples are illustrative; the choice to do replanning without calling the LLM is sensible for cost; and the effort to ground retrieval in visual state is a reasonable extension beyond LLM-Planner. The planner-accuracy comparison across four LLMs is a nice check that the gains are not tied to one model.

The biggest soft spot is that the paper's central claim—that the SOTA gain comes from the proposed components—is not actually supported by its own Table 3. FLARE without either MMP or EAR already reaches 28.58% unseen SR, about 12 points above the cited LLM-Planner. Adding both components yields 31.79%, so the components together contribute roughly 3.2 points. That means the advertised +24.46% gap is mostly explained by the base system: the deterministic FMM navigation, the triplet subgoal representation, the prompt structure, and the language-only retrieval. The paper never re-runs LLM-Planner with the same low-level controller and format, so the Table 1 comparison is confounded. This is fixable—re-run the baseline under matched conditions or at least re-frame the contribution as the full system—but as written the attribution is overstated.

Minor issues: tau (the EAR uncertainty threshold) is never reported even though it is a free parameter in Algorithm 1; no error bars appear anywhere, which matters for a 1.5–2 point ablation difference; and the plan-accuracy metric is exact match, which is sensitive to output format when the two systems use different formats.

Should we take the result seriously? Yes. The direction is sound, the code is out, and the full-system improvement is large enough that even if the components only add a few points, the system is a strong few-shot baseline. With the attribution problem addressed, it would be a useful contribution.

Recommendation: send it to peer review, but the reviewers should press on baseline comparability and the missing hyperparameters.

Referee Report

3 major / 5 minor

Summary. The paper presents FLARE, an embodied instruction-following agent for the ALFRED benchmark in a few-shot (0.5%) setting. It contributes a Multi-Modal Planner (MMP) that retrieves k in-context demonstrations using a weighted combination of language and visual environment similarity (Eq. 1), and an Environment Adaptive Replanning (EAR) module that replaces an undetected target object with the most semantically similar detected object using Eq. (3). The system is evaluated against LLM-Planner and other baselines, reporting a test-unseen success rate of 40.88% with GPT-4 versus LLM-Planner's cited 16.42%, with additional ablations, planner-accuracy comparisons, and a tabletop robotic application.

Significance. The work addresses a real and timely problem: grounding LLM-based planning in observed environment state while using very few annotated demonstrations. Strengths include a clearly described and modular architecture, publicly released code, evaluation across four different LLMs, a component-level ablation, and a qualitative robotic demonstration. If the component-level results are considered in isolation, MMP and EAR each give a consistent but modest improvement over the base system, and the static-planner accuracy gains in Table 2 are large. The principal weakness is that the headline state-of-the-art comparison is not cleanly attributable to the proposed components, because the base system without either component already exceeds the cited LLM-Planner baseline by a wide margin.

major comments (3)

[Table 1 vs. Table 3] The headline claim in the Introduction of an up to +24.46% absolute gain over LLM-Planner is not attributable to the proposed components. Table 3(d) shows that with GPT-3.5, the base system without MMP or EAR already reaches 28.58% unseen success rate, which is +12.16 points over the cited LLM-Planner value of 16.42% in Table 1, while adding MMP and EAR together (row a) yields 31.79%, an additional +3.21 points. The paper should separate the gain due to the base system from the gain due to MMP and EAR, or re-run LLM-Planner under matched conditions, before claiming that visual grounding and replanning produce the reported margin.
[Table 1 and Table 2] The comparison with LLM-Planner is not controlled. The published LLM-Planner numbers are excerpted rather than re-run with the same low-level action policy, subgoal representation, prompt format, and LLM version; the paper itself uses deterministic FMM navigation, a different triplet representation (Eq. (2)), and a different prompt structure (Fig. 8). This also affects Table 2, where plan accuracy is judged by exact match to ground-truth plans and the two systems use different plan formats. Please provide a matched re-run or explicitly qualify the comparison as a system-level benchmark difference rather than evidence about the proposed components.
[Section 3.2 and title] The title and contributions describe 'efficient replanning,' and Section 3.2 contrasts EAR with LLM-based replanning, but the paper reports no runtime, token cost, or LLM query-count measurements. Without such measurements, the computational-efficiency claim is not empirically supported. Please add at least an LLM-call count or token-cost comparison with and without EAR, or revise the efficiency claim accordingly.

minor comments (5)

[Table 1 caption] The word 'path-length-weighteed' should be 'path-length-weighted.'
[Section 4.3] The sentence 'Despite with a relatively less capable language models' is ungrammatical; consider 'Despite using a relatively less capable language model...'
[Section 4.5 and Figure 6 caption] The phrase 'unrounded plan' should be 'ungrounded plan.'
[Supplementary Figure 9] In the supplementary text for Figure 9, 'CoffeMaker' is a typo for 'CoffeeMaker.'
[Section 4.1] The values of w_l and w_e in Eq. (1) are said to be equal, but the actual values are not reported; please state them for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: FLARE's central claims are validated on the external ALFRED benchmark, and no proposed component is defined in terms of the reported result.

full rationale

No step in FLARE's derivation reduces to its own inputs. The Multi-Modal Planner similarity in Eq. (1) is a normalized weighted sum of cosine similarities with equal hand-set weights (wl and we), and k is fixed at 9 following the prior work; none of these values is fitted to the reported success rates. The Environment Adaptive Replanning rule in Eq. (3) is an argmax over cosine similarities of frozen language embeddings, which is an algorithmic replacement rule rather than a fitted prediction. The main results are measured against the external ALFRED benchmark, and the cited LLM-Planner numbers are published empirical results used as baselines, not premises that logically force FLARE's outputs. The paper's self-citations (e.g., CAPEAM, ReALFRED, prior perception/policy work) are baseline comparisons or background context, and none carries a load-bearing uniqueness or ansatz claim. The ablation result in Table 3(d), showing the base system already exceeds the cited LLM-Planner unseen SR, raises a legitimate question about how much of the headline gain is attributable to the proposed components, but that is an experimental-comparability and attribution issue, not circularity: the base-system measurement is itself an independent empirical result, and the paper never defines FLARE's success in terms of LLM-Planner's number. Accordingly, the circularity burden is minimal and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on benchmark assumptions and on hand-set components: the retrieval weights and the EAR uncertainty threshold (unreported). No free parameters are fitted to the test success rate, so circularity is low, but the undisclosed tau and the lack of sensitivity analysis weaken the empirical grounding.

free parameters (3)

Multi-modal similarity weights w_l, w_e = Set equal (exact value not reported, likely 0.5 each)
In Eq. (1) the language and environment similarities are weighted equally by hand; no sensitivity analysis is provided.
Number of in-context examples k = 9
Adopted from LLM-Planner (Song et al. 2023) for fairness; not tuned for FLARE.
EAR uncertainty threshold tau = Not reported
Algorithm 1 triggers replanning only when uncertainty exceeds tau, but tau is never specified in the paper; it directly changes how often EAR acts.

assumptions (4)

domain assumption ALFRED is a valid proxy for embodied instruction-following competence.
The entire evaluation rests on this benchmark's success and goal-condition metrics.
domain assumption Cosine similarity in text embedding space is a reliable proxy for physical substitutability of objects.
EAR replaces missing targets with the most similar detected object name (Eq. 3); wrong replacements would break tasks.
domain assumption The agent's egocentric view at command reception contains enough scene information to improve retrieval.
MMP conditions retrieval on surrounding views; the ablation shows a small gain, so this is an empirical assumption.
domain assumption Published LLM-Planner results are comparable to the authors' runs.
Table 1 excerpts SR and GC from Song et al. 2023 without re-running under matched conditions; if baseline conditions differ, the comparison is invalid.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examples." pith.science (2026). https://pith.science/paper/HQM47SVQ

@misc{pith2026241217288,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HQM47SVQ}},
  note         = {Machine review of arXiv:2412.17288}
}

read the original abstract

Learning a perception and reasoning module for robotic assistants to plan steps to perform complex tasks based on natural language instructions often requires large free-form language annotations, especially for short high-level instructions. To reduce the cost of annotation, large language models (LLMs) are used as a planner with few data. However, when elaborating the steps, even the state-of-the-art planner that uses LLMs mostly relies on linguistic common sense, often neglecting the status of the environment at command reception, resulting in inappropriate plans. To generate plans grounded in the environment, we propose FLARE (Few-shot Language with environmental Adaptive Replanning Embodied agent), which improves task planning using both language command and environmental perception. As language instructions often contain ambiguities or incorrect expressions, we additionally propose to correct the mistakes using visual cues from the agent. The proposed scheme allows us to use a few language pairs thanks to the visual cues and outperforms state-of-the-art approaches. Our code is available at https://github.com/snumprlab/flare.

Figures

Figures reproduced from arXiv: 2412.17288 by the authors.

**Figure 1.** Overview of the proposed FLARE. Our agent consists of (1) ‘Multi-Modal Planner (MMP)’ and (2) ‘Environment Adaptive Replanning (EAR)’. MMP takes into account both the agent’s initial surrounding views and received instructions to generate a sequence of subgoals by prompting an LLM (e.g., GPT-4). When the agent gets stuck while executing a plan, EAR adjusts the ungrounded plan to a physically grounded one with visu… view at source ↗

**Figure 2.** Detailed architecture of FLARE. It comprises ‘Multi-Modal Planner (MMP)’ and ‘Environment Adaptive Replanning (EAR)’. 1 MMP retrieves the top k relevant training data pairs with instruction and expert demonstration (indicated with Expert Demon.), based on the agent’s initial panoramic surrounding views and language instructions, then plans a sequence of actions through LLMs (e.g., GPT-4) with these examples. 2 When … view at source ↗

**Figure 3.** Multi-Modal Planner. MMP selects top k expert demonstrations based on ‘multi-modal similarity’ (Eq. (1)) and then converts them into subgoal triplets (An, On, Rn). MMP uses subgoal triplets, along with a text prompt, to guide an LLM in generating task-specific subgoal sequences from natural language instructions. a sofa would fail to recognize a couch. This may result in plans that are not well grounded in environ… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Environment Adaptive Replanning. EAR corrects a plan by listing detected objects and calculating semantic similarities to replace inaccurately referenced items (e.g., TrashCan). This ensures that the plan is grounded in the environment. visual information) when genera…

**Figure 5.** Figure 5: Benefits of proposed multi-modal planner (MMP). An agent without MMP misinterprets the task, simply placing a SoapBar in the SinkBasin. In contrast, an agent with MMP seems to comprehend an objective of cleaning, generating a plausible plan and subsequently completing…

**Figure 7.** Figure 7: An example of robotic task applications. Baseline model (Zeng et al. 2023) generates an ungrounded plan due to ambiguous instructions (e.g., tool). In contrast, FLARE generates a grounded plan and successfully exectues actions. initially and notices that StatueStep ma…

**Figure 9.** Figure 9: A qualitative example of our agent with and without ‘Environmental Adaptive Replanning’ (EAR). FLARE w/o EAR fails to locate CoffeMaker, leading to task failure. In contrast, FLARE requests replanning to EAR and it replaces partially incorrect subset of the subgoal (…

**Figure 10.** Figure 10: Another qualitative example of our agent with and without ‘Environmental Adaptive Replanning’ (EAR). FLARE w/o EAR fails to find a Cup and achieves zero goal conditions (GC). Conversely, FLARE requests replanning to EAR and it replaces a partially incorrect subset of…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 45 canonical work pages

[1]

Ahn, M.; Brohan, A.; Brown, N.; Chebotar, Y.; Cortes, O.; David, B.; Finn, C.; Fu, C.; Gopalakrishnan, K.; Hausman, K.; et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. In CoRL

work page 2022
[2]

Anderson, P.; Wu, Q.; Teney, D.; Bruce, J.; Johnson, M.; S \"u nderhauf, N.; Reid, I.; Gould, S.; and van den Hengel, A. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR

work page 2018
[3]

Bhambri, S.; Kim, B.; and Choi, J. 2023. Multi-level Compositional Reasoning for Interactive Instruction Following. In AAAI

work page 2023
[4]

Blukis, V.; Paxton, C.; Fox, D.; Garg, A.; and Artzi, Y. 2021. A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution. In CoRL

work page 2021
[5]

Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.;...

2020
[6]

Chang, A.; Dai, A.; Funkhouser, T.; Halber, M.; Niessner, M.; Savva, M.; Song, S.; Zeng, A.; and Zhang, Y. 2017. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV

work page 2017
[7]

S.; Sathyendra, K

Chaplot, D. S.; Sathyendra, K. M.; Pasumarthi, R. K.; Rajagopal, D.; and Salakhutdinov, R. 2017. Gated-attention architectures for task-oriented language grounding. In AAAI

work page 2017
[8]

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374

arXiv 2021

Show all 56 references

[9]

Das, A.; Datta, S.; Gkioxari, G.; Lee, S.; Parikh, D.; and Batra, D. 2018. Embodied Question Answering. In CVPR

2018
[10]

Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805

2018 arXiv
[11]

Driess, D.; Xia, F.; Sajjadi, M. S. M.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; Huang, W.; Chebotar, Y.; Sermanet, P.; Duckworth, D.; Levine, S.; Vanhoucke, V.; Hausman, K.; Toussaint, M.; Greff, K.; Zeng, A.; Mordatch, I.; and Florence...

2023
[12]

P.; Kim, Y.; Han, W.; Herrasti, A.; et al

Ehsani, K.; Gupta, T.; Hendrix, R.; Salvador, J.; Weihs, L.; Zeng, K.-H.; Singh, K. P.; Kim, Y.; Han, W.; Herrasti, A.; et al. 2024. SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World. In CVPR

2024
[13]

Fan, L.; Wang, G.; Jiang, Y.; Mandlekar, A.; Yang, Y.; Zhu, H.; Tang, A.; Huang, D.-A.; Zhu, Y.; and Anandkumar, A. 2022. MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. In NeurIPS Datasets and Benchmarks Track

2022
[14]

Fang, K.; Liu, F.; Abbeel, P.; and Levine, S. 2024. MOKA: Open-Vocabulary Robotic Manipulation through Mark-Based Visual Prompting. In RSS

2024
[15]

J.; Aydin, A.; Anvari, M.; Chakravarthy, A

Ge, Y.; Tang, Y.; Xu, J.; Gokmen, C.; Li, C.; Ai, W.; Martinez, B. J.; Aydin, A.; Anvari, M.; Chakravarthy, A. K.; Yu, H.-X.; Wong, J.; Srivastava, S.; Lee, S.; Zha, S.; Itti, L.; Li, Y.; Martin-Martin, R.; Liu, M.; Zhang, P.; Zhang, R.; Fei-Fei, L.; and Wu, J. 2024. BEHAVIOR ...

2024
[16]

Gordon, D.; Kembhavi, A.; Rastegari, M.; Redmon, J.; Fox, D.; and Farhadi, A. 2018. Iqa: Visual question answering in interactive environments. In CVPR

2018
[17]

Huang, W.; Abbeel, P.; Pathak, D.; and Mordatch, I. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML

2022
[18]

Huang, W.; Xia, F.; Xiao, T.; Chan, H.; Liang, J.; Florence, P.; Zeng, A.; Tompson, J.; Mordatch, I.; Chebotar, Y.; Sermanet, P.; Jackson, T.; Brown, N.; Luu, L.; Levine, S.; Hausman, K.; and Ichter, B. 2023. Inner Monologue: Embodied Reasoning through Planning with Language M...

2023
[19]

Inoue, Y.; and Ohashi, H. 2022. Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Following. arXiv:2211.03267

2022 arXiv
[20]

Jiang, Y.; Gupta, A.; Zhang, Z.; Wang, G.; Dou, Y.; Chen, Y.; Fei-Fei, L.; Anandkumar, A.; Zhu, Y.; and Fan, L. 2023. VIMA: General Robot Manipulation with Multimodal Prompts. In ICML

2023
[21]

Kim, B.; Kim, J.; Kim, Y.; Min, C.; and Choi, J. 2023. Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agents. In ICCV

2023
[22]

Kim, T.; Min, C.; Kim, B.; Kim, J.; Jeung, W.; and Choi, J. 2024. ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environment. In ECCV

2024
[23]

Kolve, E.; Mottaghi, R.; Han, W.; VanderBilt, E.; Weihs, L.; Herrasti, A.; Gordon, D.; Zhu, Y.; Gupta, A.; and Farhadi, A. 2017. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv:1712.05474

2017 arXiv
[24]

Liang, J.; Huang, W.; Xia, F.; Xu, P.; Hausman, K.; Ichter, B.; Florence, P.; and Zeng, A. 2023. Code as policies: Language model programs for embodied control. In ICRA

2023
[25]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. In NeurIPS

2023
[26]

Majumdar, A.; Ajay, A.; Zhang, X.; Putta, P.; Yenamandra, S.; Henaff, M.; Silwal, S.; Mcvay, P.; Maksymets, O.; Arnaud, S.; et al. 2024. Openeqa: Embodied question answering in the era of foundation models. In CVPR

2024
[27]

Y.; Chaplot, D

Min, S. Y.; Chaplot, D. S.; Ravikumar, P.; Bisk, Y.; and Salakhutdinov, R. 2022. FILM: Following Instructions in Language with Modular Methods. In ICLR

2022
[28]

Murray, M.; and Cakmak, M. 2022. Following natural language instructions for household tasks with landmark guided search and reinforced pose adjustment. RA-L

2022
[29]

Nguyen, V.-Q.; Suganuma, M.; Okatani; and Takayuki. 2021. Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Tasks. In IJCAI

2021
[30]

Pashevich, A.; Schmid, C.; Sun; and Chen. 2021. Episodic Transformer for Vision-and-Language Navigation. In ICCV

2021
[31]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning transferable visual models from natural language supervision. In ICML

2021
[32]

Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR

2020
[33]

S.; Seo, J.; Kim, T.; Choi, J.; and Horowitz, R

Ryu, H.; Kim, J.; Chang, J.; Ahn, H. S.; Seo, J.; Kim, T.; Choi, J.; and Horowitz, R. 2024. Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation. In CVPR

2024
[34]

Sarch, G.; Wu, Y.; Tarr, M.; and Fragkiadaki, K. 2023. Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models. In EMNLP

2023
[35]

Sethian, J. A. 1996. A fast marching level set method for monotonically advancing fronts. In PNAS

1996
[36]

Shridhar, M.; Thomason, J.; Gordon, D.; Bisk, Y.; Han, W.; Mottaghi, R.; Zettlemoyer, L.; and Fox, D. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR

2020
[37]

Singh, I.; Blukis, V.; Mousavian, A.; Goyal, A.; Xu, D.; Tremblay, J.; Fox, D.; Thomason, J.; and Garg, A. 2023. Progprompt: Generating situated robot task plans using large language models. In ICRA

2023
[38]

P.; Bhambri, S.; Kim, B.; Mottaghi, R.; and Choi, J

Singh, K. P.; Bhambri, S.; Kim, B.; Mottaghi, R.; and Choi, J. 2021. Factorizing Perception and Policy for Interactive Instruction Following. In ICCV

2021
[39]

H.; Kil, J.; Pan, T.-Y.; Sadler, B

Song, C. H.; Kil, J.; Pan, T.-Y.; Sadler, B. M.; Chao, W.-L.; and Su, Y. 2022. One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones. In CVPR

2022
[40]

H.; Wu, J.; Washington, C.; Sadler, B

Song, C. H.; Wu, J.; Washington, C.; Sadler, B. M.; Chao, W.-L.; and Su, Y. 2023. LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models. In ICCV

2023
[41]

D.; and Toshev, A

Szot, A.; Schwarzer, M.; Agrawal, H.; Mazoure, B.; Metcalf, R.; Talbott, W.; Mackraz, N.; Hjelm, R. D.; and Toshev, A. T. 2024. Large Language Models as Generalizable Policies for Embodied Tasks. In ICLR

2024
[42]

Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288

2023 arXiv
[43]

Uppal, S.; Agarwal, A.; Xiong, H.; Shaw, K.; and Pathak, D. 2024. SPIN: Simultaneous Perception, Interaction and Navigation. In CVPR

2024
[44]

Wang, G.; Xie, Y.; Jiang, Y.; Mandlekar, A.; Xiao, C.; Zhu, Y.; Fan, L.; and Anandkumar, A. 2024 a . Voyager: An Open-Ended Embodied Agent with Large Language Models. TMLR

2024
[45]

Wang, Y.-J.; Zhang, B.; Chen, J.; and Sreenath, K. 2024 b . Prompt a robot to walk with large language models. In CDC

2024
[46]

Wu, J.; Antonova, R.; Kan, A.; Lepert, M.; Zeng, A.; Song, S.; Bohg, J.; Rusinkiewicz, S.; and Funkhouser, T. 2023. Tidybot: Personalized robot assistance with large language models. In IROS

2023
[47]

R.; He, Z.; Sax, A.; Malik, J.; and Savarese, S

Xia, F.; Zamir, A. R.; He, Z.; Sax, A.; Malik, J.; and Savarese, S. 2018. Gibson env: Real-world perception for embodied agents. In CVPR

2018
[48]

Xu, X.; Luo, S.; Yang, Y.; Li, Y.-L.; and Lu, C. 2024. DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Control. arXiv:2407.14758

2024 arXiv
[49]

Yang, Y.; Zhou, T.; Li, K.; Tao, D.; Li, L.; Shen, L.; He, X.; Jiang, J.; and Shi, Y. 2024. Embodied multi-modal agent trained by an llm from a parallel textworld. In CVPR

2024
[50]

Zeng, A.; Attarian, M.; Ichter, B.; Choromanski, K.; Wong, A.; Welker, S.; Tombari, F.; Purohit, A.; Ryoo, M.; Sindhwani, V.; et al. 2023. Socratic models: Composing zero-shot multimodal reasoning with language. In ICLR

2023
[51]

V.; et al

Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022. Opt: Open pre-trained transformer language models. arXiv:2205.01068

2022 arXiv
[52]

Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS Datasets and Benchmarks Track

2023
[53]

Zheng, S.; jiazheng liu; Feng, Y.; and Lu, Z. 2024. Steve-Eye: Equipping LLM -based Embodied Agents with Visual Perception in Open Worlds. In ICLR

2024
[54]

Zhu, Y.; Gordon, D.; Kolve, E.; Fox, D.; Fei-Fei, L.; Gupta, A.; Mottaghi, R.; and Farhadi, A. 2017. Visual semantic planning using deep successor representations. In ICCV

2017
[55]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
[56]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Ahn, M.; Brohan, A.; Brown, N.; Chebotar, Y.; Cortes, O.; David, B.; Finn, C.; Fu, C.; Gopalakrishnan, K.; Hausman, K.; et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. In CoRL

work page 2022

[2] [2]

Anderson, P.; Wu, Q.; Teney, D.; Bruce, J.; Johnson, M.; S \"u nderhauf, N.; Reid, I.; Gould, S.; and van den Hengel, A. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR

work page 2018

[3] [3]

Bhambri, S.; Kim, B.; and Choi, J. 2023. Multi-level Compositional Reasoning for Interactive Instruction Following. In AAAI

work page 2023

[4] [4]

Blukis, V.; Paxton, C.; Fox, D.; Garg, A.; and Artzi, Y. 2021. A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution. In CoRL

work page 2021

[5] [5]

Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.;...

2020

[6] [6]

Chang, A.; Dai, A.; Funkhouser, T.; Halber, M.; Niessner, M.; Savva, M.; Song, S.; Zeng, A.; and Zhang, Y. 2017. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV

work page 2017

[7] [7]

S.; Sathyendra, K

Chaplot, D. S.; Sathyendra, K. M.; Pasumarthi, R. K.; Rajagopal, D.; and Salakhutdinov, R. 2017. Gated-attention architectures for task-oriented language grounding. In AAAI

work page 2017

[8] [8]

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374

arXiv 2021