Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One model with one set of weights can follow step-by-step instructions, dialogue, and object-goal searches on seven navigation benchmarks, matching or beating task-specific agents.

desk verdict A genuinely new state-adaptive MoE routing for multi-task VLN with a solid controlled baseline study, but the seven-task claim is weakened by an unvalidated ObjectNav data projection and an overstated comparison to specialists. read the letter →

arxiv 2412.05552 v1 pith:K2EOQEQO submitted 2024-12-07 cs.CV cs.AIcs.CLcs.LGcs.RO

Gengze Zhou , Yicong Hong , Zun Wang , Chongyang Zhao , Mohit Bansal , Qi Wu This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LGcs.RO
keywords vision-and-languagenavigationmixtureofexpertsmulti-tasklearninginstructionfollowingobjectgoalembodiedAIstate-adaptiveroutingbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that all forms of language-guided visual navigation fall on one spectrum—from step-by-step instructions, through dialogue and referring expressions, to bare object-category goals—and that a single agent, trained once, can cover the whole spectrum with one set of weights. Its model, SAME, uses a Mixture of Experts where a router picks different expert sub-networks at every timestep based on the agent's current visual-linguistic state. Trained on seven benchmarks simultaneously (R2R, RxR-EN, REVERIE, CVDN, SOON, ObjectNav-MP3D, R2R-CE), SAME is reported to match or beat specialists on six of the seven and to beat a multi-task DUET baseline on all of them. If correct, this would mean navigation agents no longer need per-benchmark fine-tuning, and that contrasting training data can be mixed productively once the model can specialize its skills.

What carries the argument

The load-bearing mechanism is the State-Adaptive Mixture of Experts (SAME) layer—a sparse MoE placed on the visual query of every cross-attention layer in a DUET-style transformer. At each timestep, a linear router reads the concatenation of the instruction's [CLS] token and the mean of the current panoramic view features, selects the top-k experts, and combines the selected experts' outputs with router probabilities; a load-balancing loss (weight λ = 0.8) prevents a few experts from dominating all routes. The paper shows that this state-based routing outperforms token-wise routing, task-wise routing, and routing with task embeddings appended, and that placing the experts on the visual query beats placing them in the feed-forward network or on the textual key/value projections, because cross-modal attention is the point at which navigation behaviors differentiate.

What would settle it

For the MP3D validation split, compute the fraction of projected episodes in which the projected endpoint is a different object than the human demonstrator reached, or in which the target object is never visible along the projected path; if that fraction is substantial, the reported ObjectNav numbers measure a shortcut-friendlier task than continuous ObjectNav, and the seven-task claim would need re-evaluation.

Watch

Extended reading notes

Core claim

The central claim is that navigation task conflicts sit not in the shared visual-semantic backbone but in how decisions are derived from it: the paper shows that training one DUET transformer on mixed navigation data degrades coarse-grained target search (REVERIE success drops 6–7% when ObjectNav data is added), while a model that routes through a sparse mixture of experts on the visual-query projection of cross-attention, with the router fed by the mean panoramic feature plus the instruction's [CLS] token, recovers and exceeds that performance. With this design, SAME is reported to reach 7.07 goal progress on CVDN test, 76.3% success on discrete ObjectNav-MP3D validation, and 73.9% success on R2R test, closely matching the best single-task models on R2R and REVERIE and clearly outperforming the multi-task-tuned baseline on every one of the seven tasks. The same checkpoint transfers zero-shot to continuous environments through a waypoint predictor, reporting 47% success on R2R-CE and 43% success on continuous ObjectNav-MP3D.

Load-bearing premise

The load-bearing assumption is that snapping the continuous human object-search demonstrations onto a discrete grid of navigable points—and throwing out any demonstration whose end point moves more than half a meter—still leaves training data that teaches the same exploration skill that continuous ObjectNav agents are judged on.

Editorial extensions

If this is right

  • A single checkpoint can serve seven navigation tasks at once, so downstream systems could replace seven fine-tuned navigation models with one model trained on mixed data, cutting storage and adaptation cost.
  • Coarse-grained and zero-grained navigation (REVERIE, SOON, ObjectNav) inherit better visual-semantic grounding from fine-grained instruction data, so future data collection can prioritize instruction-trajectory pairs even when the deployment task is pure object search.
  • The router conditions only on the current observation and language, not on task labels, so the same architecture can absorb a new task by adding its dataset to training without changing the routing inputs.
  • Discrete-to-continuous transfer via waypoint prediction works for a multi-task agent as well as for single-task ones, with zero-shot continuous success of 47% on R2R-CE and 43% on ObjectNav-MP3D.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The routing signal (mean view plus [CLS] token) discards spatial structure; a natural extension is to route on a spatial summary of the 36 views and test whether experts specialize even more cleanly.
  • Because no task embedding is used, the agent could discover skill groupings that cut across the seven labeled tasks; inspecting which experts fire during exploration-heavy versus instruction-heavy timesteps would test whether the learned specializations match the paper's exploration-versus-instruction-following story.
  • The discrete projection of Habitat-Web data is the main unvalidated link; re-training SAME on a denser connectivity graph or on continuous actions would show whether the reported ObjectNav numbers are an artifact of the 0.5-meter endpoint filter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SAME, a State-Adaptive Mixture of Experts model for unifying seven language-guided navigation tasks into a single policy. The model is built on the DUET architecture, initializes from ScaleVLN, and routes among experts based on a learned combination of the current visual observation and the language instruction rather than on task identity or individual tokens. The authors report experiments on R2R, RxR-EN, REVERIE, CVDN, SOON, ObjectNav-MP3D, and R2R-CE, and claim that one set of weights outperforms or is highly comparable to task-specific agents. The main evidence is a controlled comparison against a same-initialization multi-task DUET baseline (ScaleVLN†), plus a series of ablations on routing signal, expert placement, and training schema.

Significance. If the claims hold, the paper makes a useful contribution: it demonstrates that a single navigation policy can handle instruction granularities from fine-grained VLN to zero-grained ObjectNav, and the idea of routing experts based on the agent's multimodal state is well motivated. The experimental design comparing SAME against a multi-task DUET baseline initialized from the same ScaleVLN weights is a strength, because it isolates the effect of the MoE mechanism. The routing ablations in Tables 2 and 3 are informative and go beyond simply reporting final numbers. However, the seven-task claim is currently not fully supported: the ObjectNav evidence rests on an unvalidated discrete projection of continuous demonstrations, the REVERIE and SOON gaps against specialists contradict the 'highly comparable' wording, and all key comparisons are single runs without variance estimates.

major comments (5)
  1. [Section 2.2, Tables 4 and 5] The ObjectGoal Navigation evidence rests on an unvalidated projection of continuous Habitat-Web demonstrations onto the discrete MP3D connectivity graph. The text states that each viewpoint is matched to the nearest node, repeated nodes are merged, and 'disconnected paths and paths with an ending position more than 0.5m away from the original endpoint are removed,' yielding 58,803 training trajectories with average length 20 steps from demonstrations of average length 243 steps. Crucially, the paper does not report how many standard MP3D validation episodes survive the same filtering, nor how the dropped episodes are distributed across scenes and goal categories. If the filter preferentially removes long or hard episodes, SAME's Table 4 ObjectNav row (76.3 SR/42.7 SPL) and the Table 5 comparison (43 SR/21 SPL vs SemExp 28/11, PONI 32/12, Habitat-Web 35/10) are not measuring the same skill as the continuous-space baselines evaluated on unfiltered standard episodes. The authors should report retained-episode statistics, evaluate on the full MP3D validation split without endpoint filtering, or provide a continuous-space evaluation of SAME before the ObjectNav part of the seven-task claim can be accepted.
  2. [Section 4.1 and Abstract] The headline claim 'outperforms or achieves highly comparable performance to task-specific agents' is contradicted by the REVERIE and SOON rows of Table 4. On REVERIE Val Unseen, SAME obtains 46.4 SR/36.1 SPL versus 57.0/41.8 for ScaleVLN and 56.0/39.7 for VER; on SOON Val Unseen, SAME obtains 36.1 SR versus 40.4 for GOAT and 57.0 for ScaleVLN. These are gaps of 10-21 SR points, not 'highly comparable.' The sentence 'SAME performs at the same level as VER and GOAT comparing the SPL on R2R and REVERIE' refers only to SPL and does not cover SR. The claim should be restricted to the tasks where parity actually holds (CVDN, R2R in part), or the abstract and introduction should be revised to describe SAME as a competitive unified agent with task-specific trade-offs rather than as one that outperforms or matches specialists across all seven tasks.
  3. [Section 4.1, Table 5] The R2R-CE result depends on an external waypoint predictor: the text says, 'We follow Hong et al. [37] and deploy the waypoint predictor to bridge the gap between discrete and continuous,' but the paper does not specify which checkpoint was used, how the predictor was trained, or whether it is fixed for all compared methods. Moreover, SAME's R2R-CE numbers (47 SR/38 SPL) are below ScaleVLN (55/51) and BEVBert (59/50), so R2R-CE cannot be cited as evidence of outperformance. The paper does state that this is zero-shot inference, but the abstract-level seven-task claim does not carry this caveat. Please either remove R2R-CE from the 'outperforms or highly comparable' statement, provide a properly trained R2R-CE variant, or report full details of the waypoint module and clarify that the R2R-CE numbers are zero-shot with an external component.
  4. [Tables 2-7 and Table 4] All ablations and the main SAME-vs-ScaleVLN† comparison are single runs with no variance estimates. Since the claimed advantage over the same-initialization baseline averages about 3 SR points and individual task differences are sometimes zero (R2R Val Unseen is 76 vs 76), run-to-run noise could change the conclusions. Please report mean±std over at least three seeds for the key comparisons (SAME vs ScaleVLN† in Table 4, and the routing/position ablations in Tables 2 and 3), or provide statistical significance tests for the observed differences.
  5. [Section 4, Implementation Details] The paper never states the number of experts N or the top-k value used in the MoE layers, nor does it report the total parameter count after replacing visual-query projections with MoE layers. These values are required to reproduce the method and to interpret the routing ablations, since the router behavior and the load-balancing loss both depend on N and top-k. Please add these implementation details explicitly.
minor comments (5)
  1. [Section 4.1] Grammar and typos: 'SAME achieve State-of-the-Art performance' and 'SAME peform significantly better' should be corrected, and 'Dagger' should be written consistently as 'DAgger'.
  2. [Equations (7)-(9)] The load-balancing loss notation is inconsistent: Equation (8) sums over N experts but Equation (7) multiplies by N, and the index ranges are not fully defined. Please align the notation and define the batch size K explicitly.
  3. [Supplementary Table 8] The table is titled 'Full results of SAME on all VLN benchmarks' but does not include R2R-CE or ObjectNav-MP3D rows; please add those rows or rename the table to avoid misleading the reader.
  4. [Supplementary Section A] A sentence about decoupling vision-language-action alignment and improving VLN-CE success rates by approximately 20% appears twice in consecutive paragraphs; please remove the duplicate.
  5. [Throughout] The spacing in 'O BJECT NAV' is unusual and appears inconsistently; please use 'ObjectNav' consistently across the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: SAME's seven-task results come from held-out evaluation; the ObjectNav discretization raises benchmark comparability, not circularity.

full rationale

The paper does not claim to derive its benchmark numbers from its model definition; the reported results are empirical outcomes of training on held-out evaluation splits. The MoE routing mechanism, the balance loss coefficient (lambda = 0.8), and the data sampling ratio are training choices and hyperparameters, not fitted constants that predetermine the reported success rates. The ScaleVLN initialization is a self-citation by overlapping authors, but it is disclosed, and the paper controls for it by using the same initialization for the non-MoE multi-task baseline (ScaleVLN†), so the comparison isolates the effect of the proposed MoE routing rather than reducing to the pretrained weights. The ObjectNav evaluation uses a discretized projection of Habitat-Web demonstrations and validation episodes with a 0.5 m endpoint filter; this raises a valid question about whether SAME's ObjectNav numbers measure the same episodes as continuous-space baselines, but it is a benchmark-comparability and evaluation-validity concern, not circularity, because the projection does not encode SAME's predictions and the reported scores still come from an external evaluation protocol. No equation in the paper is equivalent to its own input by construction, and no fitted parameter is renamed as a prediction. Therefore, no significant circularity is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims are empirical and do not rest on a mathematical derivation, so the ledger is short. The main load-bearing choices are the hand-set data mixing ratio, the tuned balance coefficient, and the unreported MoE size. The Habitat-Web discretization is the strongest unvalidated processing assumption. No new physical entities are introduced, and no results are predefined by construction.

free parameters (3)
  • MoE load-balancing coefficient lambda = 0.8
    Chosen after sweeping 0.2, 0.5, 0.8, 1.0 in Table 7; the load-balancing loss depends on it and the paper concludes lambda=0.8 is needed for routing quality.
  • Training data sampling ratio = 10:1:1:1:1:1:2
    Hand-set ratio over ScaleVLN, R2R, RxR-EN, CVDN, REVERIE, SOON, Habitat-Web; not swept, but determines the multi-task training mix and is load-bearing for the unified-model results.
  • Number of experts N and top-k = not reported
    The MoE layer uses an unreported number of experts and top-k; these affect capacity and routing and are not specified in the paper.
assumptions (3)
  • ad hoc to paper Habitat-Web demonstrations transferred to the discrete MP3D graph by nearest-node matching with a 0.5m endpoint filter preserve enough signal for discrete ObjectNav learning.
    Section 2.2: 'we match each viewpoint in the trajectory from Habitat-Web to the nearest nodes on G* based on Euclidean distance... paths with an ending position more than 0.5m away from the original endpoint are removed.' This conversion is the foundation of the ObjectNav component of the seven-task claim.
  • domain assumption All seven tasks can be expressed as action selection over a shared discrete graph with the same action space (teleport/stop), making unified training meaningful.
    Section 2.1 formulates every task as a graph navigation problem with the same action space; this is a modeling assumption that ignores differences like continuous collision avoidance in ObjectNav.
  • domain assumption ScaleVLN pretrained weights provide a valid common initialization for all multi-task variants compared in the paper.
    Used in Section 3.3 and all experiments; the authors cite [106], which includes a co-author of the present paper, as the pretraining source. The comparison SAME vs ScaleVLN+ is fair because both use the same initialization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts." pith.science (2026). https://pith.science/paper/K2EOQEQO

@misc{pith2026241205552,
  author       = {Pith},
  title        = {Pith review of: SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K2EOQEQO}},
  note         = {Machine review of arXiv:2412.05552}
}
read the original abstract

The academic field of learning instruction-guided visual navigation can be generally categorized into high-level category-specific search and low-level language-guided navigation, depending on the granularity of language instruction, in which the former emphasizes the exploration process, while the latter concentrates on following detailed textual commands. Despite the differing focuses of these tasks, the underlying requirements of interpreting instructions, comprehending the surroundings, and inferring action decisions remain consistent. This paper consolidates diverse navigation tasks into a unified and generic framework -- we investigate the core difficulties of sharing general knowledge and exploiting task-specific capabilities in learning navigation and propose a novel State-Adaptive Mixture of Experts (SAME) model that effectively enables an agent to infer decisions based on different-granularity language and dynamic observations. Powered by SAME, we present a versatile agent capable of addressing seven navigation tasks simultaneously that outperforms or achieves highly comparable performance to task-specific agents.

Figures

Figures reproduced from arXiv: 2412.05552 by the authors.

Figure 1
Figure 1. We consolidate diverse navigation tasks into a unified language-guided navigation framework sorted by language granularity. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of MoE position and experts’ routing methods. SAME routing based on multimodal features from visual observations [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Iterative navigator-generator collaboration, where the navigator filters generated instructions and the rebuilt generator rewrites low-quality ones, raises R2R navigation SPL to 78% and instruction SPICE to 26.2.

  2. CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    CoNav lets a frozen 3D-text model pass spatial text hints to a lightly fine-tuned image-text navigation agent, improving path efficiency on several VLN benchmarks, though not all claimed state-of-the-art results hold.

Reference graph

Works this paper leans on

127 extracted references · 44 canonical work pages · cited by 2 Pith papers

  1. [37]

    Bridging the gap between learning in discrete and contin- uous environments for vision-and-language navigation

    Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and contin- uous environments for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 15439–15449, 2022. 3, 7, 15

    2022

  2. [1]

    Zero experience required: Plug & play modu- lar transfer learning for semantic visual navigation

    Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kris- ten Grauman. Zero experience required: Plug & play modu- lar transfer learning for semantic visual navigation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17031–17041, 2022. 8

    2022

  3. [2]

    Neighbor-view enhanced model for vi- sion and language navigation

    Dong An, Yuankai Qi, Yan Huang, Qi Wu, Liang Wang, and Tieniu Tan. Neighbor-view enhanced model for vi- sion and language navigation. In Proceedings of the 29th ACM International Conference on Multimedia , pages 5101–5109, 2021. 8

    2021

  4. [3]

    Bevbert: Topo-metric map pre-training for language-guided navigation

    Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Topo-metric map pre-training for language-guided navigation. arXiv preprint arXiv:2212.04385, 2022. 6, 7

    arXiv 2022

  5. [4]

    Etpnav: Evolving topo- logical planning for vision-language navigation in continu- ous environments

    Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. Etpnav: Evolving topo- logical planning for vision-language navigation in continu- ous environments. arXiv preprint arXiv:2304.03047, 2023. 7

    arXiv 2023

  6. [5]

    On evaluation of embodied navigation agents

    Peter Anderson, Angel Chang, Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018. 3, 8

    arXiv 2018

  7. [6]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S ¨underhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674– 3683, 2018. 1, 2, 3, ...

    2018

  8. [7]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S ¨underhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3674– 3683, 2018. 8

    2018

Show all 127 references
  1. [8]

    ObjectNav Revis- ited: On Evaluation of Embodied Agents Navigating to Ob- jects

    Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Olek- sandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. ObjectNav Revis- ited: On Evaluation of Embodied Agents Navigating to Ob- jects. In arXiv:2006.13171, 2020. 1, 2, 3, 15

    2006 arXiv

  2. [9]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 2017 Inter- national Conference on 3D Vision (3DV) , pages 667–676. IEEE, 2017. 3, 8

    2017

  3. [10]

    Object goal navi- gation using goal-oriented semantic exploration

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Ab- hinav Gupta, and Russ R Salakhutdinov. Object goal navi- gation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247–4258,

  4. [11]

    Neural topological slam for visual navigation

    Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for visual navigation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 12875–12884, 2020. 1, 8

    2020

  5. [12]

    Mapgpt: Map- guided prompting for unified vision-and-language naviga- tion

    Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xi- aodan Liang, and Kwan-Yee K Wong. Mapgpt: Map- guided prompting for unified vision-and-language naviga- tion. arXiv preprint arXiv:2401.07314, 2024. 8

    2024 arXiv

  6. [13]

    Topological planning with transformers for vision-and-language navigation

    Kevin Chen, Junshen K Chen, Jo Chuang, Marynel V´azquez, and Silvio Savarese. Topological planning with transformers for vision-and-language navigation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11276–11286, 2021. 8

    2021

  7. [14]

    History aware multimodal transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. Advances in Neural Infor- mation Processing Systems, 34:5834–5847, 2021. 1, 5, 8

    2021

  8. [15]

    History aware multimodal transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. Advances in Neural Infor- mation Processing Systems, 34:5834–5847, 2021. 7, 8

    2021

  9. [16]

    Think global, act local: Dual-scale graph transformer for vision-and-language nav- igation

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language nav- igation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16537– ...

    2022

  10. [17]

    Learning from unla- beled 3d environments for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Learning from unla- beled 3d environments for vision-and-language navigation. In European Conference on Computer Vision , pages 638–

  11. [18]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104–120. Springer, 2020. 1, 8

    2020

  12. [19]

    Octavius: Mitigating task interference in mllms via moe

    Zeren Chen, Ziqin Wang, Zhen Wang, Huayang Liu, Zhen- fei Yin, Si Liu, Lu Sheng, Wanli Ouyang, Yu Qiao, and Jing Shao. Octavius: Mitigating task interference in mllms via moe. arXiv preprint arXiv:2311.02684, 3, 2023. 8

    2023 arXiv

  13. [20]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 7

    2023

  14. [21]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language mod- els

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language mod- els. arXiv preprint arXiv:2401.06066, 2024. 1, 8 9

    2024 arXiv

  15. [22]

    Robothor: An open simulation-to-real embodied ai platform

    Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-real embodied ai platform. In Proceedings of the IEEE/CVF conference on compu...

  16. [23]

    Procthor: Large-scale embodied ai using procedural generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Ad- vances in Neural Information Processing Systems , 35: 59...

    2022

  17. [24]

    Foam: A follower-aware speaker model for vision-and-language navigation

    Zi-Yi Dou and Nanyun Peng. Foam: A follower-aware speaker model for vision-and-language navigation. arXiv preprint arXiv:2206.04294, 2022. 8

    2022 arXiv

  18. [25]

    Manipulathor: A framework for visual object manipulation

    Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli Vander- Bilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object manipulation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pag...

    2021

  19. [26]

    Eva: Exploring the limits of masked visual represen- tation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual represen- tation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2023

  20. [27]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. 4, 6, 8

    2022

  21. [28]

    Speaker-follower models for vision-and-language naviga- tion

    Daniel Fried, Ronghang Hu, V olkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-language naviga- tion. Advances in Neural Information Processing Systems...

    2018

  22. [29]

    Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation.arXiv preprint arXiv:2203.10421, 2022

    SY Gadre, M Wortsman, G Ilharco, L Schmidt, and S Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation.arXiv preprint arXiv:2203.10421, 2022. 8

    2022 arXiv

  23. [30]

    Room-and-object aware knowledge reasoning for remote embodied referring expression

    Chen Gao, Jinyu Chen, Si Liu, Luting Wang, Qiong Zhang, and Qi Wu. Room-and-object aware knowledge reasoning for remote embodied referring expression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3064–3073, 2021. 1

    2021

  24. [31]

    Mixture of cluster-conditional lora experts for vision-language instruction tuning

    Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint arXiv:2312.12379, 2023. 8

    2023 arXiv

  25. [32]

    Airbert: In-domain pretrain- ing for vision-and-language navigation

    Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In-domain pretrain- ing for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 1634–1643, 2021. 1, 5, 8

    2021

  26. [33]

    No rl, no simulation: Learning to navigate without navigat- ing

    Meera Hahn, Devendra Singh Chaplot, Shubham Tulsiani, Mustafa Mukadam, James M Rehg, and Abhinav Gupta. No rl, no simulation: Learning to navigate without navigat- ing. Advances in Neural Information Processing Systems , 34:26661–26673, 2021. 8

    2021

  27. [34]

    Towards learning a generic agent for vision- and-language navigation via pre-training

    Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision- and-language navigation via pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137–13146, 2020. 1, 5, 7, 8

    2020

  28. [35]

    Learning Language-Guided Visual Naviga- tion

    Yicong Hong. Learning Language-Guided Visual Naviga- tion. PhD thesis, The Australian National University (Aus- tralia), 2023. 8

    2023

  29. [36]

    A recurrent vision-and- language bert for navigation

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez- Opazo, and Stephen Gould. A recurrent vision-and- language bert for navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1643–1653, 2021. 1, 7, 8

    2021

  30. [38]

    Are you looking? ground- ing to multiple modalities in vision-and-language naviga- tion

    Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. Are you looking? ground- ing to multiple modalities in vision-and-language naviga- tion. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 6551–6557,

  31. [39]

    Trans- ferable representation learning in vision-and-language nav- igation

    Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, and Eugene Ie. Trans- ferable representation learning in vision-and-language nav- igation. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 7404–7413, 2019. 8

    2019

  32. [40]

    General evaluation for instruction con- ditioned navigation using dynamic time warping

    Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. General evaluation for instruction con- ditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446, 2019. 3

    1907 arXiv

  33. [41]

    Adaptive mixtures of local experts

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991. 1, 8

    1991

  34. [42]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. 1, 8

    2024 arXiv

  35. [43]

    Hierarchical mix- tures of experts and the em algorithm

    Michael I Jordan and Robert A Jacobs. Hierarchical mix- tures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994. 1, 8

    1994

  36. [44]

    A new path: Scaling vision- and-language navigation with synthetic instructions and im- itation learning

    Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh. A new path: Scaling vision- and-language navigation with synthetic instructions and im- itation learning. In Proceedings of the IEEE/CVF Confer...

    2023

  37. [45]

    Tactical rewind: Self-correction via backtracking in vision-and-language navigation

    Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Sid- dhartha Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In Pro- ceedings of the IEEE/CVF conference on computer vision ...

    2019

  38. [46]

    Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal naviga- tion

    Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X Chang, and Manolis Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal naviga-...

  39. [47]

    Goat-bench: A benchmark for multi-modal life- long navigation

    Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sri- ram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mottaghi. Goat-bench: A benchmark for multi-modal life- long navigation. In Proceedings of the IEEE/CVF Confer- enc...

    2024

  40. [48]

    Ai2-thor: An interactive 3d environment for visual ai

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Vander- Bilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017. 8

    2017 arXiv

  41. [49]

    Sim-2-sim transfer for vision- and-language navigation in continuous environments

    Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision- and-language navigation in continuous environments. In European Conference on Computer Vision, pages 588–603. Springer, 2022. 15

    2022

  42. [50]

    Beyond the nav-graph: Vision-and- language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Ba- tra, and Stefan Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. In Eu- ropean Conference on Computer Vision , pages 104–120. Springer, 2020. 2, 6, 15

    2020

  43. [51]

    Waypoint models for instruction-guided navigation in continuous environments

    Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction-guided navigation in continuous environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15162–15171, 2021. 15

    2021

  44. [52]

    Room-across-room: Multilingual vision-and-language navigation with dense spatiotempo- ral grounding

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotempo- ral grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pag...

    2020

  45. [53]

    Room-across-room: Multilingual vision-and-language navigation with dense spatiotempo- ral grounding

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotempo- ral grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pag...

    2020

  46. [54]

    Gshard: Scaling giant mod- els with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant mod- els with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. 6, 8

    2006 arXiv

  47. [55]

    Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation

    Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation. arXiv preprint arXiv:2305.19195, 2023. 8

    2023 arXiv

  48. [56]

    Improving vision-and-language navigation by generating future-view image semantics

    Jialu Li and Mohit Bansal. Improving vision-and-language navigation by generating future-view image semantics. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 10803–10812, 2023. 8

    2023

  49. [57]

    Improving cross- modal alignment in vision language navigation via syntac- tic information

    Jialu Li, Hao Tan, and Mohit Bansal. Improving cross- modal alignment in vision language navigation via syntac- tic information. arXiv preprint arXiv:2104.09580, 2021. 8

    2021 arXiv

  50. [58]

    Envedit: Environment editing for vision-and-language navigation

    Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15407–15417, 2022. 8

    2022

  51. [59]

    Visualbert: A simple and perfor- mant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. 1, 8

    1908 arXiv

  52. [60]

    Robust navigation with language pretraining and stochastic sampling

    Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling. arXiv preprint arXiv:1909.02244, 2019. 1, 8

    1909 arXiv

  53. [61]

    Oscar: Object-semantics aligned pre- training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xi- aowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre- training for vision-language tasks. In European Conference on Computer Vision, pages 121–137. Springer, 2020. 1, 8

    2020

  54. [62]

    Kerm: Knowledge enhanced reason- ing for vision-and-language navigation

    Xiangyang Li, Zihan Wang, Jiahao Yang, Yaowei Wang, and Shuqiang Jiang. Kerm: Knowledge enhanced reason- ing for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2583–2592, 2023. 8

    2023

  55. [63]

    Adapt: Vision- language navigation with modality-aligned action prompts

    Bingqian Lin, Yi Zhu, Zicong Chen, Xiwen Liang, Jianzhuang Liu, and Xiaodan Liang. Adapt: Vision- language navigation with modality-aligned action prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15396–15406, 2022. 8

    2022

  56. [64]

    Navcot: Boosting llm-based vision-and- language navigation via learning disentangled reasoning

    Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Navcot: Boosting llm-based vision-and- language navigation via learning disentangled reasoning. arXiv preprint arXiv:2403.07376, 2024. 8

    2024 arXiv

  57. [65]

    Scene-intuitive agent for remote embodied visual grounding

    Xiangru Lin, Guanbin Li, and Yizhou Yu. Scene-intuitive agent for remote embodied visual grounding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7036–7045, 2021. 1

    2021

  58. [66]

    Bird’s-eye-view scene graph for vision-language naviga- tion

    Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird’s-eye-view scene graph for vision-language naviga- tion. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10968–10980, 2023. 8

    2023

  59. [67]

    V olumetric envi- ronment representation for vision-language navigation

    Rui Liu, Wenguan Wang, and Yi Yang. V olumetric envi- ronment representation for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 16317–16328, 2024. 7, 8 11

    2024

  60. [68]

    Task-customized masked autoencoder via mixture of cluster-conditional ex- perts

    Zhili Liu, Kai Chen, Jianhua Han, Lanqing Hong, Hang Xu, Zhenguo Li, and James T Kwok. Task-customized masked autoencoder via mixture of cluster-conditional ex- perts. arXiv preprint arXiv:2402.05382, 2024. 8

    2024 arXiv

  61. [69]

    Dis- cuss before moving: Visual language navigation via multi- expert discussions

    Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. Dis- cuss before moving: Visual language navigation via multi- expert discussions. arXiv preprint arXiv:2309.11382, 2023. 8

    2023 arXiv

  62. [70]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04882, 2024. 8

    2024 arXiv

  63. [71]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learn- ing Representations, 2018. 6

    2018

  64. [72]

    Self- monitoring navigation agent via auxiliary progress estima- tion

    Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self- monitoring navigation agent via auxiliary progress estima- tion. arXiv preprint arXiv:1901.03035, 2019. 8

    1901 arXiv

  65. [73]

    The regretful agent: Heuristic-aided navigation through progress estimation

    Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 6732–6740, 2019. 8

    2019

  66. [74]

    Improving vision- and-language navigation with image-text pairs from the web

    Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter An- derson, Devi Parikh, and Dhruv Batra. Improving vision- and-language navigation with image-text pairs from the web. In European Conference on Computer Vision, pages 259–274. Springer, 2020. 1, 5, 8

    2020

  67. [75]

    Thda: Treasure hunt data augmentation for se- mantic navigation

    Oleksandr Maksymets, Vincent Cartillier, Aaron Gokaslan, Erik Wijmans, Wojciech Galuba, Stefan Lee, and Dhruv Batra. Thda: Treasure hunt data augmentation for se- mantic navigation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 15374– 1538...

    2021

  68. [76]

    Visual representations for semantic target driven navigation

    Arsalan Mousavian, Alexander Toshev, Marek Fi ˇser, Jana Koˇseck´a, Ayzaan Wahid, and James Davidson. Visual representations for semantic target driven navigation. In 2019 International Conference on Robotics and Automa- tion (ICRA), pages 8846–8852. IEEE, 2019. 8

    2019

  69. [77]

    Multimodal con- trastive learning with limoe: the language-image mixture of experts

    Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal con- trastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35:9564–9576, 2022. 8

    2022

  70. [78]

    Lang- nav: Language as a perceptual representation for naviga- tion

    Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim. Lang- nav: Language as a perceptual representation for naviga- tion. arXiv preprint arXiv:2310.07889, 2023. 8

    2023 arXiv

  71. [79]

    Counterfac- tual vision-and-language navigation: Unravelling the un- seen

    Amin Parvaneh, Ehsan Abbasnejad, Damien Teney, Javen Qinfeng Shi, and Anton van den Hengel. Counterfac- tual vision-and-language navigation: Unravelling the un- seen. Advances in Neural Information Processing Systems, 33:5296–5307, 2020. 8

    2020

  72. [80]

    Reverie: Remote embodied visual referring ex- pression in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring ex- pression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2020

  73. [81]

    Hop+: History-enhanced and order-aware pre-training for vision-and-language naviga- tion

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History-enhanced and order-aware pre-training for vision-and-language naviga- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 1, 5, 7, 8

    2023

  74. [82]

    March in chat: Interactive prompting for remote embodied referring expression

    Yanyuan Qiao, Yuankai Qi, Zheng Yu, Jing Liu, and Qi Wu. March in chat: Interactive prompting for remote embodied referring expression. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 15758– 15767, 2023. 8

    2023

  75. [83]

    Llm as copilot for coarse-grained vision-and-language navigation

    Yanyuan Qiao, Qianyi Liu, Jiajun Liu, Jing Liu, and Qi Wu. Llm as copilot for coarse-grained vision-and-language navigation. In European Conference on Computer Vision, pages 459–476. Springer, 2025. 8

    2025

  76. [84]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  77. [85]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wi- jmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew West- bury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai....

    2021

  78. [86]

    Poni: Potential functions for objectgoal navigation with interaction-free learning

    Santhosh Kumar Ramakrishnan, Devendra Singh Chap- lot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman. Poni: Potential functions for objectgoal navigation with interaction-free learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2022

  79. [87]

    Habitat-web: Learning embodied object-search strategies from human demonstrations at scale

    Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Ab- hishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5173–5183, 2022. 3, 7, 8, 15

    2022

  80. [88]

    Pirlnav: Pretraining with imitation and rl fine- tuning for objectnav

    Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Ab- hishek Das. Pirlnav: Pretraining with imitation and rl fine- tuning for objectnav. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 17896–17906, 2023. 8, 15

    2023

  81. [89]

    Scaling vision with sparse mix- ture of experts

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr´e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mix- ture of experts. Advances in Neural Information Processing Systems, 34:8583–8595, 2021. 8

    2021

  82. [90]

    A reduction of imitation learning and structured prediction to no-regret online learning

    St ´ephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the four- teenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and...

    2011

  83. [91]

    Habitat: A Platform for Embodied AI Re- search

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Re- search. ICCV, 2019. 3

    2019

  84. [92]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision ,...

    2019

  85. [93]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,

    arXiv

  86. [94]

    Scaling vision-language models with sparse mixture of experts

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226, 2023. 8

    2023 arXiv

  87. [95]

    Rrex-bot: Remote re- ferring expressions with a bag of tricks

    Gunnar A Sigurdsson, Jesse Thomason, Gaurav S Sukhatme, and Robinson Piramuthu. Rrex-bot: Remote re- ferring expressions with a bag of tricks. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5203–5210. IEEE, 2023. 1

    2023

  88. [96]

    Vl-bert: Pre-training of generic visual-linguistic representations

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530, 2019. 1, 8

    1908 arXiv

  89. [97]

    Lxmert: Learning cross- modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-...

    2019

  90. [98]

    Learning to navi- gate unseen environments: Back translation with environ- mental dropout

    Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navi- gate unseen environments: Back translation with environ- mental dropout. In Proceedings of NAACL-HLT , pages 2610–2621, 2019. 1, 7, 8

    2019

  91. [99]

    Vision-and-dialog navigation

    Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In Con- ference on Robot Learning, pages 394–406, 2020. 1, 2, 6, 8, 16

    2020

  92. [100]

    Active visual information gathering for vision-language navigation

    Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. Active visual information gathering for vision-language navigation. In European Conference on Computer Vision, pages 307–322. Springer, 2020. 8

    2020

  93. [101]

    Vision-and-language naviga- tion via causal learning

    Liuyi Wang, Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision-and-language naviga- tion via causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13139–13150, 2024. 7

    2024

  94. [102]

    Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation

    Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 37–53, 2018. 8

    2018

  95. [103]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation

    Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer...

    2019

  96. [104]

    Environment-agnostic multitask learning for natural language grounded naviga- tion

    Xin Eric Wang, Vihan Jain, Eugene Ie, William Yang Wang, Zornitsa Kozareva, and Sujith Ravi. Environment-agnostic multitask learning for natural language grounded naviga- tion. In European Conference on Computer Vision, pages 413–430. Springer, 2020. 7

    2020

  97. [105]

    Adamix: Mixture-of-adaptations for parameter- efficient model tuning

    Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xi- aodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jian- feng Gao. Adamix: Mixture-of-adaptations for parameter- efficient model tuning. arXiv preprint arXiv:2205.12410 ,

    arXiv

  98. [106]

    Scal- ing data generation in vision-and-language navigation

    Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mo- hit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scal- ing data generation in vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12009–12020, 2023. 5, 6, 7, 8

    2023

  99. [107]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15625–15636, 2023. 7, 8

    2023

  100. [108]

    Last-mile embodied visual navigation

    Justin Wasserman, Karmesh Yadav, Girish Chowdhary, Ab- hinav Gupta, and Unnat Jain. Last-mile embodied visual navigation. In Conference on Robot Learning, pages 666–

  101. [109]

    DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames

    Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames. InInternational Conference on Learning Representations (ICLR), 2020. 8

    2020

  102. [110]

    Gibson env: Real-world perception for embodied agents

    Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jiten- dra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 9068–9079, 2018. 8

    2018

  103. [111]

    Openmoe: An early effort on open mixture-of-experts language models

    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024. 1, 8

    2024 arXiv

  104. [112]

    Offline visual repre- sentation learning for embodied navigation

    Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual repre- sentation learning for embodied navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023 ,

    2023

  105. [113]

    Visual semantic navigation using scene priors

    Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. arXiv preprint arXiv:1810.06543, 2018. 8

    2018 arXiv

  106. [114]

    Auxiliary tasks and exploration enable objectgoal naviga- tion

    Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectgoal naviga- tion. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 16117–16126, 2021. 8 13

    2021

  107. [115]

    Homerobot: Open-vocabulary mobile ma- nipulation

    Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al. Homerobot: Open-vocabulary mobile ma- nipulation. arXiv preprint arXiv:2306.11565, 2023. 8

    2023 arXiv

  108. [116]

    Mc-gpt: Empowering vision-and-language navigation with memory map and reasoning chains

    Zhaohuan Zhan, Lisha Yu, Sijie Yu, and Guang Tan. Mc-gpt: Empowering vision-and-language navigation with memory map and reasoning chains. arXiv preprint arXiv:2405.10620, 2024. 8

    2024 arXiv

  109. [117]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and Wang He. Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852, 2024. 7, 8

    2024 arXiv

  110. [118]

    Vision-and-language navigation today and tomorrow: A survey in the era of foundation models

    Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-and-language navigation today and tomorrow: A survey in the era of foundation models. arXiv preprint arXiv:2407.07035, 2024. 8

    2024 arXiv

  111. [119]

    Mind the gap: Improving success rate of vision-and-language navigation by revisiting oracle success routes

    Chongyang Zhao, Yuankai Qi, and Qi Wu. Mind the gap: Improving success rate of vision-and-language navigation by revisiting oracle success routes. In Proceedings of the 31st ACM International Conference on Multimedia , pages 4349–4358, 2023. 8

    2023

  112. [120]

    Target- driven structured transformer planner for vision-language navigation

    Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang, Haibing Ren, Huaxia Xia, and Si Liu. Target- driven structured transformer planner for vision-language navigation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4194–4203, 2022. 8

    2022

  113. [121]

    Towards learning a generalist model for embod- ied navigation

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Li- wei Wang. Towards learning a generalist model for embod- ied navigation. arXiv preprint arXiv:2312.02010, 2023. 7, 8

    2023 arXiv

  114. [122]

    Navgpt: Explicit reasoning in vision-and-language navigation with large lan- guage models

    Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large lan- guage models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7641–7649, 2024

    2024

  115. [123]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. In European Conference on Computer Vision, pages 260–278. Springer,

  116. [124]

    Vision-language navigation with self-supervised auxiliary reasoning tasks

    Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 10012–10022, 2020. 8

    2020

  117. [125]

    Soon: Scenario oriented object navigation with graph-based exploration

    Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. Soon: Scenario oriented object navigation with graph-based exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12689–12699, 2021. 1, 2, 6, 8, 16

    2021

  118. [126]

    Target-driven visual navigation in indoor scenes using deep reinforce- ment learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforce- ment learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. IEEE,

    2017

  119. [127]

    where” and “how

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St- moe: Designing stable and transferable sparse expert mod- els. arXiv preprint arXiv:2202.08906, 2022. 6 14 SAME: Learning Generic Language-Guided Visual Navigation wi...

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.