Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

MUSON, a curated 800-frame dataset with five-step reasoning annotations, lets a three-billion-parameter vision-language model reach 86.25 percent decision accuracy for socially compliant navigation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:46 UTC pith:O5FW2YSE

load-bearing objection The dataset curation is sensible, but the benchmark numbers are unsupported because there's no train/test split, and the arXiv metadata abstract contradicts the paper's own abstract. the 4 major comments →

arxiv 2512.22867 v2 pith:O5FW2YSE submitted 2025-12-28 cs.CV cs.RO

MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments

Zhuonan Liu , Xinyu Zhang , Zishuo Wang , Runji Cai , Tomohito Kawabata , Qianyi Li , Xuance Peng , Tianze Yu
show 3 more authors
Zhen Xiong Xuesu Xiao Ling Xiao
This is my paper
classification cs.CV cs.RO
keywords socially compliant navigationvision-language modelschain-of-thoughtmultimodal datasetaction distributionegocentric perceptionbenchmarkfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces MUSON, a reasoning-oriented multimodal dataset for short-horizon social navigation, and argues that its design—structured five-step chain-of-thought annotations, a balanced six-action space, and physics-aware curation—is what lets small vision-language models learn socially compliant decisions. The central claim is that fine-tuning on MUSON lifts lightweight models to high decision accuracy (86.25% for Qwen2.5-VL-3B) with low collision rate (6.88%), demonstrating that the dataset is an effective, reusable benchmark. The paper also argues that the quality of CoT annotations matters: the same model trained with CoT improves on MUSON but degrades on an existing dataset with inconsistent annotations. A sympathetic reader would care because it offers a cheap, publicly available route to making small, deployable navigation models safer and more interpretable.

Core claim

The discovery is a dataset-plus-benchmark combination: 800 egocentric frames from real indoor and outdoor scenes, each annotated with a five-step chain of thought (perception, prediction, reasoning, action, explanation) in an ego-centric reference frame and a deliberately balanced distribution over six discrete actions—move forward, forward left, forward right, stop, turn left, turn right. The paper shows that after full-parameter supervised fine-tuning on this data, even a 3B-parameter vision-language model can reach 86.25% decision accuracy and a 6.88% collision rate, outperforming other small VLMs in the study. The stronger conceptual claim is that the dataset's geometry-aware action bala

What carries the argument

MUSON itself is the key machinery: a set of 800 curated egocentric frames with (1) a five-step chain-of-thought annotation schema that forces the model to parse perception, prediction, reasoning, action, and explanation in order; (2) a standardized six-action discrete space balanced by physical priors (56.6% forward) and geometric symmetry (equal left/right counts); and (3) a Human-AI-Human verification pipeline that arbitrates disagreements. The work it does is to convert implicit social norms into explicit, structured supervisory signals that small VLMs can absorb during fine-tuning.

Load-bearing premise

The reported accuracy and collision rate are only meaningful if they were measured on frames the model never saw during the 30-epoch fine-tuning; the paper never defines a train/test split, so the numbers could reflect memorization instead of generalization.

What would settle it

Re-run the fine-tuning and evaluation with an explicit, documented train/test split (e.g., 80/20 stratified by scene and action); if accuracy drops to near the untuned baseline or collision rate rises above 0.2, the benchmark's central generalization claim is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Fine-tuned lightweight VLMs can make socially compliant decisions from single egocentric frames, without long-horizon goals or maps.
  • CoT supervision helps small models when annotations are consistent, but hurts when they are inconsistent, implying data quality is a bottleneck for reasoning-oriented navigation.
  • Balancing the action distribution to preserve rare safety-critical behaviors (stop, turns) lets models learn those corner cases without sacrificing forward momentum.
  • The four evaluation tasks (action planning, scene perception, CoT reasoning, explanation) provide a multidimensional protocol for future social-navigation benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reported accuracy holds on a true held-out split, MUSON-style datasets could become a standard pre-training step for lightweight embodied models, reducing the need for large-scale instruction data.
  • The geometric balancing idea—equal left/right counts to defeat positional bias—could transfer to other egocentric decision benchmarks in driving and manipulation.
  • The Human-AI-Human annotation pipeline could be reused, but without inter-annotator agreement numbers, the ground-truth 'socially compliant' label remains subjective; measuring agreement is a cheap next step.
  • The paper's front abstract and body disagree on the best model (Qwen3-VL-8B vs Qwen2.5-VL-3B); this discrepancy should be resolved before relying on the headline numbers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MUSON, an 800-frame egocentric dataset for short-horizon socially compliant navigation, with a five-step chain-of-thought annotation schema (perception, prediction, reasoning, action, explanation), a six-action discrete decision space, and a curated action distribution intended to mitigate long-tail bias while preserving safety-critical corner cases. The dataset is derived from MuSoHu and is accompanied by a Human-AI-Human annotation pipeline. Four small VLMs are fine-tuned on MUSON; the authors report that Qwen2.5-VL-3B achieves the best decision accuracy (0.8625) and lowest collision rate (0.0688), and that CoT supervision helps on MUSON but hurts on SNEI. The paper claims MUSON is an effective and reusable benchmark for socially compliant navigation.

Significance. If the claims hold, MUSON would be a useful community resource: its structured CoT annotations, balanced action distribution, and explicit modeling of both dynamic and static constraints address real limitations of existing social-navigation datasets such as SNEI. The proposed collision-rate metric and the comparison of several small VLMs are also practically relevant, especially for lightweight deployed systems. The dataset release and the public benchmark baselines are strengths. However, the central empirical conclusion — that fine-tuning on MUSON yields strong, socially compliant decisions — is currently not supported by a clearly documented evaluation protocol, and the paper contains unresolved internal inconsistencies about dataset scale, model set, and headline results. The contribution is therefore conditional on substantial corrections.

major comments (4)
  1. [§V-A and Table II] The experimental setup states 30 epochs of full-parameter SFT with global batch size 32 on 800 samples, but no train/validation/test split or evaluation protocol is defined anywhere. With 800 curated samples and 30 epochs, the reported Qwen2.5-VL-3B Acc=0.8625, CR=0.0688 may be in-sample fitting. The phrase 'strongest overall fitting performance' in §V-B reinforces this concern. A benchmark claim requires held-out or cross-validated metrics; without this, the central reusable-benchmark conclusion is unsupported.
  2. [Abstract vs. body] The body of the paper (Abstract, §III-A, Table I, Table II, Conclusion) consistently describes an 800-frame dataset with Qwen2.5-VL-3B as best (Acc 0.8625, CR 0.0688), while the arXiv metadata abstract states MUSON contains 10,110 samples, benchmarks ten small-to-medium VLMs, and reports Qwen3-VL-8B with Acc 0.7765 and CR 0.0609 as best. These are irreconcilable statements about dataset scale, model set, and headline results. The authors must state which version is correct and provide a single consistent set of numbers, or the paper cannot be evaluated.
  3. [§III-C] AI-Assisted Quality Control uses a TinyLLaVA model 'trained on MUSON' to flag samples whose inferred reasoning or decisions deviate from human annotations, and the same model family is later benchmarked on MUSON. This creates a self-referential QC loop: the auditor is fitted to the labels it is asked to validate. Although final arbitration is human, no inter-annotator agreement, number of flagged samples, or distribution of human-AI disagreements is reported. Without such statistics, the reliability of the ground-truth labels — on which all benchmark numbers rest — is not established.
  4. [§V-B and Table III] The comparison with SNEI under Action-Only and CoT settings lacks essential details. It is unclear whether SNEI was split into train/test, how many SNEI samples were used, and whether the same number of epochs, prompts, and evaluation metrics were used. The claim that CoT 'degrades performance on SNEI' (0.5833→0.4667) is therefore difficult to interpret; it could reflect insufficient data, different annotation formats, or evaluation on training frames rather than annotation inconsistency in SNEI.
minor comments (5)
  1. [§I/Abstract] The release URL is inconsistent: the abstract mentions HuggingFace while the contribution section mentions a GitHub release. Please provide a single, stable public URL.
  2. [§IV-A, Eq. (1)] The collision rate treats 'turn left' and 'turn right' as avoidance actions grouped with 'stop.' This is nonstandard and should be justified, or the metric should separate true stopping from turning maneuvers.
  3. [§III-B] The action balancing thresholds (56.6% move-forward, 118 oblique samples, 32 turn samples) are presented as physically grounded, but no sensitivity analysis is given. State whether the reported benchmark conclusions are robust to reasonable variations of these targets.
  4. [Fig. 3] Only the TinyLLaVA fine-tuning loss/gradient curves are shown. Label the figure accordingly and indicate whether the other models showed similar convergence dynamics.
  5. [Table III] The header 'Five-Turn Conversation' is used interchangeably with 'CoT'; align the terminology with the five-step CoT annotation schema to avoid confusion.

Circularity Check

2 steps flagged

Table II reports in-sample fit as benchmark prediction: no train/test split is defined, so the headline accuracy/collision rate reduce to memorization of the training labels.

specific steps
  1. fitted input called prediction [Section V-A 'Implementation Details'; Section V-B 'Main Evaluations'; Table II]
    "Training is conducted for a total of 30 epochs to ensure adequate learning of rare samples under the long-tailed data distribution. ... Qwen2.5-VL-3B achieves the strongest overall fitting performance. ... Table II: MAIN EVALUATION RESULTS ON MUSON DATASET."

    The evaluation protocol is absent: models are trained by full-parameter SFT for 30 epochs on MUSON, and Table II reports accuracy, Macro-F1, CR, and SBERTScore 'on MUSON dataset'. No train/validation/test split or held-out set is defined anywhere in Section V. The only characterization of Table II is 'fitting performance', indicating the numbers are fit values. Therefore the headline 0.8625 accuracy and 0.0688 CR measure reproduction of the training labels rather than out-of-sample prediction, and the conclusion that MUSON is reusable and that models 'generalize effectively' rests on fitted values, not on a prediction.

  2. other [Section III-C, 'AI-Assisted Quality Control', step 2]
    "AI-Assisted Audit. We employ a TinyLLaVA model trained on MUSON to perform automated logical consistency checks. Samples in which the model’s inferred reasoning or decisions substantially deviate from human annotations are flagged for further review."

    This quality-control model is trained on the very dataset whose labels it is asked to audit: a TinyLLaVA model 'trained on MUSON' checks MUSON's human annotations. Since it has been fitted to those annotations, deviation from human annotations can only indicate model error or a label it did not memorize; it cannot provide independent evidence of annotation fidelity. The final human arbitration stage breaks the self-reference for the final GT, but the 'AI-Assisted' audit as described is a self-referential loop.

full rationale

Most of MUSON is a data-construction and empirical benchmarking paper rather than a formal derivation chain: the action taxonomy, geometric balancing, five-step CoT schema, and SNEI comparisons stand on their own and are not circular. The central claim that fine-tuning on MUSON yields a reusable benchmark is compromised, however, by the evaluation as reported. Section V-A trains models with full-parameter SFT for 30 epochs on MUSON; Table II then reports Acc/Macro-F1/CR/SBERTScore 'on MUSON dataset' with no train/validation/test split or evaluation protocol. The text itself calls this 'overall fitting performance', the signature of a fitted value presented as a prediction. With only 800 samples and 30 epochs, memorization is plausible, so the 0.8625 accuracy and 0.0688 CR do not demonstrate generalization unless a held-out split is assumed, and the paper never states one. A secondary self-reference occurs in Section III-C: the TinyLLaVA auditor is trained on MUSON and used to audit MUSON annotations; this is a circular QC loop, though final human arbitration keeps it from being the main result. The self-citations in the paper (e.g., SocialNav-MoE, LLM-advisor, MuSoHu) are not load-bearing: MuSoHu is cited as an external data source, and no imported uniqueness theorem forces the conclusions. Score 6 because the headline benchmark result, as written, reduces to in-sample fit; this is partial circularity rather than a full definitional collapse.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

All free parameters are hand-chosen curation/definitional choices that directly shape the benchmark numbers; there is no fitted model in the dataset construction. The axioms are domain assumptions about what 'socially compliant' means and about the metrics used to measure it.

free parameters (3)
  • Move-forward retention ratio = 0.566 (453/800)
    Hand-chosen in §III-B: 'we retained approximately 56.6% of move forward samples' to reflect physical priors; this determines the action distribution that the benchmark then measures accuracy on.
  • Oblique/turn count targets = 118/118 and 32/32
    Exact counts chosen for geometric symmetry (§III-B) so that left/right are equal; the specific numbers are arbitrary and affect Macro-F1 and CR results.
  • A_go/A_stop partition in collision rate = A_go={move forward, forward-left, forward-right}; A_stop={stop, turn-left, turn-right}
    Hand-defined in Eq. (1). Classifying turn actions as 'avoidance' determines which prediction errors count as collisions, so the CR numbers are a function of this choice.
axioms (4)
  • domain assumption SBERT cosine similarity is a valid proxy for logical/semantic consistency of navigation reasoning
    §IV-B uses Eq. (2) to equate embedding similarity with reasoning quality, stated as a posit ('Navigation as Semantics'), with no validation against human-judged reasoning correctness.
  • domain assumption The six-action discrete space is a sufficient decision target for short-horizon social navigation
    §III-B defines the action set; real navigation decisions involving speed, yielding timing, and trajectory shape are collapsed into six tokens, so the benchmark measures a restricted proxy.
  • domain assumption The human-AI-human pipeline yields correct ground-truth actions and annotations
    §III-C describes two annotators plus a senior expert but reports no inter-annotator agreement; the benchmark treats these labels as ground truth.
  • domain assumption Collision-rate asymmetry operationalizes safety cost
    Eq. (1) assumes aggressive errors are the only costly errors and conservative errors are negligible, following [30]; this is a modeling choice, not a measured cost.

pith-pipeline@v1.3.0-alltime-deepseek · 10872 in / 21745 out tokens · 203728 ms · 2026-08-03T13:46:55.159333+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments." pith.science (2026). https://pith.science/paper/O5FW2YSE

@misc{pith2026251222867,
  author       = {Pith},
  title        = {Pith review of: MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5FW2YSE}},
  note         = {Machine review of arXiv:2512.22867}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Socially compliant navigation requires structured reasoning about dynamic pedestrians and physical constraints to ensure safe and interpretable decisions. Vision-language models (VLMs) provide a promising foundation for this task because they can integrate visual observations with language-based social knowledge. However, existing untuned VLMs still struggle to reliably understand fine-grained social norms, making task-specific fine-tuning essential. At the same time, no large-scale egocentric dataset is available this task. To address these challenges, we introduce MUSON, a multimodal dataset for short-horizon social navigation containing 10,110 egocentric samples collected across diverse indoor and outdoor social scenes. MUSON adopts a structured five-step chain-of-thought annotation framework comprising perception, prediction, reasoning, action, and explanation. It explicitly models static physical constraints and employs a standardized six-action decision space. Compared with existing social-navigation datasets, MUSON provides consistent annotations for reasoning, actions, and explanations. We evaluate ten representative small-to-medium VLMs on MUSON. Qwen3-VL-8B achieves the strongest decision-level performance, attaining the highest action accuracy of 0.7765 and Macro-F1 score of 0.7490, as well as the lowest collision rate of 0.0609. These results demonstrate that MUSON is an effective and reusable benchmark for advancing socially compliant navigation. The dataset is publicly available at https://github.com/MUSON-dataset/MUSON/releases/tag/v1.0.

Figures

Figures reproduced from arXiv: 2512.22867 by Ling Xiao, Qianyi Li, Runji Cai, Tianze Yu, Tomohito Kawabata, Xinyu Zhang, Xuance Peng, Xuesu Xiao, Zhen Xiong, Zhuonan Liu, Zishuo Wang.

Figure 1
Figure 1. Figure 1: The MUSON dataset construction pipeline. MUSON covers diverse real-world scenes across indoor and outdoor environments with varying crowd densities. Continuous navigation behaviors are discretized into six egocentric action categories with a rational distribution that preserves safety-critical corner cases to mitigate long-tail issues. Each sample is annotated using a structured five-step CoT, followed by … view at source ↗
Figure 2
Figure 2. Figure 2: Visualization comparisons of samples from SNEI and MUSON. C. Decision Accuracy and Fairness Accuracy: This metric measures the prediction correctness across all samples, which is calculated as: Accuracy = 1 N X N i=1 I(a i pred = a i gt), (3) Macro-averaged F1 Score: Given the long-tailed distribution of navigation data (dominated by move forward), the above￾mentioned accuracy metric (Eq. 3) tends to mask … view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of high-level decision results for different SVLMs on MUSON. TABLE III IMPACT OF COT QUALITY ON DECISION ACCURACY. QUANTITATIVE COMPARISON WITH MOE-LLAVA UNDER ACTION-ONLY AND FIVE-TURN CONVERSATION SETTINGS. FIVE-TURN TRAINING IMPROVES ACCURACY ON MUSON BUT DEGRADES PERFORMANCE ON SNEI, LIKELY DUE TO INCONSISTENT REASONING, ACTIONS, AND EXPLANATIONS IN SNEI DATASET. Configuration Acc. ↑ SNEI… view at source ↗
Figure 3
Figure 3. Figure 3: Training loss and gradient norm curves. (a) Fine-tuning loss curve of TinyLLaVA on MUSON. (b) Corresponding gradient norm curve. size is 4, resulting in a global batch size of 32 across all GPUs. Training is conducted for a total of 30 epochs to ensure adequate learning of rare samples under the long-tailed data distribution. B. Main Evaluations Comparative Analysis of SVLM Baselines. As shown in Table II,… view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of high-level reasoning quality for different SVLMs on MUSON. ful supervision. In contrast, on MUSON, CoT consistently improves performance (0.5375 → 0.5813), demonstrating that well-structured and logically consistent annotations allow CoT to function as an effective reasoning scaffold. These results confirm that MUSON provides reliable supervision for reasoning-oriented navigation, whereas … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation

    cs.RO 2026-07 reject novelty 5.0

    SOPD claims selective entropy-based on-policy distillation improves lightweight VLM social navigation, but its entropy mask is mathematically vacuous as printed.

  2. Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

    cs.RO 2026-07 conditional novelty 5.0

    Conditional VLM reasoning triggered by personal-space violations improves social-navigation success by up to 20 points over RL-only baselines while keeping most steps on a fast RL policy.

  3. Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control

    cs.RO 2026-06 unverdicted novelty 4.0

    Survey organizing VLM-based social robot navigation into reasoning, planning, and bridging components with a proposed roadmap for hybrid deployable systems.

Reference graph

Works this paper leans on

34 extracted references · 10 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3674–3683

    2018

  2. [2]

    Room-across- room: Multilingual vision-and-language navigation with dense spa- tiotemporal grounding,

    A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge, “Room-across- room: Multilingual vision-and-language navigation with dense spa- tiotemporal grounding,” inProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 4379– 4412

    2020

  3. [3]

    Convoi: Context-aware naviga- tion using vision language models in outdoor and indoor environments,

    A. J. Sathyamoorthy, K. Weerakoon, M. Elnoor, A. Zore, B. Ichter, F. Xia, J. Tan, W. Yu, and D. Manocha, “Convoi: Context-aware naviga- tion using vision language models in outdoor and indoor environments,” inProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 13 837–13 844. 8

    2024

  4. [4]

    Gson: A group-based social navigation framework with large multimodal model,

    S. Luo, P. Sun, J. Zhu, Y . Deng, C. Yu, A. Xiao, and X. Wang, “Gson: A group-based social navigation framework with large multimodal model,” IEEE Robotics and Automation Letters, 2025

    2025

  5. [5]

    Socialnav-moe: A mixture-of- experts vision language model for socially compliant navigation with reinforcement fine-tuning,

    T. Kawabata, X. Zhang, and L. Xiao, “Socialnav-moe: A mixture-of- experts vision language model for socially compliant navigation with reinforcement fine-tuning,”arXiv preprint arXiv:2512.14757, 2025

    arXiv 2025

  6. [6]

    Social-llava: Enhancing robot navigation through human-language reasoning in social spaces,

    A. Payandeh, D. Song, M. Nazeri, J. Liang, P. Mukherjee, A. H. Raj, Y . Kong, D. Manocha, and X. Xiao, “Social-llava: Enhancing robot navigation through human-language reasoning in social spaces,”arXiv preprint arXiv:2501.09024, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,

    J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,” inProceedings of the European Conference on Computer Vision (ECCV), 2020

    2020

  8. [8]

    Gnm: A general navigation model to drive any robot,

    D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine, “Gnm: A general navigation model to drive any robot,” inProceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 7226–7233

    2023

  9. [9]

    Navid: Video-based vlm plans the next step for vision- and-language navigation,

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang, “Navid: Video-based vlm plans the next step for vision- and-language navigation,”arXiv preprint arXiv:2402.15852, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments,

    R. Martin-Martin, M. Patel, H. Rezatofighi, A. Shenoi, J. Gwak, E. Frankel, A. Sadeghian, and S. Savarese, “Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 6748–6765, 2021

    2021

  11. [11]

    Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,

    H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 807–11 814, 2022

    2022

  12. [12]

    End- to-end driving via conditional imitation learning,

    F. Codevilla, M. M ¨uller, A. L´opez, V . Koltun, and A. Dosovitskiy, “End- to-end driving via conditional imitation learning,” inProceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 4693–4700

    2018

  13. [13]

    Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst,

    M. Bansal, A. Krizhevsky, and A. Ogale, “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst,”arXiv preprint arXiv:1812.03079, 2018

    Pith/arXiv arXiv 2018

  14. [14]

    Exploring the limitations of behavior cloning for autonomous driving,

    F. Codevilla, E. Santana, A. M. L ´opez, and A. Gaidon, “Exploring the limitations of behavior cloning for autonomous driving,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 9329–9338

    2019

  15. [15]

    Palm-e: an embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yuet al., “Palm-e: an embodied multimodal language model,” inProceedings of the 40th International Conference on Machine Learning (ICML), 2023, pp. 8469–8488

    2023

  16. [16]

    Rt-2: Vision-language-action models trans- fer web knowledge to robotic control,

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahidet al., “Rt-2: Vision-language-action models trans- fer web knowledge to robotic control,” inProceedings of the Conference on Robot Learning (CoRL), 2023, pp. 2165–2183

    2023

  17. [17]

    Inner monologue: Embodied reasoning through planning with language models,

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Inner monologue: Embodied reasoning through planning with language models,” inPro- ceedings of the Conference on Robot Learning (CoRL), 2022, pp. 1769– 1782

    2022

  18. [18]

    React: Synergizing reasoning and acting in language mod- els,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language mod- els,” inProceedings of the 11th International Conference on Learning Representations (ICLR), 2022

    2022

  19. [19]

    V oxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxposer: Composable 3d value maps for robotic manipulation with language models,” inProceedings of the Conference on Robot Learning (CoRL), 2023, pp. 540–562

    2023

  20. [20]

    Chatgpt for robotics: Design principles and model abilities,

    S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “Chatgpt for robotics: Design principles and model abilities,”IEEE Access, vol. 12, pp. 5565–5578, 2024

    2024

  21. [21]

    Chatgpt empowered long-step robot control in various environments: A case application,

    N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, “Chatgpt empowered long-step robot control in various environments: A case application,” inProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 86– 93

    2023

  22. [22]

    Llm-advisor: An llm benchmark for cost-efficient path planning across multiple terrains,

    L. Xiao and T. Yamasaki, “Llm-advisor: An llm benchmark for cost-efficient path planning across multiple terrains,”arXiv preprint arXiv:2503.01236, 2025

    arXiv 2025

  23. [23]

    Tinyllama: An open-source small language model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,”arXiv preprint arXiv:2401.02385, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Mobilevlm: A fast, strong and open vision language assistant for mobile devices,

    X. Chu, L. Qiao, X. Lin, S. Xu, Y . Yang, Y . Hu, F. Wei, X. Zhang, B. Zhang, X. Weiet al., “Mobilevlm: A fast, strong and open vision language assistant for mobile devices,”arXiv preprint arXiv:2312.16886, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Tinyllava: A framework of small-scale large multimodal models,

    B. Zhou, Y . Hu, X. Weng, J. Jia, J. Luo, X. Liu, J. Wu, and L. Huang, “Tinyllava: A framework of small-scale large multimodal models,”arXiv preprint arXiv:2402.14289, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Moe-llava: Mixture of experts for large vision-language models,

    B. Lin, Z. Tang, Y . Ye, J. Cui, B. Zhu, P. Jin, J. Huang, J. Zhang, Y . Pang, M. Ninget al., “Moe-llava: Mixture of experts for large vision-language models,”arXiv preprint arXiv:2401.15947, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,” inProceedings of the Advances in neural information processing systems (NeurIPS), vol. 35, 2022, pp. 24 824–24 837

    2022

  29. [29]

    Toward human-like social robot navigation: A large-scale, multi-modal, social human navigation dataset,

    D. M. Nguyen, M. Nazeri, A. Payandeh, A. Datar, and X. Xiao, “Toward human-like social robot navigation: A large-scale, multi-modal, social human navigation dataset,” inProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 7442– 7447

    2023

  30. [30]

    Costnav: A navigation benchmark for cost- aware evaluation of embodied agents,

    H. Seong, S. Kim, M. Kim, Y . Cho, M. Joe, S. Choi, J. Jung, J. Youn, Y . Kim, S. Seonget al., “Costnav: A navigation benchmark for cost- aware evaluation of embodied agents,”arXiv preprint arXiv:2511.20216, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL), 2002, pp. 311–318

    2002

  32. [32]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” inProceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982–3992

    2019

  33. [33]

    Nvila: Efficient frontier visual language models,

    Z. Liu, L. Zhu, B. Shi, Z. Zhang, Y . Lou, S. Yang, H. Xi, S. Cao, Y . Gu, D. Liet al., “Nvila: Efficient frontier visual language models,” in Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025, pp. 4122–4134

    2025

  34. [34]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2. 5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025