Pith. sign in

REVIEW 4 major objections 6 minor 5 cited by

Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fine-tuning a vision-language model on 40K human-annotated questions about crowded social scenes produces human-like reasoning about what a robot should do, and that the result outscores GPT-4V and Gemini in…

desk verdict The SNEI dataset is a genuinely useful new resource, but the claimed outperformance over GPT-4V and Gemini is unsubstantiated because the evaluation rests on 50 items that may come from the training pool. read the letter →

arxiv 2501.09024 v1 pith:JTGL7GUW submitted 2024-12-30 cs.CV cs.HCcs.RO

Amirreza Payandeh , Daeun Song , Mohammad Nazeri , Jing Liang , Praneel Mukherjee , Amir Hossain Raj , Yangzhe Kong , Dinesh Manocha
show 1 more author
Xuesu Xiao
This is my paper · ORCID
classification cs.CVcs.HCcs.RO
keywords socialrobotnavigationvision-languagemodelsvisualquestionansweringchain-of-thoughtreasoninghuman-robotinteractiondatasetexplainableAIcrowdedenvironments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that language can be the missing link between what a social robot sees and how it should act in crowded human spaces. To that end the authors built SNEI, a dataset of more than 40,000 human-annotated visual question-answer pairs drawn from 2,000 human-robot interactions, organized into perception, prediction, chain-of-thought reasoning, final action, and explanation. They fine-tuned a vision-language model, Social-LLaVA, on SNEI and report that it outperforms GPT-4V and Gemini 1.5 Pro across fifteen human-judge scores on fifty social-navigation questions. A proof-of-concept deployment maps its language outputs to simple robot actions. If the claim holds, social navigation can move from hand-coded rules toward explainable, language-driven reasoning.

What carries the argument

The load-bearing mechanism is SNEI, a vision-language dataset of more than 40,000 human-annotated visual question-answer pairs built from 2,000 scenarios selected from the SCAND social-navigation demonstration corpus. Each annotation is organized into five stages — perception, prediction, chain-of-thought reasoning, final action, and explanation — so a model trained on it must verbalize the why before the action. The paper's model, Social-LLaVA, is LLaVA-v1.5-7B fine-tuned with LoRA on this dataset; the five-stage language chain, not a new planner or controller, is what carries the argument from image to high-level action.

What would settle it

Retrain Social-LLaVA with every image used in the fifty-question human evaluation held out of the SNEI pool, and rerun the same fifteen-judge scoring; if its average scores fall to the level of zero-shot GPT-4V and Gemini, the reported superiority is memorization, not learned reasoning.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that general-purpose vision-language models fail at social navigation because their training data lacks the spatial and social common sense of unstructured, crowded environments, and that a compact, fully human-annotated dataset of social-navigation VQAs closes this gap. Social-LLaVA, which is LLaVA-v1.5-7B fine-tuned with LoRA on SNEI, outputs a five-stage chain: it describes what the robot perceives, predicts what people will do, reasons about the safest response, names a high-level action such as 'stop and wait for clear path,' and explains its choice. In the paper's human evaluation, this fine-tuned model scored between 3.95 and 4.19 across the five stages, ahead of GPT-4V and Gemini 1.5 Pro's 2.77 to 3.87 range, and the authors report that the larger closed models produced hallucinations and socially blocking suggestions. The contribution is framed as the dataset rather than algorithmic novelty.

Load-bearing premise

The evaluation assumes the fifty human-judged questions were not part of the fine-tuning set, but the paper reports no train/test split to guarantee that.

Editorial extensions

If this is right

  • Social-LLaVA can generate human-readable explanations of navigation decisions, making the robot's behavior inspectable rather than a hidden planner output.
  • The SNEI five-stage answer format gives the social-navigation field a standardized benchmark for comparing perception, prediction, reasoning, action, and explanation quality separately.
  • A VLM fine-tuned on SNEI can propose safe high-level actions in crowded scenes without hand-crafted behavioral rules, based on the reported comparison against general-purpose models.
  • The proof-of-concept robot experiment shows that high-level language instructions can be mapped to simple actions such as going straight, turning left, and turning right.
  • If the human-judge scores generalize, robots can use chain-of-thought language reasoning in ambiguous social cases, such as not interrupting a conversation, where obstacle-avoidance-only policies fail.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to hold out all fifty evaluation images during fine-tuning and rerun the human scoring; without that split, the comparison to zero-shot models cannot distinguish learned reasoning from memorized annotations.
  • Because SNEI annotates single images, extending the same five-stage annotation to short video clips would test whether prediction and reasoning improve with temporal context; the paper identifies video data as future work.
  • The real-robot demonstration maps language to actions with a hard-coded table, so the social-compliance claim currently depends on that fixed mapping; a learned grounding module would make the pipeline self-contained.
  • Re-annotating or testing in a different country or culture would reveal how much of what SNEI encodes is a universal social norm rather than a local convention; the paper acknowledges cultural variability as an open issue.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SNEI, a human-annotated VQA dataset for social robot navigation built from 2K SCAND scenarios and containing 40K question-answer pairs spanning perception, prediction, chain-of-thought reasoning, final action, and explanation. The authors fine-tune LLaVA-v1.5-7B with LoRA (Social-LLaVA) on SNEI and report that it outperforms GPT-4V and Gemini 1.5 Pro in human-judged scores on 50 VQA tasks. A simple real-robot proof of concept with a hard-coded language-to-action mapping is also presented. The paper's main claims are the value of SNEI as a dataset and the superiority of Social-LLaVA over zero-shot foundation models.

Significance. If the comparative result were established, the paper would make a useful contribution: a human-annotated, explainable VQA dataset for social navigation is currently missing, and the chain-of-thought annotation structure is well matched to downstream language-guided navigation. The authors are appropriately modest about algorithmic novelty, explicitly framing Social-LLaVA as a proof-of-concept baseline. However, the current evaluation does not support the central comparative claim because the evaluation items appear to be drawn from the same SNEI pool used for fine-tuning, no statistical analysis is provided, and the dataset itself is not released. The real-robot experiment is transparently acknowledged to rely on a hard-coded action mapping and therefore does not validate end-to-end navigation.

major comments (4)
  1. [Section IV-B, Table I] The comparison against GPT-4V and Gemini is confounded by a possible train/test overlap. Social-LLaVA is fine-tuned on SNEI for 15 epochs (Section IV-A), while the 50 VQAs used for the human evaluation are apparently drawn from the same SNEI pool, since Fig. 3 is described as "one of the examples from our dataset" and the paper never reports a held-out split. If any of the evaluation items were used in training, Social-LLaVA would have seen both the images and gold answers, making the margins in Table I (e.g., Final Action 2.77/3.46 vs. 4.19) uninterpretable as evidence of reasoning. Please specify how the 50 tasks were selected, state the exact split, and either release the evaluation set or evaluate on an external benchmark.
  2. [Section IV-B] The evaluation reports only mean scores over fifteen judges (Table I) and provides no variance, confidence intervals, significance tests, or inter-rater reliability. The abstract and conclusions use the word "significantly" ("significantly higher scores"), but no statistical test is performed. Please report per-item scores, standard deviations, a paired comparison (e.g., Wilcoxon signed-rank or similar) across the 50 items, and a measure of judge agreement.
  3. [Section IV-A and Section IV-B] The dataset and evaluation VQAs are not released, despite SNEI being the paper's central contribution; the website footnote alone does not allow reviewers or readers to verify the 40K/2K statistics, reproduce the fine-tuning, or audit the evaluation items. A dataset paper should release the data, annotation instructions, and code, with a clear license, before the comparative claims can be assessed.
  4. [Section IV-A/IV-B] The experiments do not include a control fine-tuning condition, such as LLaVA fine-tuned on a comparable amount of general VQA or instruction data. Without such a control, it is unclear whether Social-LLaVA's scores reflect the specific content of SNEI or simply the effect of any fine-tuning on the base model, which weakens the paper's conclusion about the dataset's value.
minor comments (6)
  1. [Section IV-B] The text calls the evaluation "qualitative" although it produces numeric scores; clarify that this is a human-scored quantitative evaluation.
  2. [Table I] The number of VQAs per task (Perception, Prediction, CoT, Final Action, Explanation) is not specified; the abstract says 50 VQA total, but it is unclear how these are distributed across tasks and whether each judge scored all 50.
  3. [Fig. 3] The caption says "Phrases in blue indicate accurate reasoning... while highlighted phrases mark instances of hallucination," but the figure does not visibly distinguish these formatting cues; use a legend or explicit callouts.
  4. [Section IV-B] The paper does not report the decoding parameters (e.g., temperature, top-p, number of samples) for GPT-4V, Gemini, or Social-LLaVA, nor whether multiple samples were taken to account for stochasticity.
  5. [Section IV-C] The real-robot experiment reports only a single qualitative demonstration with no success criteria, number of trials, or failure cases; please add these details or clearly mark the figure as an illustrative example.
  6. [General] The manuscript contains inconsistent formatting of the model name ("Social-LLaV A" with a space in several places), and the reference to the project website appears only in the footnote; add a proper data-availability entry.

Circularity Check

1 steps flagged · score 6.0 of 10

The Table I claim that Social-LLaVA outperforms GPT-4V and Gemini is partly circular: the model is fine-tuned on SNEI and the judged VQAs are presented as examples from the same dataset, with no reported held-out split.

  1. fitted input called prediction [Section IV-A (Social-LLaVA fine-tuning) and Section IV-B (Qualitative VQA Evaluation, Table I)]
    "To test our Social-LLaVA model fine-tuned on our SNEI dataset, a large-scale VQA benchmark with a guaranteed performance is required. However, to the best of our knowledge, such a benchmark for social robot navigation does not currently exist. Therefore, we choose to qualitatively analyze the results using human judges. ... In Fig. 3 we showcase one of the examples from our dataset and the answers from the three models. We fine-tune the LLaVA-v1.5-7B on our dataset using Low-rank Adaptation (LoRA), with a batch size of 4 on a single A100 GPU for 15 epochs."

    Social-LLaVA is fine-tuned on SNEI for 15 epochs with no reported train/test split. The qualitative evaluation then judges outputs on examples 'from our dataset'—the same SNEI pool used for fine-tuning. Any VQA drawn from that pool gives Social-LLaVA direct exposure to the gold free-form answer during training, while GPT-4V and Gemini are evaluated zero-shot. The Table I scores therefore measure how well the model reproduces training-set annotations, not whether it generalizes or reasons. Without showing that the 50 judged VQAs are disjoint from the training set, the central comparative claim reduces to a fitted-input evaluation by construction.

full rationale

The paper's central quantitative claim is the Table I comparison: Social-LLaVA outperforms GPT-4V and Gemini on fifteen human-judge scores over 50 VQAs. The model is produced by fine-tuning LLaVA-v1.5-7B on SNEI for 15 epochs, and the evaluation section selects examples 'from our dataset' without reporting any train/test split or showing that the 50 VQAs are disjoint from the fine-tuning pool. Since the same SNEI pool supplies both training labels and judged items, the human scores largely verify how well the model memorized the annotated answers; zero-shot baselines are evaluated without that exposure, so the comparison is confounded. This is the central circular step. The dataset construction (SNEI) and the LoRA fine-tuning recipe are otherwise independent contributions; citations to SCAND and the authors' prior navigation papers are not load-bearing for the claimed comparison. No equations or derivation chain reduce to their own inputs; the circularity is in the evaluation design, not in a mathematical argument. Score 6 reflects that the headline superiority claim is partially forced by the training/evaluation overlap, though the exact overlap is not explicitly shown.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about annotation validity, single-image sufficiency, and human-judge reliability, plus unreleased artifacts. No free parameters in the mathematical sense are fitted to derive an equation; rather, training hyperparameters and evaluation choices are underspecified.

free parameters (4)
  • Evaluation VQA subset (n=50) = not specified
    The 50 tasks used for human evaluation are sampled from SNEI with no stated criterion or held-out status; the choice directly determines the reported scores.
  • LoRA rank and learning rate = not reported
    Fine-tuning configuration for Social-LLaVA is only partially specified (batch size 4, 15 epochs on one A100); LoRA rank and learning rate are omitted, affecting reproducibility.
  • Prompt-tuning examples for GPT-4V and Gemini = three examples
    Hand-chosen examples used to prompt baselines; not controlled across models.
  • Hard-coded action mapping = go straight, turn left, turn right
    Real-robot demonstration maps language descriptions to actions by a fixed rule, not learned.
assumptions (4)
  • domain assumption Single RGB images from SCAND contain enough information to infer pedestrian intent, predict motion, and determine socially compliant high-level actions.
    SNEI annotations and model training are based on single images (Section V-C: 'we currently limit our data annotation to only images'), yet prediction tasks require future information not present in a static frame.
  • domain assumption Human annotators' VQA answers represent valid ground truth for socially compliant robot behavior.
    The dataset is built on subjective social norms; the paper acknowledges ambiguity in social navigation (Section V-B) but treats annotations as labels for fine-tuning and evaluation.
  • domain assumption Average human-judge scores on a 1-5 scale are a reliable measure of VLM output quality.
    Table I reports only means; no inter-rater reliability, variance, or statistical significance is provided (Section IV-B).
  • domain assumption Fine-tuning on SNEI transfers to real-world robot operation.
    Section IV-C deploys the model with a hard-coded language-to-action mapping; no quantitative navigation metrics are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces." pith.science (2026). https://pith.science/paper/JTGL7GUW

@misc{pith2026250109024,
  author       = {Pith},
  title        = {Pith review of: Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTGL7GUW}},
  note         = {Machine review of arXiv:2501.09024}
}
read the original abstract

Most existing social robot navigation techniques either leverage hand-crafted rules or human demonstrations to connect robot perception to socially compliant actions. However, there remains a significant gap in effectively translating perception into socially compliant actions, much like how human reasoning naturally occurs in dynamic environments. Considering the recent success of Vision-Language Models (VLMs), we propose using language to bridge the gap in human-like reasoning between perception and socially aware robot actions. We create a vision-language dataset, Social robot Navigation via Explainable Interactions (SNEI), featuring 40K human-annotated Visual Question Answers (VQAs) based on 2K human-robot social interactions in unstructured, crowded public spaces, spanning perception, prediction, chain-of-thought reasoning, action, and explanation. We fine-tune a VLM, Social-LLaVA, using SNEI to demonstrate the practical application of our dataset. Social-LLaVA outperforms state-of-the-art models like GPT-4V and Gemini, based on the average of fifteen different human-judge scores across 50 VQA. Deployed onboard a mobile robot, Social-LLaVA enables human-like reasoning, marking a promising step toward socially compliant robot navigation in dynamic public spaces through language reasoning.

Figures

Figures reproduced from arXiv: 2501.09024 by the authors.

Figure 1
Figure 1. Bridging perception to socially compliant action [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Social-LLaVA: Proof-of-concept real-world experiment demonstrating the robot’s ability to understand context and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results of our Social-LLaVA model fine-tuned on our [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Logic-Guided Socially-aware Robot Navigation World Model

    cs.RO 2025-10 conditional novelty 6.0 of 10

    NaviWM couples a spatial-temporal world model with a deductive chain-of-thought, formalizing social navigation rules as first-order logic, and reports improved success and lower violation rates in simulated crowded na...

  2. HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation

    cs.RO 2025-08 reject novelty 6.0 of 10

    HALO learns a vision-based navigation reward from human preference rankings on egocentric video, and an IQL policy using it beats several baselines in 10-trial real-world tests.

  3. Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments

    cs.RO 2025-06 conditional novelty 6.0 of 10

    Narrate2Nav uses Barlow Twins alignment to distill language-based reasoning from a large teacher into a small RGB-only navigation model, reporting lower trajectory error and higher goal-reaching success than four baselines.

  4. Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Conditional VLM reasoning triggered by personal-space violations improves social-navigation success by up to 20 points over RL-only baselines while keeping most steps on a fast RL policy.

  5. MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments

    cs.CV 2025-12 reject novelty 5.0 of 10

    MUSON is a curated dataset of egocentric navigation images with chain-of-thought labels on which the main text reports Qwen2.5-VL-3B reaching 0.8625 action accuracy, while the arXiv abstract reports a different 10,110...

Reference graph

Works this paper leans on

50 extracted references · 17 canonical work pages · cited by 5 Pith papers

  1. [1]

    Conflict avoidance in social navigation—a survey,

    R. Mirsky, X. Xiao, J. Hart, and P. Stone, “Conflict avoidance in social navigation—a survey,” ACM Transactions on Human-Robot Interaction, vol. 13, no. 1, pp. 1–36, 2024

    2024

  2. [2]

    Principles and guidelines for evaluating social robot navigation algorithms,

    A. Francis, C. P ´erez-d’Arpino, C. Li, F. Xia, A. Alahi, R. Alami, A. Bera, A. Biswas, J. Biswas, R. Chandra et al. , “Principles and guidelines for evaluating social robot navigation algorithms,” ACM Transactions on Human-Robot Interaction , 2024

    work page 2024

  3. [3]

    Core challenges of social robot navigation: A survey,

    C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Stein- feld, and J. Oh, “Core challenges of social robot navigation: A survey,” ACM Transactions on Human-Robot Interaction , vol. 12, no. 3, pp. 1– 39, 2023

    2023

  4. [4]

    Reciprocal n-body collision avoidance,

    J. Van Den Berg, S. J. Guy, M. Lin, and D. Manocha, “Reciprocal n-body collision avoidance,” in Robotics Research: The 14th Interna- tional Symposium ISRR . Springer, 2011, pp. 3–19

    2011

  5. [5]

    Social force model for pedestrian dynam- ics,

    D. Helbing and P. Molnar, “Social force model for pedestrian dynam- ics,” Physical review E , vol. 51, no. 5, p. 4282, 1995

    1995

  6. [6]

    Social-aware robot navigation in urban environments,

    G. Ferrer, A. Garrell, and A. Sanfeliu, “Social-aware robot navigation in urban environments,” in 2013 European Conference on Mobile Robots. IEEE, 2013, pp. 331–336

    work page 2013

  7. [7]

    Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,

    H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,” IEEE Robotics and Automation Letters , 2022

    work page 2022

  8. [8]

    Toward human-like social robot navigation: A large-scale, multi- modal, social human navigation dataset,

    D. M. Nguyen, M. Nazeri, A. Payandeh, A. Datar, and X. Xiao, “Toward human-like social robot navigation: A large-scale, multi- modal, social human navigation dataset,” in IEEE/RSJ International Conference on Intelligent Robots and Systems , 2023, pp. 7442–7447

    work page 2023

Show all 50 references
  1. [9]

    Sacson: Scalable autonomous control for social navigation,

    N. Hirose, D. Shah, A. Sridhar, and S. Levine, “Sacson: Scalable autonomous control for social navigation,” IEEE Robotics and Au- tomation Letters , 2023

    2023

  2. [10]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”

  3. [11]

    Vanp: Learning where to see for navigation with self-supervised vision-action pre-training,

    M. Nazeri, J. Wang, A. Payandeh, and X. Xiao, “Vanp: Learning where to see for navigation with self-supervised vision-action pre-training,”

  4. [12]

    A protocol for validating social navigation policies,

    S. Pirk, E. Lee, X. Xiao, L. Takayama, A. Francis, and A. Toshev, “A protocol for validating social navigation policies,” arXiv preprint arXiv:2204.05443, 2022

    2022 arXiv

  5. [13]

    Robot companion: A social- force based approach with human awareness-navigation in crowded environments,

    G. Ferrer, A. Garrell, and A. Sanfeliu, “Robot companion: A social- force based approach with human awareness-navigation in crowded environments,” in 2013 IEEE/RSJ International Conference on Intel- ligent Robots and Systems . IEEE, 2013, pp. 1688–1694

    2013

  6. [14]

    Human-robot proxemics: physical and psychological distancing in human-robot interaction,

    J. Mumm and B. Mutlu, “Human-robot proxemics: physical and psychological distancing in human-robot interaction,” in Proceedings of the 6th international conference on Human-robot interaction , 2011, pp. 331–338

    2011

  7. [15]

    Robot navigation in large-scale social maps: An action recognition approach,

    K. Charalampous, I. Kostavelis, and A. Gasteratos, “Robot navigation in large-scale social maps: An action recognition approach,” Expert Systems with Applications , vol. 66, pp. 261–273, 2016

    2016

  8. [16]

    Exploring reflective limitation of behavior cloning in autonomous vehicles,

    M. H. Nazeri and M. Bohlouli, “Exploring reflective limitation of behavior cloning in autonomous vehicles,” in 2021 IEEE International Conference on Data Mining (ICDM) , 2021, pp. 1252–1257

    2021

  9. [17]

    Targeted learning: A hybrid approach to social robot navigation,

    A. H. Raj, Z. Hu, H. Karnan, R. Chandra, A. Payandeh, L. Mao, P. Stone, J. Biswas, and X. Xiao, “Targeted learning: A hybrid approach to social robot navigation,” inIEEE International Conference on Robotics and Automation , 2024

    2024

  10. [18]

    Data-driven hri: Learn- ing social behaviors by example from human–human interaction,

    P. Liu, D. F. Glas, T. Kanda, and H. Ishiguro, “Data-driven hri: Learn- ing social behaviors by example from human–human interaction,” IEEE Transactions on Robotics , vol. 32, no. 4, pp. 988–1008, 2016

    2016

  11. [19]

    Appld: Adaptive planner parameter learning from demonstration,

    X. Xiao, B. Liu, G. Warnell, J. Fink, and P. Stone, “Appld: Adaptive planner parameter learning from demonstration,” IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 4541–4547, 2020

    2020

  12. [20]

    Learning model pre- dictive controllers with real-time attention for real-world navigation,

    X. Xiao, T. Zhang, K. M. Choromanski, T.-W. E. Lee, A. Francis, J. Varley, S. Tu, S. Singh, P. Xu, F. Xia, S. M. Persson, L. Takayama, R. Frostig, J. Tan, C. Parada, and V . Sindhwani, “Learning model pre- dictive controllers with real-time attention for real-world navigation,...

    2022

  13. [21]

    Deep representation learning: Fundamentals, technologies, applications, and open challenges,

    A. Payandeh, K. T. Baghaei, P. Fayyazsanavi, S. B. Ramezani, Z. Chen, and S. Rahimi, “Deep representation learning: Fundamentals, technologies, applications, and open challenges,” IEEE Access, vol. 11, pp. 137 621–137 659, 2023

    2023

  14. [22]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” 2023. [Online]. Available: https://arxiv.org/abs/2305.06500

    2023 arXiv

  15. [23]

    Mmbench: Is your multi-modal model an all-around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin, “Mmbench: Is your multi-modal model an all-around player?” 2024. [Online]. Available: https://arxiv.org/abs/2307.06281

    2024 arXiv

  16. [24]

    Svit: Scaling up visual instruction tuning,

    B. Zhao, B. Wu, M. He, and T. Huang, “Svit: Scaling up visual instruction tuning,” 2023. [Online]. Available: https://arxiv.org/abs/ 2307.04087

    2023 arXiv

  17. [25]

    Visual instruction tuning towards general-purpose multimodal model: A survey,

    J. Huang, J. Zhang, K. Jiang, H. Qiu, and S. Lu, “Visual instruction tuning towards general-purpose multimodal model: A survey,” arXiv preprint arXiv:2312.16602, 2023

    2023 arXiv

  18. [26]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Doll ´ar, “Microsoft coco: Common objects in context,” 2015. [Online]. Available: https://arxiv.org/abs/1405.0312

    2015 arXiv

  19. [27]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark,

    Z. Yin, J. Wang, J. Cao, Z. Shi, D. Liu, M. Li, L. Sheng, L. Bai, X. Huang, Z. Wang, J. Shao, and W. Ouyang, “Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark,” 2023. [Online]. Available: https://arxiv.org/abs/2306. 06687

    2023

  20. [28]

    Allava: Harnessing gpt4v-synthesized data for lite vision-language models,

    G. H. Chen, S. Chen, R. Zhang, J. Chen, X. Wu, Z. Zhang, Z. Chen, J. Li, X. Wan, and B. Wang, “Allava: Harnessing gpt4v-synthesized data for lite vision-language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.11684

    2024 arXiv

  21. [29]

    Drivelm: Driving with graph visual question answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual question answering,” 2024. [Online]. Available: https://arxiv.org/abs/2312.14150

    2024 arXiv

  22. [30]

    Lingoqa: Video question answering for autonomous driving,

    A.-M. Marcu, L. Chen, J. H ¨unermann, A. Karnsund, B. Hanotte, P. Chidananda, S. Nair, V . Badrinarayanan, A. Kendall, J. Shotton, E. Arani, and O. Sinavski, “Lingoqa: Video question answering for autonomous driving,” 2024. [Online]. Available: https://arxiv.org/abs/ 2312.14115

    2024 arXiv

  23. [31]

    Lima: Less is more for alignment,

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy, “Lima: Less is more for alignment,” 2023. [Online]. Available: https://arxiv.org/abs/2305.11206

    2023 arXiv

  24. [32]

    Alpagasus: Training a better alpaca with fewer data,

    L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V . Yadav, Z. Tang, V . Srinivasan, T. Zhou, H. Huang, and H. Jin, “Alpagasus: Training a better alpaca with fewer data,” 2024. [Online]. Available: https://arxiv.org/abs/2307.08701

    2024 arXiv

  25. [33]

    Instruction mining: Instruction data selection for tuning large language models,

    Y . Cao, Y . Kang, C. Wang, and L. Sun, “Instruction mining: Instruction data selection for tuning large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2307.06290

    2024 arXiv

  26. [34]

    Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4,

    L. Wei, Z. Jiang, W. Huang, and L. Sun, “Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4,” 2023. [Online]. Available: https://arxiv.org/abs/2308.12067

    2023 arXiv

  27. [35]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2201.11903

    2023 arXiv

  28. [36]

    J. A. Groeger, Understanding driving: Applying cognitive psychology to a complex everyday task . Routledge, 2013

    2013

  29. [37]

    Marr, Vision : a computational investigation into the human representation and processing of visual information

    D. Marr, Vision : a computational investigation into the human representation and processing of visual information . Cambridge, Mass: The MIT Press, Jul. 2010

    2010

  30. [38]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2024, pp. 14 455–14 465

    2024

  31. [39]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  32. [40]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    2023 arXiv

  33. [41]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”

  34. [42]

    Spatialrgpt: Grounded spatial reasoning in vision language models,

    A.-C. Cheng, H. Yin, Y . Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu, “Spatialrgpt: Grounded spatial reasoning in vision language models,” 2024. [Online]. Available: https://arxiv.org/abs/2406.01584

    2024 arXiv

  35. [43]

    Structured spatial reasoning with open vocabulary object detectors,

    N. Nejatishahidin, M. R. V ongala, and J. Kosecka, “Structured spatial reasoning with open vocabulary object detectors,” 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:273233636

    2024

  36. [44]

    Vision- and-language navigation: A survey of tasks, methods, and future directions,

    J. Gu, E. Stefani, Q. Wu, J. Thomason, and X. Wang, “Vision- and-language navigation: A survey of tasks, methods, and future directions,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) . Association for Compu...

    2022 doi

  37. [45]

    Gpt-driver: Learning to drive with gpt,

    J. Mao, Y . Qian, H. Zhao, and Y . Wang, “Gpt-driver: Learning to drive with gpt,” arXiv preprint arXiv:2310.01415 , 2023

    2023 arXiv

  38. [46]

    Verifiably following complex robot instructions with foundation models,

    B. Quartey, E. Rosen, S. Tellex, and G. Konidaris, “Verifiably following complex robot instructions with foundation models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.11498

    2024 arXiv

  39. [47]

    Vision-and-language navigation today and tomorrow: A survey in the era of foundation models,

    Y . Zhang, Z. Ma, J. Li, Y . Qiao, Z. Wang, J. Chai, Q. Wu, M. Bansal, and P. Kordjamshidi, “Vision-and-language navigation today and tomorrow: A survey in the era of foundation models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.07035

    2024 arXiv

  40. [2021]

    Available: https://arxiv.org/abs/2106.09685

    [Online]. Available: https://arxiv.org/abs/2106.09685

    arXiv

  41. [2023]

    Available: https://arxiv.org/abs/2304.08485

    [Online]. Available: https://arxiv.org/abs/2304.08485

    arXiv

  42. [2024]

    Available: https://arxiv.org/abs/2403.08109

    [Online]. Available: https://arxiv.org/abs/2403.08109

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.