{"id":"be985961-da3b-4208-93e0-adaef1bc6088","arxiv_id":"2501.09024","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 40K-pair vision-language dataset for social navigation and a fine-tuned VLM that reportedly beats GPT-4V and Gemini in human-judged scene reasoning.","lead":"This paper introduces SNEI, a human-annotated dataset of 40,000 visual question-answer pairs for social robot navigation, and fine-tunes a vision-language model, Social-LLaVA, on it. The authors report that Social-LLaVA outperforms GPT-4V and Gemini in human-judged reasoning about crowded scenes, but the evaluation lacks a held-out split and the dataset is not yet released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed outperformance over GPT-4V and Gemini rests on human scoring of 50 VQAs that may come from the same SNEI pool used for LoRA fine-tuning; with no train/test split reported, the comparison can measure memorization rather than reasoning.","rationale":"The paper's stated purpose is a dataset contribution, and it self-labels Social-LLaVA as a proof-of-concept, but the abstract and introduction make a strong comparative claim. That claim's validity depends entirely on whether the 50 evaluation VQAs are outside the training distribution. The text provides no split; the only shown example is from the dataset; and fine-tuning on SNEI is described without any holdout. Under these conditions the most plausible reading is that the evaluation is contaminated. The reader's weakest assumption is the same, and the requested check, a clean held-out split with an overlap audit, would settle it. Other issues, such as missing error bars, hard-coded action grounding, and no released dataset or code, reinforce rejection but are secondary. The human annotation process and the authors' transparency about the open grounding problem are points in favor, but they do not fix the evaluation. I therefore keep the reader's REJECT verdict unchanged.","tokens_in":10228,"tokens_out":3653,"duration_ms":37463,"concrete_test":"Release SNEI with an explicit train/held-out split and check overlap: compute exact or image-level matching between the 50 evaluation VQAs and the fine-tuning subset, e.g., by comparing SCAND frame IDs or image hashes. Then re-run the 15-judge scoring on a held-out set of at least 50 VQAs whose source frames never appear in training, and report per-task means with confidence intervals. If Social-LLaVA does not beat GPT-4V and Gemini on that held-out set, the claimed superiority is an artifact of training/evaluation overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: Social-LLaVA outperforms GPT-4V and Gemini on the average of fifteen human-judge scores across 50 VQA social navigation tasks (Abstract; Section IV-B, Table I). The model is fine-tuned on SNEI (Section IV-A), and the evaluation section states that no external benchmark exists, so human judges are used. The paper never reports that the 50 evaluation VQAs are disjoint from SNEI; Fig. 3 presents one of the examples from the dataset. Since SNEI contains 40K VQAs from 2K SCAND scenarios and the model is fine-tuned for 15 epochs, any evaluation item from the same pool gives Social-LLaVA direct exposure to the input image and, more importantly, to the gold free-form answer text. GPT-4V and Gemini are evaluated zero-shot (with three in-context examples). The margins in Table I, e.g., Final Action 2.77/3.46 versus 4.19, are therefore fully confounded by possible training/evaluation overlap. The paper also omits inter-rater reliability, confidence intervals, and significance tests, but the absence of a held-out split is the decisive flaw because it makes the headline comparison uninterpretable. The real-robot proof-of-concept in Section IV-C uses a hard-coded language-to-action map, so it cannot rescue the navigation claim. The SNEI dataset may still be valuable if released, but the paper's central comparative claim is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SNEI, a human-annotated VQA dataset for social robot navigation built from 2K SCAND scenarios and containing 40K question-answer pairs spanning perception, prediction, chain-of-thought reasoning, final action, and explanation. The authors fine-tune LLaVA-v1.5-7B with LoRA (Social-LLaVA) on SNEI and report that it outperforms GPT-4V and Gemini 1.5 Pro in human-judged scores on 50 VQA tasks. A simple real-robot proof of concept with a hard-coded language-to-action mapping is also presented. The paper's main claims are the value of SNEI as a dataset and the superiority of Social-LLaVA over zero-shot foundation models.","tokens_in":10499,"tokens_out":7400,"duration_ms":72408,"significance":"If the comparative result were established, the paper would make a useful contribution: a human-annotated, explainable VQA dataset for social navigation is currently missing, and the chain-of-thought annotation structure is well matched to downstream language-guided navigation. The authors are appropriately modest about algorithmic novelty, explicitly framing Social-LLaVA as a proof-of-concept baseline. However, the current evaluation does not support the central comparative claim because the evaluation items appear to be drawn from the same SNEI pool used for fine-tuning, no statistical analysis is provided, and the dataset itself is not released. The real-robot experiment is transparently acknowledged to rely on a hard-coded action mapping and therefore does not validate end-to-end navigation.","major_comments":[{"comment":"The comparison against GPT-4V and Gemini is confounded by a possible train/test overlap. Social-LLaVA is fine-tuned on SNEI for 15 epochs (Section IV-A), while the 50 VQAs used for the human evaluation are apparently drawn from the same SNEI pool, since Fig. 3 is described as \"one of the examples from our dataset\" and the paper never reports a held-out split. If any of the evaluation items were used in training, Social-LLaVA would have seen both the images and gold answers, making the margins in Table I (e.g., Final Action 2.77/3.46 vs. 4.19) uninterpretable as evidence of reasoning. Please specify how the 50 tasks were selected, state the exact split, and either release the evaluation set or evaluate on an external benchmark.","section":"Section IV-B, Table I"},{"comment":"The evaluation reports only mean scores over fifteen judges (Table I) and provides no variance, confidence intervals, significance tests, or inter-rater reliability. The abstract and conclusions use the word \"significantly\" (\"significantly higher scores\"), but no statistical test is performed. Please report per-item scores, standard deviations, a paired comparison (e.g., Wilcoxon signed-rank or similar) across the 50 items, and a measure of judge agreement.","section":"Section IV-B"},{"comment":"The dataset and evaluation VQAs are not released, despite SNEI being the paper's central contribution; the website footnote alone does not allow reviewers or readers to verify the 40K/2K statistics, reproduce the fine-tuning, or audit the evaluation items. A dataset paper should release the data, annotation instructions, and code, with a clear license, before the comparative claims can be assessed.","section":"Section IV-A and Section IV-B"},{"comment":"The experiments do not include a control fine-tuning condition, such as LLaVA fine-tuned on a comparable amount of general VQA or instruction data. Without such a control, it is unclear whether Social-LLaVA's scores reflect the specific content of SNEI or simply the effect of any fine-tuning on the base model, which weakens the paper's conclusion about the dataset's value.","section":"Section IV-A/IV-B"}],"minor_comments":[{"comment":"The text calls the evaluation \"qualitative\" although it produces numeric scores; clarify that this is a human-scored quantitative evaluation.","section":"Section IV-B"},{"comment":"The number of VQAs per task (Perception, Prediction, CoT, Final Action, Explanation) is not specified; the abstract says 50 VQA total, but it is unclear how these are distributed across tasks and whether each judge scored all 50.","section":"Table I"},{"comment":"The caption says \"Phrases in blue indicate accurate reasoning... while highlighted phrases mark instances of hallucination,\" but the figure does not visibly distinguish these formatting cues; use a legend or explicit callouts.","section":"Fig. 3"},{"comment":"The paper does not report the decoding parameters (e.g., temperature, top-p, number of samples) for GPT-4V, Gemini, or Social-LLaVA, nor whether multiple samples were taken to account for stochasticity.","section":"Section IV-B"},{"comment":"The real-robot experiment reports only a single qualitative demonstration with no success criteria, number of trials, or failure cases; please add these details or clearly mark the figure as an illustrative example.","section":"Section IV-C"},{"comment":"The manuscript contains inconsistent formatting of the model name (\"Social-LLaV A\" with a space in several places), and the reference to the project website appears only in the footnote; add a proper data-availability entry.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The issues are fixable in principle, but if the authors cannot demonstrate a held-out evaluation or provide the dataset, the paper should not be accepted. The current submission gives no way for a reader to test the central comparison, and the dataset release is essential for a dataset paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's actually new: SNEI is the first human-annotated VQA dataset for social robot navigation, with 40K items spanning perception, prediction, chain-of-thought reasoning, action, and explanation. That is a real contribution, and the annotation structure is thoughtful. The paper also explicitly disclaims algorithmic novelty and treats Social-LLaVA as a baseline, which is honest.\n\nThe trouble is the evaluation. The headline result—Social-LLaVA beats GPT-4V and Gemini on average scores from 15 human judges over 50 VQA tasks—is not interpretable as stated. The paper fine-tunes on SNEI for 15 epochs and then evaluates on 50 tasks from the same distribution, but never states that those 50 are disjoint from training. If they overlap, the comparison measures memorization, not reasoning. That is load-bearing, not stylistic. There are also no error bars or significance tests, and the 50-item sample is small.\n\nThe paper is transparent about some limitations: it acknowledges that language-to-action grounding remains open and that the real-robot demo uses a hard-coded mapping. Good. But that means the real-robot section doesn't provide evidence of navigation improvement either.\n\nThe dataset itself is not released, which blocks independent verification. If SNEI ships, it could become a useful benchmark. As it stands, the central comparative claim needs a proper held-out split, more evaluation items, and ideally inter-rater agreement. The authors should also release data and code.\n\nBottom line: this is a plausible dataset paper with an unsupported comparison. Worth reading for the data construction, but the empirical claims should not be taken at face value. I would send it to review with the expectation that the evaluation be redone on a clean split.","headline":"The SNEI dataset is a genuinely useful new resource, but the claimed outperformance over GPT-4V and Gemini is unsubstantiated because the evaluation rests on 50 items that may come from the training pool.","tokens_in":11101,"tokens_out":1737,"would_cite":false,"duration_ms":17584,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning a vision-language model on 40K human-annotated questions about crowded social scenes produces human-like reasoning about what a robot should do, and that the result outscores GPT-4V and Gemini in…","keywords":["social robot navigation","vision-language models","visual question answering","chain-of-thought reasoning","human-robot interaction","social navigation dataset","explainable AI","crowded environments"],"falsifier":"Retrain Social-LLaVA with every image used in the fifty-question human evaluation held out of the SNEI pool, and rerun the same fifteen-judge scoring; if its average scores fall to the level of zero-shot GPT-4V and Gemini, the reported superiority is memorization, not learned reasoning.","tokens_in":9989,"feed_emoji":"🤖","tokens_out":9858,"duration_ms":85769,"temperature":0.7,"pith_summary":"This paper sets out to show that language can be the missing link between what a social robot sees and how it should act in crowded human spaces. To that end the authors built SNEI, a dataset of more than 40,000 human-annotated visual question-answer pairs drawn from 2,000 human-robot interactions, organized into perception, prediction, chain-of-thought reasoning, final action, and explanation. They fine-tuned a vision-language model, Social-LLaVA, on SNEI and report that it outperforms GPT-4V and Gemini 1.5 Pro across fifteen human-judge scores on fifty social-navigation questions. A proof-of-concept deployment maps its language outputs to simple robot actions. If the claim holds, social navigation can move from hand-coded rules toward explainable, language-driven reasoning.","feed_headline":"Fine-tuned AI model outscores GPT-4V and Gemini at social navigation","feed_subtitle":"A 40K-question human-annotated dataset teaches robots chain-of-thought reasoning in crowded spaces","key_machinery":"The load-bearing mechanism is SNEI, a vision-language dataset of more than 40,000 human-annotated visual question-answer pairs built from 2,000 scenarios selected from the SCAND social-navigation demonstration corpus. Each annotation is organized into five stages — perception, prediction, chain-of-thought reasoning, final action, and explanation — so a model trained on it must verbalize the why before the action. The paper's model, Social-LLaVA, is LLaVA-v1.5-7B fine-tuned with LoRA on this dataset; the five-stage language chain, not a new planner or controller, is what carries the argument from image to high-level action.","core_discovery":"The central claim, stated on the paper's own terms, is that general-purpose vision-language models fail at social navigation because their training data lacks the spatial and social common sense of unstructured, crowded environments, and that a compact, fully human-annotated dataset of social-navigation VQAs closes this gap. Social-LLaVA, which is LLaVA-v1.5-7B fine-tuned with LoRA on SNEI, outputs a five-stage chain: it describes what the robot perceives, predicts what people will do, reasons about the safest response, names a high-level action such as 'stop and wait for clear path,' and explains its choice. In the paper's human evaluation, this fine-tuned model scored between 3.95 and 4.19 across the five stages, ahead of GPT-4V and Gemini 1.5 Pro's 2.77 to 3.87 range, and the authors report that the larger closed models produced hallucinations and socially blocking suggestions. The contribution is framed as the dataset rather than algorithmic novelty.","pith_inferences":["A natural next experiment is to hold out all fifty evaluation images during fine-tuning and rerun the human scoring; without that split, the comparison to zero-shot models cannot distinguish learned reasoning from memorized annotations.","Because SNEI annotates single images, extending the same five-stage annotation to short video clips would test whether prediction and reasoning improve with temporal context; the paper identifies video data as future work.","The real-robot demonstration maps language to actions with a hard-coded table, so the social-compliance claim currently depends on that fixed mapping; a learned grounding module would make the pipeline self-contained.","Re-annotating or testing in a different country or culture would reveal how much of what SNEI encodes is a universal social norm rather than a local convention; the paper acknowledges cultural variability as an open issue."],"forward_implications":["Social-LLaVA can generate human-readable explanations of navigation decisions, making the robot's behavior inspectable rather than a hidden planner output.","The SNEI five-stage answer format gives the social-navigation field a standardized benchmark for comparing perception, prediction, reasoning, action, and explanation quality separately.","A VLM fine-tuned on SNEI can propose safe high-level actions in crowded scenes without hand-crafted behavioral rules, based on the reported comparison against general-purpose models.","The proof-of-concept robot experiment shows that high-level language instructions can be mapped to simple actions such as going straight, turning left, and turning right.","If the human-judge scores generalize, robots can use chain-of-thought language reasoning in ambiguous social cases, such as not interrupting a conversation, where obstacle-avoidance-only policies fail."],"supporting_citations":[{"why":"Supplies the 2,000 human-robot interaction scenarios from which all SNEI VQA annotations are drawn.","marker":"[7]"},{"why":"Provides the LLaVA-v1.5-7B architecture and pretrained vision-language weights that Social-LLaVA fine-tunes.","marker":"[10]"},{"why":"Inspires the perception-prediction-planning language structure used for SNEI's chain-of-thought VQAs.","marker":"[29]"},{"why":"Defines chain-of-thought prompting, the reasoning format that SNEI annotations follow.","marker":"[35]"},{"why":"Supports the claim that general VLM training data limits spatial reasoning, motivating a social-navigation-specific dataset.","marker":"[38]"},{"why":"GPT-4V is one of the two zero-shot baselines that Social-LLaVA is compared against in the human evaluation.","marker":"[39]"},{"why":"Gemini 1.5 Pro is the other zero-shot baseline in the human evaluation.","marker":"[40]"},{"why":"Describes LoRA, the low-rank adaptation method used to fine-tune Social-LLaVA on SNEI.","marker":"[41]"},{"why":"Supports the design choice of a small, fully human-annotated dataset over semi-automated collection.","marker":"[31]"}],"fun_headline_variants":["Social-LLaVA: 40K crowdsourced Q&As teach robots to reason like humans","Fine-tuned 7B VLM beats GPT-4V and Gemini in social robot navigation","How a 7B model with 40K human VQAs outscores giants in social spaces","Social-LLaVA: 40K human-annotated Q&As give robots chain-of-thought navigation","Outperforming GPT-4V: Social-LLaVA's 40K dataset is the key"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the fifty human-judged questions were not part of the fine-tuning set, but the paper reports no train/test split to guarantee that.","fun_headline_variants_meta":{"raw":{"variants":["Social-LLaVA: 40K crowdsourced Q&As teach robots to reason like humans","Fine-tuned 7B VLM beats GPT-4V and Gemini in social robot navigation","How a 7B model with 40K human VQAs outscores giants in social spaces","Social-LLaVA: 40K human-annotated Q&As give robots chain-of-thought navigation","Outperforming GPT-4V: Social-LLaVA's 40K dataset is the key"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":3168,"prompt_tokens":985,"completion_tokens":2183,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2054}},"tokens_in":601,"tokens_out":2183,"duration_ms":14057,"temperature":1.0,"reasoning_tokens":2054,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:56:40.600698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain Social-LLaVA with every image used in the fifty-question human evaluation held out of the SNEI pool, and rerun the same fifteen-judge scoring; if its average scores fall to the level of zero-shot GPT-4V and Gemini, the reported superiority is memorization, not learned reasoning.","supporting_citations":[{"cited_title":"Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,","cited_arxiv_id":null,"evidence_quote":"Supplies the 2,000 human-robot interaction scenarios from which all SNEI VQA annotations are drawn."},{"cited_title":"Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that general VLM training data limits spatial reasoning, motivating a social-navigation-specific dataset."}],"review_version":1}