{"id":"d3cfb9f0-4c53-42e4-b14e-4e9a6bf5e419","arxiv_id":"2512.22867","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MUSON is a curated dataset of egocentric navigation images with chain-of-thought labels on which the main text reports Qwen2.5-VL-3B reaching 0.8625 action accuracy, while the arXiv abstract reports a different 10,110-sample dataset with Qwen3-VL-8B at 0.7765.","lead":"The paper introduces MUSON, a human-annotated dataset of egocentric indoor and street images with five-step reasoning labels for choosing safe navigation actions, plus benchmark results for small vision-language models. A generalist should look because the dataset is proposed as a reusable training-and-evaluation resource for making lightweight robot navigation socially compliant and explainable — but the submission's two abstracts report irreconcilably different dataset size","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II's headline numbers (Acc 0.8625, CR 0.0688) are reported without any documented train/test split; after 30 epochs of SFT on 800 samples they may be in-sample and cannot support the claim that MUSON is a reusable benchmark.","rationale":"The strongest claim is that MUSON is an effective and reusable benchmark based on Qwen2.5-VL-3B's 0.8625 action accuracy and 0.0688 CR after SFT. For that to be true, the accuracy must be measured on data not seen during training. The paper gives no such split. Section V only gives training hyperparameters, and Table II gives aggregate metrics. Thirty epochs on an 800-frame dataset is a classic memorization regime. My concrete test would force the authors to expose the split or rerun cross-validated; if the held-out numbers drop, the central claim is refuted. I agree with the reader's weakest assumption. The abstract/metadata inconsistency (10,110 vs 800; ten vs four models) is real and must be fixed, but it is a consistency/audit problem. The missing split is the deeper scientific flaw because it makes even a corrected paper unverifiable. I still credit the authors for a clear data-collection/annotation design and a reasonable six-action space; those could form a useful artifact after revision. No independent code or machine-checked support is provided, so the burden remains on the reported experiments.","tokens_in":11248,"tokens_out":5100,"duration_ms":52182,"concrete_test":"Obtain the exact train/test split and evaluation script from the authors; if no split exists, rerun Qwen2.5-VL-3B with the same hyperparameters under 5-fold cross-validation (or a fixed 80/10/10 split) on the same 800 frames and recompute Table II. If held-out accuracy is materially below 0.8625 or collision rate above 0.0688, the benchmark claim fails to support generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section V-A ('Implementation Details') the paper trains with full-parameter SFT for 30 epochs (batch 32, lr 2e-5) and then Table II reports Acc/Macro-F1/CR for four SVLMs 'on MUSON dataset'. Nowhere in the paper is a train/validation/test split specified, nor an evaluation protocol stating which frames are held out. With only 800 curated samples, 30 epochs is ample to memorize labels; if the metrics in Table II were computed on the training set, they measure fitting, not socially compliant generalization. The core claim—that fine-tuning on MUSON yields strong reusable benchmark performance (Qwen2.5-VL-3B, Acc 0.8625, CR 0.0688)—therefore rests on an unstated and unverified assumption. The phrase 'strongest overall fitting performance' in Section V-B reinforces the worry that in-sample fit is being reported. This is not a disagreement with the field's consensus or a matter of annotation taste; it is a missing experimental control that any learning benchmark needs. The dataset may be useful and the curation ideas sound, but without this control the headline conclusion is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MUSON, an 800-frame egocentric dataset for short-horizon socially compliant navigation, with a five-step chain-of-thought annotation schema (perception, prediction, reasoning, action, explanation), a six-action discrete decision space, and a curated action distribution intended to mitigate long-tail bias while preserving safety-critical corner cases. The dataset is derived from MuSoHu and is accompanied by a Human-AI-Human annotation pipeline. Four small VLMs are fine-tuned on MUSON; the authors report that Qwen2.5-VL-3B achieves the best decision accuracy (0.8625) and lowest collision rate (0.0688), and that CoT supervision helps on MUSON but hurts on SNEI. The paper claims MUSON is an effective and reusable benchmark for socially compliant navigation.","tokens_in":11498,"tokens_out":4237,"duration_ms":45429,"significance":"If the claims hold, MUSON would be a useful community resource: its structured CoT annotations, balanced action distribution, and explicit modeling of both dynamic and static constraints address real limitations of existing social-navigation datasets such as SNEI. The proposed collision-rate metric and the comparison of several small VLMs are also practically relevant, especially for lightweight deployed systems. The dataset release and the public benchmark baselines are strengths. However, the central empirical conclusion — that fine-tuning on MUSON yields strong, socially compliant decisions — is currently not supported by a clearly documented evaluation protocol, and the paper contains unresolved internal inconsistencies about dataset scale, model set, and headline results. The contribution is therefore conditional on substantial corrections.","major_comments":[{"comment":"The experimental setup states 30 epochs of full-parameter SFT with global batch size 32 on 800 samples, but no train/validation/test split or evaluation protocol is defined anywhere. With 800 curated samples and 30 epochs, the reported Qwen2.5-VL-3B Acc=0.8625, CR=0.0688 may be in-sample fitting. The phrase 'strongest overall fitting performance' in §V-B reinforces this concern. A benchmark claim requires held-out or cross-validated metrics; without this, the central reusable-benchmark conclusion is unsupported.","section":"§V-A and Table II"},{"comment":"The body of the paper (Abstract, §III-A, Table I, Table II, Conclusion) consistently describes an 800-frame dataset with Qwen2.5-VL-3B as best (Acc 0.8625, CR 0.0688), while the arXiv metadata abstract states MUSON contains 10,110 samples, benchmarks ten small-to-medium VLMs, and reports Qwen3-VL-8B with Acc 0.7765 and CR 0.0609 as best. These are irreconcilable statements about dataset scale, model set, and headline results. The authors must state which version is correct and provide a single consistent set of numbers, or the paper cannot be evaluated.","section":"Abstract vs. body"},{"comment":"AI-Assisted Quality Control uses a TinyLLaVA model 'trained on MUSON' to flag samples whose inferred reasoning or decisions deviate from human annotations, and the same model family is later benchmarked on MUSON. This creates a self-referential QC loop: the auditor is fitted to the labels it is asked to validate. Although final arbitration is human, no inter-annotator agreement, number of flagged samples, or distribution of human-AI disagreements is reported. Without such statistics, the reliability of the ground-truth labels — on which all benchmark numbers rest — is not established.","section":"§III-C"},{"comment":"The comparison with SNEI under Action-Only and CoT settings lacks essential details. It is unclear whether SNEI was split into train/test, how many SNEI samples were used, and whether the same number of epochs, prompts, and evaluation metrics were used. The claim that CoT 'degrades performance on SNEI' (0.5833→0.4667) is therefore difficult to interpret; it could reflect insufficient data, different annotation formats, or evaluation on training frames rather than annotation inconsistency in SNEI.","section":"§V-B and Table III"}],"minor_comments":[{"comment":"The release URL is inconsistent: the abstract mentions HuggingFace while the contribution section mentions a GitHub release. Please provide a single, stable public URL.","section":"§I/Abstract"},{"comment":"The collision rate treats 'turn left' and 'turn right' as avoidance actions grouped with 'stop.' This is nonstandard and should be justified, or the metric should separate true stopping from turning maneuvers.","section":"§IV-A, Eq. (1)"},{"comment":"The action balancing thresholds (56.6% move-forward, 118 oblique samples, 32 turn samples) are presented as physically grounded, but no sensitivity analysis is given. State whether the reported benchmark conclusions are robust to reasonable variations of these targets.","section":"§III-B"},{"comment":"Only the TinyLLaVA fine-tuning loss/gradient curves are shown. Label the figure accordingly and indicate whether the other models showed similar convergence dynamics.","section":"Fig. 3"},{"comment":"The header 'Five-Turn Conversation' is used interchangeably with 'CoT'; align the terminology with the five-step CoT annotation schema to avoid confusion.","section":"Table III"}],"recommendation":"major_revision","confidential_remarks":"The internal inconsistency between the arXiv metadata abstract and the body suggests the paper may have been submitted with an outdated or incorrect abstract. This should be resolved before any further review. The missing train/test split is the most serious technical concern: it is not a stylistic issue but a missing experimental control for a benchmark paper. The dataset curation and annotation design are interesting and potentially valuable, so I do not recommend rejection if the authors can provide a clean held-out evaluation and reconcile the conflicting statements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me save you some time: the dataset itself looks like a reasonable curation effort, but the benchmark claim sits on sand. The paper reports accuracies after 30 epochs of full SFT on 800 samples with no train/test split or evaluation protocol anywhere in Section V. That's not a minor omission; it means Table II could be in-sample fitting, and the phrase 'strongest overall fitting performance' in V-B does not reassure me. On top of that, the arXiv metadata abstract says 10,110 samples, ten models, and Qwen3-VL-8B at 0.7765, while the paper's own abstract says 800 samples, four models, Qwen2.5-VL-3B at 0.8625. The paper does not have one consistent central claim.\n\nWhat's genuinely new: 800 frames from MuSoHu re-annotated with a five-step CoT (perception, prediction, reasoning, action, explanation), a deliberately balanced six-action space with left/right symmetry, and a collision-rate metric that treats aggressive errors as more costly than conservative ones. That last idea is sensible for short-horizon social navigation. The comparison in Table III — CoT hurts on SNEI but helps on MUSON — is a nice sanity check, assuming the underlying splits are honest. The curation pipeline (Human-AI-Human) is transparent, though the AI auditor is trained on MUSON itself, so its flagging is circular to some degree; final arbitration is human, which mitigates it. No inter-annotator agreement is reported, so 'socially compliant' ground truth rests on two annotators plus one expert with no quantified consistency check.\n\nProportion: the core dataset idea is fine and the paper is clearly written. But the missing split is load-bearing. If the authors add a proper train/validation/test protocol, release the split and code, reconcile the abstracts, and report agreement statistics, this could be a useful resource for the niche community evaluating lightweight VLMs for social navigation. As is, the central claim is unverified.\n\nI'd send it to peer review rather than desk reject — the issues are fixable and the artifact is new — but I'd tell the referee to focus on the evaluation protocol.","headline":"The dataset curation is sensible, but the benchmark numbers are unsupported because there's no train/test split, and the arXiv metadata abstract contradicts the paper's own abstract.","tokens_in":12103,"tokens_out":3140,"would_cite":false,"duration_ms":29887,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MUSON, a curated 800-frame dataset with five-step reasoning annotations, lets a three-billion-parameter vision-language model reach 86.25 percent decision accuracy for socially compliant navigation.","keywords":["socially compliant navigation","vision-language models","chain-of-thought","multimodal dataset","action distribution","egocentric perception","benchmark","fine-tuning"],"falsifier":"Re-run the fine-tuning and evaluation with an explicit, documented train/test split (e.g., 80/20 stratified by scene and action); if accuracy drops to near the untuned baseline or collision rate rises above 0.2, the benchmark's central generalization claim is refuted.","tokens_in":11078,"feed_emoji":"🤖","tokens_out":4673,"duration_ms":44410,"temperature":0.7,"pith_summary":"This paper introduces MUSON, a reasoning-oriented multimodal dataset for short-horizon social navigation, and argues that its design—structured five-step chain-of-thought annotations, a balanced six-action space, and physics-aware curation—is what lets small vision-language models learn socially compliant decisions. The central claim is that fine-tuning on MUSON lifts lightweight models to high decision accuracy (86.25% for Qwen2.5-VL-3B) with low collision rate (6.88%), demonstrating that the dataset is an effective, reusable benchmark. The paper also argues that the quality of CoT annotations matters: the same model trained with CoT improves on MUSON but degrades on an existing dataset with inconsistent annotations. A sympathetic reader would care because it offers a cheap, publicly available route to making small, deployable navigation models safer and more interpretable.","feed_headline":"800-frame dataset lifts small AI models to 86% navigation accuracy","feed_subtitle":"Five-step reasoning labels and balanced actions let lightweight models plan safe pedestrian interactions.","key_machinery":"MUSON itself is the key machinery: a set of 800 curated egocentric frames with (1) a five-step chain-of-thought annotation schema that forces the model to parse perception, prediction, reasoning, action, and explanation in order; (2) a standardized six-action discrete space balanced by physical priors (56.6% forward) and geometric symmetry (equal left/right counts); and (3) a Human-AI-Human verification pipeline that arbitrates disagreements. The work it does is to convert implicit social norms into explicit, structured supervisory signals that small VLMs can absorb during fine-tuning.","core_discovery":"The discovery is a dataset-plus-benchmark combination: 800 egocentric frames from real indoor and outdoor scenes, each annotated with a five-step chain of thought (perception, prediction, reasoning, action, explanation) in an ego-centric reference frame and a deliberately balanced distribution over six discrete actions—move forward, forward left, forward right, stop, turn left, turn right. The paper shows that after full-parameter supervised fine-tuning on this data, even a 3B-parameter vision-language model can reach 86.25% decision accuracy and a 6.88% collision rate, outperforming other small VLMs in the study. The stronger conceptual claim is that the dataset's geometry-aware action bala","pith_inferences":["If the reported accuracy holds on a true held-out split, MUSON-style datasets could become a standard pre-training step for lightweight embodied models, reducing the need for large-scale instruction data.","The geometric balancing idea—equal left/right counts to defeat positional bias—could transfer to other egocentric decision benchmarks in driving and manipulation.","The Human-AI-Human annotation pipeline could be reused, but without inter-annotator agreement numbers, the ground-truth 'socially compliant' label remains subjective; measuring agreement is a cheap next step.","The paper's front abstract and body disagree on the best model (Qwen3-VL-8B vs Qwen2.5-VL-3B); this discrepancy should be resolved before relying on the headline numbers."],"forward_implications":["Fine-tuned lightweight VLMs can make socially compliant decisions from single egocentric frames, without long-horizon goals or maps.","CoT supervision helps small models when annotations are consistent, but hurts when they are inconsistent, implying data quality is a bottleneck for reasoning-oriented navigation.","Balancing the action distribution to preserve rare safety-critical behaviors (stop, turns) lets models learn those corner cases without sacrificing forward momentum.","The four evaluation tasks (action planning, scene perception, CoT reasoning, explanation) provide a multidimensional protocol for future social-navigation benchmarks."],"fun_headline_variants":["10k egocentric samples teach small VLMs safe urban navigation","Small VLM hits 86% accuracy on social navigation with new dataset","MUSON dataset: five-step reasoning for robots that respect pedestrians","Lightweight models learn to avoid pedestrians via chain-of-thought dataset"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported accuracy and collision rate are only meaningful if they were measured on frames the model never saw during the 30-epoch fine-tuning; the paper never defines a train/test split, so the numbers could reflect memorization instead of generalization.","fun_headline_variants_meta":{"raw":{"variants":["10k egocentric samples teach small VLMs safe urban navigation","Small VLM hits 86% accuracy on social navigation with new dataset","MUSON dataset: five-step reasoning for robots that respect pedestrians","Lightweight models learn to avoid pedestrians via chain-of-thought dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3265,"prompt_tokens":799,"completion_tokens":2466,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":2391}},"tokens_in":543,"tokens_out":2466,"duration_ms":16712,"temperature":1.0,"reasoning_tokens":2391,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T13:46:55.159333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the fine-tuning and evaluation with an explicit, documented train/test split (e.g., 80/20 stratified by scene and action); if accuracy drops to near the untuned baseline or collision rate rises above 0.2, the benchmark's central generalization claim is refuted.","supporting_citations":[],"review_version":1}