{"id":"0e7ce0f8-ceef-4429-84f4-77dc382ec5ca","arxiv_id":"2608.12515","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Vision-language models classify proxemic danger in egocentric robot images only slightly better than random, except Qwen-VL which detects high-danger scenes with high recall that is not tied to accurate person localization.","lead":"Three open-source vision-language models classified proxemic danger in robot camera images at near random accuracy overall. One model, Qwen-VL, flagged high-danger scenes with high recall under advanced prompting, but its labels were not tied to locating the person, so safety deployment is not yet justified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated manual danger-level labels in §3.1 are the load-bearing assumption; without inter-annotator or distance-threshold validation, the near-random VLM result may be an artifact of label noise.","rationale":"The reader's verdict was CONDITIONAL, and its weakest-assumption analysis identified the same load-bearing concern: the manual four-level labels have no demonstrated reliability. I agree with that identification. This concern is more fundamental than the missing geometric baseline or the small fine-tuning set because it undermines the validity of the dependent variable itself. The reported effect sizes are small (weighted F1 in the range 0.21–0.24 against a 0.25 random baseline), so even modest label noise could shift the conclusion. The manuscript itself flags the risk in §3.1, which counts as honest disclosure but does not remove the need for a reliability check. The conditional verdict remains appropriate: the authors should supply inter-annotator agreement statistics and/or validate the manual labels against distance thresholds derived from JRDB's 3D annotations before the benchmark can be accepted as evidence that VLMs cannot perform proxemic reasoning. A secondary issue — the absence of a simple geometric or person-size baseline — also deserves attention, but the label-validity problem is the first thing to settle because it determines whether the benchmark measures what it claims to measure.","tokens_in":6914,"tokens_out":9571,"duration_ms":98315,"concrete_test":"Have at least three annotators independently re-label a random subset of 200 held-out JRDB images using the exact §3.1 protocol, with no knowledge of the original labels. Compute Cohen's kappa and weighted kappa for the four danger levels. If unweighted kappa is below 0.6 or weighted kappa is below 0.7, treat the ground truth as too noisy to support the reported metrics and re-run the headline evaluation (Table 1) only on the subset of images with unanimous labels; the central near-random conclusion should then be re-examined under this clean-label subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative result — that four-class proxemic reasoning is unsolved — depends entirely on the quality of the manual four-level labels in §3.1. The paper concedes that these labels were assigned 'manually rather than on the basis of precise distance thresholds, which could potentially lead to inaccuracies,' but it reports no inter-annotator agreement, no reliability study, and no validation against JRDB's 3D geometry. If the labels are noisy or systematically biased between adjacent zones (e.g., moderate versus low, or low versus minimum), then every reported accuracy, weighted F1, and recall figure is attenuated toward chance, and the near-random result could be an artifact of the target rather than a statement about VLM capability. The issue is especially acute for the high-danger class, where Qwen's best configuration trades precision (0.182) for recall (0.79); small label errors in that class strongly affect the headline 'unsolved' and the interpretation that a useful safety label has been produced. Without label reliability evidence, the benchmark cannot distinguish 'VLMs cannot assess proxemic risk' from 'this label set is not a reliable proxy for proxemic risk.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether three open-weight vision-language models (InternVL3.5-4B, Qwen2.5-VL-3B, and SmolVLM2-2.2B) can classify egocentric robot images from JRDB into four proxemic danger levels (high, moderate, low, minimum). The authors introduce a 1,243-image labeled dataset, evaluate three prompting strategies and two rounds of QLoRA fine-tuning, and compare against a random baseline. Their main findings are that overall weighted F1 remains close to the random baseline (0.21–0.24) for all configurations, that Qwen with the advanced prompt reaches high-danger recall of 0.79 (at low precision), and that correct danger classification does not correspond to better person-localization IoU. The conclusion is that four-class proxemic reasoning from single-view egocentric frames remains unsolved for current VLMs.","tokens_in":7092,"tokens_out":5405,"duration_ms":44039,"significance":"If the results are robust, the paper provides a useful negative result: it suggests that fine-grained proxemic reasoning is not yet within the reach of open-weight VLMs, and it documents a striking dissociation between classification labels and spatial grounding. The study is reasonably designed for a first pass: it uses open models, a transparent random baseline, multiple prompts, and two fine-tuning stages, and the authors commit to releasing the dataset. The main reasons for caution are the unvalidated manual labels and the absence of statistical inference, which I detail below. The qualitative conclusion is plausible, but the quantitative evidence, as presented, does not fully support the strength of the claims.","major_comments":[{"comment":"The ground truth is the load-bearing assumption. The paper states that labels were assigned manually rather than by precise distance thresholds and concedes potential inaccuracies, but reports no inter-annotator agreement, no reliability study, and no validation against JRDB's available 3D annotations. Since all metrics are computed against these labels, label noise or systematic confusion between adjacent zones (e.g., moderate vs. low) would attenuate every reported accuracy, weighted F1, and recall toward the random baseline. The central negative conclusion therefore cannot be distinguished from 'the label set is an unreliable proxy for proxemic risk' without reliability evidence. Please report Cohen's kappa (or a similar measure) on a re-annotated subset and, if possible, derive distance-based labels from JRDB's 3D human boxes to validate the manual labels.","section":"Section 3.1"},{"comment":"The paper reports point estimates without confidence intervals or significance tests. For example, the claim that fine-tuning yields no gain (ΔwF1 < 0.02) and the claim that correct and incorrect classifications have indistinguishable grounding (mean IoU 0.204 vs. 0.215) rest on differences that may be within sampling noise for a test set on the order of 1,000 images. Because the paper's headline is a null/negative result, this is not a cosmetic issue: bootstrap confidence intervals or paired tests (e.g., McNemar's test for classification, a Wilcoxon test for IoU) are needed to show that the null is not simply due to low statistical power.","section":"Section 4, Tables 1–3; Section 5.2"},{"comment":"The fine-tuning description is not reproducible. The paper specifies r=8, alpha=16, dropout=0.05, and the adapted projections, but omits the number of epochs, learning rate, LR schedule, batch size, optimizer, sequence length, and total compute. The 200-image training set (50 per class) is very small, and the paper itself notes this in §5.3, but no sensitivity analysis is provided. Please report these details and, ideally, results with a larger training set or at least error bars across seeds.","section":"Section 3.4"},{"comment":"The headline high-danger recall of 0.79 is achieved at precision 0.182 (F1 0.296). The paper appropriately cautions that deployability is not demonstrated, but the recall-vs-precision trade-off is not analyzed, e.g., by varying a decision threshold or by reporting a precision-recall curve. Without such analysis, the claim that 'targeted prompting and fine-tuning can improve high-danger detection' is not yet quantitatively grounded, and the practical value of this configuration as a safety trigger remains unclear.","section":"Section 4.3/Table 4; Section 5.1"},{"comment":"The dissociation between classification and grounding is an interesting claim, but it is based on a fragile comparison. Mean IoU for correctly vs incorrectly classified frames (0.204 vs 0.215) and Qwen's true positives vs false negatives in the high-danger class (0.488 vs 0.464) are not shown to be significantly different, and the high rate of IoU=0 predictions (up to 0.68 for SmolVLM) suggests that invalid bounding boxes may dominate the averages. Please report the distribution of IoU (e.g., quartiles), the proportion of valid boxes, and a statistical test before interpreting the result as evidence that classification does not use spatial grounding.","section":"Section 5.2"}],"minor_comments":[{"comment":"The phrase 'the moderate prompt yields the lowest total misalignment score' refers to a metric that is never defined. Please either define 'misalignment' or remove the claim.","section":"Sections 3.3/4.1"},{"comment":"'Tab 3' should be 'Table 3' for consistency.","section":"Section 4.2"},{"comment":"The sentence 'which is may be due to structural limitations' contains a grammatical error; revise to 'which may be due to structural limitations'.","section":"Section 5.3"},{"comment":"Several bibliography entries contain stray numbers appended to URLs (e.g., [4], [11], [14]), indicating a formatting error in the reference list. Please check the journal's reference style.","section":"References"},{"comment":"The exact prompts are not included in the paper or in an appendix. Since prompt design is a central variable, the full prompt texts should be provided for reproducibility.","section":"Section 3.3"},{"comment":"The 'stratified random baseline' is given as 0.25 for accuracy and weighted F1, but with the stated class distribution (17.9/32.7/24.1/25.3), a random classifier choosing according to the class priors would have an expected weighted F1 of about 0.26. Please clarify how the baseline was computed (e.g., uniform random versus stratified random).","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely question and the negative result would be of interest if the methodology were tightened. The central concern is the unvalidated manual ground truth in §3.1: without inter-annotator agreement or a distance-based sanity check, the 'unsolved problem' conclusion is not securely established. The authors should be given the opportunity to add label-reliability evidence and basic statistical inference (CIs, tests). If they cannot provide such evidence, the manuscript would likely be better framed as a pilot study rather than as a definitive negative benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a genuinely useful benchmark with a careful, honest write-up. It builds a proxemic danger dataset from JRDB, evaluates three open-weight VLMs under three prompts and two QLoRA stages, and reports a clean dissociation: Qwen can detect high-danger cases with high recall, but its bounding-box grounding does not correlate with classification correctness. That observation is the real contribution — it suggests that a safety label from a VLM should not be read as spatial understanding.\n\nWhat it does well: the task framing is new, the dataset is a reasonable starting point for embodied AI safety evaluation, and the authors are transparent about many limitations. They compare against a stratified random baseline, report per-class precision/recall/F1 and IoU, and explicitly concede that the best high-recall configuration is not deployable because precision is too low. The fine-tuning details are mostly there, and the related work is fairly cited.\n\nThe soft spots are real and load-bearing. Section 3.1 admits the 1,243 labels were assigned manually without precise distance thresholds, yet there is no inter-annotator agreement study, no validation against JRDB's 3D geometry, and no error bars or significance tests. The 200-image training set is very small, and the code and dataset are not yet released despite promises. Without label reliability evidence, the headline 'four-class proxemic reasoning remains unsolved' could be an artifact of noisy targets rather than a true statement about VLM capability. The stress-test note is right: the near-random results are exactly what you'd get if labels for adjacent zones (moderate/low, low/minimum) were unreliable. Even the grounding dissociation could be partly affected, since 'correct' vs 'incorrect' classifications depend on those same labels.\n\nThat said, the paper does not oversell its conclusions — the discussion acknowledges the mixed picture and the limited deployability. The central qualitative finding about Qwen's behavior is plausible and worth verifying. What's missing is the methodological scaffolding to turn a suggestive result into a solid benchmark claim.\n\nThis paper deserves a serious referee. The task and dataset are new enough that the community would benefit from a corrected, validated version. My recommendation: send it out, but with the clear expectation that the authors either release the annotation protocol with reliability measures or soften the 'unsolved problem' claim accordingly.","headline":"Honest benchmark with a plausible negative result, but the unvalidated manual labels make the headline claim fragile; worth refereeing.","tokens_in":7626,"tokens_out":1508,"would_cite":true,"duration_ms":16753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-weight vision-language models classify egocentric proxemic danger only slightly better than chance, and a correct danger label does not mean the model located the person.","keywords":["embodied AI","proxemic risk","vision-language models","egocentric robot perception","danger classification","spatial grounding","human-robot interaction","fine-tuning"],"falsifier":"Re-label the same 1,243 images using depth-based distance thresholds and have at least two independent annotators label them with the same rubric, then measure inter-annotator agreement and re-run the best model configurations on the consensus subset; if the models remain near random weighted F1 with no grounding advantage, the paper's conclusion is confirmed, but a large jump would mean the near-random scores were an artifact of unreliable targets.","tokens_in":6685,"feed_emoji":"🤖","tokens_out":8617,"duration_ms":70619,"temperature":0.7,"pith_summary":"The paper asks whether vision-language models can measure how close a person is to a robot from a single egocentric image, and whether a correct risk label reflects the model actually seeing that person. It evaluates three open-weight models on 1,243 images labeled with four proxemic danger levels, comparing three prompting strategies and two rounds of QLoRA fine-tuning against a stratified random baseline. The paper concludes that four-class proxemic reasoning from single-view egocentric frames is still unsolved: overall weighted F1 stays near the random baseline, only one model achieves useful high-danger recall, and correct classifications show no better spatial grounding than incorrect ones. This matters because robots sharing space with people need a reliable, cheap way to detect imminent violations of personal space, and the results suggest universal visual understanding cannot yet substitute for geometric or depth-based safety assessment.","feed_headline":"VLMs near chance at judging proxemic danger from robot views","feed_subtitle":"Only one open-weight model caught most high-danger scenes, and correct labels did not mean it saw the person.","key_machinery":"The machinery is a four-class proxemic danger label built on the standard intimate/personal/social/public distance zones, applied to egocentric robot images, with each prediction required to include both a danger level and a bounding box around the relevant person. The bounding box is scored by intersection-over-union (IoU) against the ground-truth person location, and the paper's central comparison cross-tabulates IoU against classification correctness per class and per configuration. Three prompt strategies (simple, moderate, and a chain-of-thought advanced prompt) and two rounds of QLoRA fine-tuning over 200 balanced images serve as the interventions, and the stratified random baseline ($wF1 = 0.25$) is the reference point that defines the claim that the task remains unsolved.","core_discovery":"In the paper's own terms, the discovery is that current open-weight VLMs cannot yet perform fine-grained proxemic reasoning from single egocentric robot frames. Across all models, prompt strategies, and fine-tuning stages, weighted F1 remains in 0.21–0.24 against a stratified random baseline of 0.25; only Qwen-VL reaches substantial recall for the high-danger class, while InternVL and SmolVLM stay near zero on that class. Fine-tuning produces gains below 0.02 in weighted F1 and can even hurt smaller models. The second, more striking finding is a dissociation between classification and grounding: mean IoU is 0.204 for correctly classified frames and 0.215 for incorrectly classified ones, and Qwen's best true-positive high-danger IoU (0.488) barely exceeds its false-negative IoU (0.464), so emitting a useful safety label does not imply that the model attended to the person who creates the risk.","pith_inferences":["A practical safety system could require both a high-danger label and a predicted bounding box whose IoU exceeds a threshold before triggering an alert; this would likely cut false alarms at the cost of some recall on detections the model cannot localize.","The pattern of near-random weighted F1 with high high-danger recall suggests the recall is bought by overpredicting the high-danger class; checking precision-recall curves and calibrating the model's confidence scores would make this explicit.","The manual labels are the main threat to the conclusion; re-labeling the same frames with depth-based distance thresholds and measuring inter-annotator agreement would show whether the near-random scores come from the models or from the target variable itself.","The same benchmark could be run on larger closed-weight models and on short video clips; success there would point to scale or temporal context as the bottleneck, while continued near-random scores would suggest the single-view proxemic task itself is the limitation."],"forward_implications":["No tested configuration supports deploying one of these open-weight VLMs as a standalone proxemic safety trigger: overall classification is near chance, and useful high-danger recall appears in only one model under a specific prompt.","A high-danger label from a VLM should not be treated as evidence that the model has located the responsible person; the IoU analysis shows label correctness and spatial grounding are largely independent.","Prompt complexity is a trade, not a free win: simple prompts favor overall F1 and grounding, while chain-of-thought prompts favor high-danger recall at the expense of precision and other classes, consistent with a formatting tax.","Lightweight QLoRA fine-tuning on a 200-image balanced set does not close the gap, and an additional round can regress small models through catastrophic forgetting.","Safety-critical robot perception should report classification and grounding jointly, because separate accuracy scores can hide a model that labels correctly without seeing the relevant person."],"supporting_citations":[{"why":"It supplies the source egocentric robot images from which the 1,243-image proxemic danger dataset is built.","marker":"[14]"},{"why":"It provides the four proxemic distance zones that define the high, moderate, low, and minimum danger classes.","marker":"[8]"},{"why":"It introduces Qwen2.5-VL, the model that achieves the useful high-danger recall and the strongest grounding.","marker":"[2]"},{"why":"It introduces InternVL, one of the three evaluated open-weight models that stays near the random baseline.","marker":"[3]"},{"why":"It introduces SmolVLM, the smallest evaluated model used as a lightweight lower bound.","marker":"[9]"},{"why":"It provides the QLoRA method used for both fine-tuning rounds.","marker":"[5]"},{"why":"It motivates the advanced chain-of-thought prompt strategy that produces the highest high-danger recall.","marker":"[7]"},{"why":"It names the formatting tax used to explain why complex prompts trade overall F1 for outlier detection.","marker":"[12]"},{"why":"It explains catastrophic forgetting, which the paper invokes for the regression of SmolVLM after the second tuning round.","marker":"[10]"}],"fun_headline_variants":["VLMs fail proxemic danger test, near random","Robot vision models can't judge danger, study finds","Safety labels don't mean VLMs see the person","Qwen-VL tops high-danger recall but still misses grounding","Open VLMs near chance at proxemic risk from robot POV"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the manual labels of 1,243 images into four danger levels being accurate enough to serve as ground truth; the paper itself notes these labels were assigned without precise distance thresholds, so systematic label noise could push every model toward the random baseline regardless of its true ability.","fun_headline_variants_meta":{"raw":{"variants":["VLMs fail proxemic danger test, near random","Robot vision models can't judge danger, study finds","Safety labels don't mean VLMs see the person","Qwen-VL tops high-danger recall but still misses grounding","Open VLMs near chance at proxemic risk from robot POV"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2900,"prompt_tokens":928,"completion_tokens":1972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1891}},"tokens_in":544,"tokens_out":1972,"duration_ms":11698,"temperature":1.0,"reasoning_tokens":1891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:06:33.230221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the same 1,243 images using depth-based distance thresholds and have at least two independent annotators label them with the same rubric, then measure inter-annotator agreement and re-run the best model configurations on the consensus subset; if the models remain near random weighted F1 with no grounding advantage, the paper's conclusion is confirmed, but a large jump would mean the near-random scores were an artifact of unreliable targets.","supporting_citations":[{"cited_title":"Doubleday (1966) 2, 4","cited_arxiv_id":null,"evidence_quote":"It provides the four proxemic distance zones that define the high, moderate, low, and minimum danger classes."},{"cited_title":"co / HuggingFaceTB / SmolVLM-Instruct4","cited_arxiv_id":null,"evidence_quote":"It introduces SmolVLM, the smallest evaluated model used as a lightweight lower bound."}],"review_version":1}