Pith. sign in

REVIEW 5 major objections 6 minor 21 references

Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Open-weight vision-language models classify egocentric proxemic danger only slightly better than chance, and a correct danger label does not mean the model located the person.

desk verdict Honest benchmark with a plausible negative result, but the unvalidated manual labels make the headline claim fragile; worth refereeing. read the letter →

arxiv 2608.12515 v1 pith:YB4DSTEF submitted 2026-08-12 cs.CV cs.RO

Vladyslava Rudas , Dmytro Kuzmenko This is my paper · ORCID
classification cs.CVcs.RO
keywords embodiedAIproxemicriskvision-languagemodelsegocentricrobotperceptiondangerclassificationspatialgroundinghuman-robotinteractionfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether vision-language models can measure how close a person is to a robot from a single egocentric image, and whether a correct risk label reflects the model actually seeing that person. It evaluates three open-weight models on 1,243 images labeled with four proxemic danger levels, comparing three prompting strategies and two rounds of QLoRA fine-tuning against a stratified random baseline. The paper concludes that four-class proxemic reasoning from single-view egocentric frames is still unsolved: overall weighted F1 stays near the random baseline, only one model achieves useful high-danger recall, and correct classifications show no better spatial grounding than incorrect ones. This matters because robots sharing space with people need a reliable, cheap way to detect imminent violations of personal space, and the results suggest universal visual understanding cannot yet substitute for geometric or depth-based safety assessment.

What carries the argument

The machinery is a four-class proxemic danger label built on the standard intimate/personal/social/public distance zones, applied to egocentric robot images, with each prediction required to include both a danger level and a bounding box around the relevant person. The bounding box is scored by intersection-over-union (IoU) against the ground-truth person location, and the paper's central comparison cross-tabulates IoU against classification correctness per class and per configuration. Three prompt strategies (simple, moderate, and a chain-of-thought advanced prompt) and two rounds of QLoRA fine-tuning over 200 balanced images serve as the interventions, and the stratified random baseline ($wF1 = 0.25$) is the reference point that defines the claim that the task remains unsolved.

What would settle it

Re-label the same 1,243 images using depth-based distance thresholds and have at least two independent annotators label them with the same rubric, then measure inter-annotator agreement and re-run the best model configurations on the consensus subset; if the models remain near random weighted F1 with no grounding advantage, the paper's conclusion is confirmed, but a large jump would mean the near-random scores were an artifact of unreliable targets.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that current open-weight VLMs cannot yet perform fine-grained proxemic reasoning from single egocentric robot frames. Across all models, prompt strategies, and fine-tuning stages, weighted F1 remains in 0.21–0.24 against a stratified random baseline of 0.25; only Qwen-VL reaches substantial recall for the high-danger class, while InternVL and SmolVLM stay near zero on that class. Fine-tuning produces gains below 0.02 in weighted F1 and can even hurt smaller models. The second, more striking finding is a dissociation between classification and grounding: mean IoU is 0.204 for correctly classified frames and 0.215 for incorrectly classified ones, and Qwen's best true-positive high-danger IoU (0.488) barely exceeds its false-negative IoU (0.464), so emitting a useful safety label does not imply that the model attended to the person who creates the risk.

Load-bearing premise

The conclusion rests on the manual labels of 1,243 images into four danger levels being accurate enough to serve as ground truth; the paper itself notes these labels were assigned without precise distance thresholds, so systematic label noise could push every model toward the random baseline regardless of its true ability.

Editorial extensions

If this is right

  • No tested configuration supports deploying one of these open-weight VLMs as a standalone proxemic safety trigger: overall classification is near chance, and useful high-danger recall appears in only one model under a specific prompt.
  • A high-danger label from a VLM should not be treated as evidence that the model has located the responsible person; the IoU analysis shows label correctness and spatial grounding are largely independent.
  • Prompt complexity is a trade, not a free win: simple prompts favor overall F1 and grounding, while chain-of-thought prompts favor high-danger recall at the expense of precision and other classes, consistent with a formatting tax.
  • Lightweight QLoRA fine-tuning on a 200-image balanced set does not close the gap, and an additional round can regress small models through catastrophic forgetting.
  • Safety-critical robot perception should report classification and grounding jointly, because separate accuracy scores can hide a model that labels correctly without seeing the relevant person.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical safety system could require both a high-danger label and a predicted bounding box whose IoU exceeds a threshold before triggering an alert; this would likely cut false alarms at the cost of some recall on detections the model cannot localize.
  • The pattern of near-random weighted F1 with high high-danger recall suggests the recall is bought by overpredicting the high-danger class; checking precision-recall curves and calibrating the model's confidence scores would make this explicit.
  • The manual labels are the main threat to the conclusion; re-labeling the same frames with depth-based distance thresholds and measuring inter-annotator agreement would show whether the near-random scores come from the models or from the target variable itself.
  • The same benchmark could be run on larger closed-weight models and on short video clips; success there would point to scale or temporal context as the bottleneck, while continued near-random scores would suggest the single-view proxemic task itself is the limitation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper investigates whether three open-weight vision-language models (InternVL3.5-4B, Qwen2.5-VL-3B, and SmolVLM2-2.2B) can classify egocentric robot images from JRDB into four proxemic danger levels (high, moderate, low, minimum). The authors introduce a 1,243-image labeled dataset, evaluate three prompting strategies and two rounds of QLoRA fine-tuning, and compare against a random baseline. Their main findings are that overall weighted F1 remains close to the random baseline (0.21–0.24) for all configurations, that Qwen with the advanced prompt reaches high-danger recall of 0.79 (at low precision), and that correct danger classification does not correspond to better person-localization IoU. The conclusion is that four-class proxemic reasoning from single-view egocentric frames remains unsolved for current VLMs.

Significance. If the results are robust, the paper provides a useful negative result: it suggests that fine-grained proxemic reasoning is not yet within the reach of open-weight VLMs, and it documents a striking dissociation between classification labels and spatial grounding. The study is reasonably designed for a first pass: it uses open models, a transparent random baseline, multiple prompts, and two fine-tuning stages, and the authors commit to releasing the dataset. The main reasons for caution are the unvalidated manual labels and the absence of statistical inference, which I detail below. The qualitative conclusion is plausible, but the quantitative evidence, as presented, does not fully support the strength of the claims.

major comments (5)
  1. [Section 3.1] The ground truth is the load-bearing assumption. The paper states that labels were assigned manually rather than by precise distance thresholds and concedes potential inaccuracies, but reports no inter-annotator agreement, no reliability study, and no validation against JRDB's available 3D annotations. Since all metrics are computed against these labels, label noise or systematic confusion between adjacent zones (e.g., moderate vs. low) would attenuate every reported accuracy, weighted F1, and recall toward the random baseline. The central negative conclusion therefore cannot be distinguished from 'the label set is an unreliable proxy for proxemic risk' without reliability evidence. Please report Cohen's kappa (or a similar measure) on a re-annotated subset and, if possible, derive distance-based labels from JRDB's 3D human boxes to validate the manual labels.
  2. [Section 4, Tables 1–3; Section 5.2] The paper reports point estimates without confidence intervals or significance tests. For example, the claim that fine-tuning yields no gain (ΔwF1 < 0.02) and the claim that correct and incorrect classifications have indistinguishable grounding (mean IoU 0.204 vs. 0.215) rest on differences that may be within sampling noise for a test set on the order of 1,000 images. Because the paper's headline is a null/negative result, this is not a cosmetic issue: bootstrap confidence intervals or paired tests (e.g., McNemar's test for classification, a Wilcoxon test for IoU) are needed to show that the null is not simply due to low statistical power.
  3. [Section 3.4] The fine-tuning description is not reproducible. The paper specifies r=8, alpha=16, dropout=0.05, and the adapted projections, but omits the number of epochs, learning rate, LR schedule, batch size, optimizer, sequence length, and total compute. The 200-image training set (50 per class) is very small, and the paper itself notes this in §5.3, but no sensitivity analysis is provided. Please report these details and, ideally, results with a larger training set or at least error bars across seeds.
  4. [Section 4.3/Table 4; Section 5.1] The headline high-danger recall of 0.79 is achieved at precision 0.182 (F1 0.296). The paper appropriately cautions that deployability is not demonstrated, but the recall-vs-precision trade-off is not analyzed, e.g., by varying a decision threshold or by reporting a precision-recall curve. Without such analysis, the claim that 'targeted prompting and fine-tuning can improve high-danger detection' is not yet quantitatively grounded, and the practical value of this configuration as a safety trigger remains unclear.
  5. [Section 5.2] The dissociation between classification and grounding is an interesting claim, but it is based on a fragile comparison. Mean IoU for correctly vs incorrectly classified frames (0.204 vs 0.215) and Qwen's true positives vs false negatives in the high-danger class (0.488 vs 0.464) are not shown to be significantly different, and the high rate of IoU=0 predictions (up to 0.68 for SmolVLM) suggests that invalid bounding boxes may dominate the averages. Please report the distribution of IoU (e.g., quartiles), the proportion of valid boxes, and a statistical test before interpreting the result as evidence that classification does not use spatial grounding.
minor comments (6)
  1. [Sections 3.3/4.1] The phrase 'the moderate prompt yields the lowest total misalignment score' refers to a metric that is never defined. Please either define 'misalignment' or remove the claim.
  2. [Section 4.2] 'Tab 3' should be 'Table 3' for consistency.
  3. [Section 5.3] The sentence 'which is may be due to structural limitations' contains a grammatical error; revise to 'which may be due to structural limitations'.
  4. [References] Several bibliography entries contain stray numbers appended to URLs (e.g., [4], [11], [14]), indicating a formatting error in the reference list. Please check the journal's reference style.
  5. [Section 3.3] The exact prompts are not included in the paper or in an appendix. Since prompt design is a central variable, the full prompt texts should be provided for reproducibility.
  6. [Table 1] The 'stratified random baseline' is given as 0.25 for accuracy and weighted F1, but with the stated class distribution (17.9/32.7/24.1/25.3), a random classifier choosing according to the class priors would have an expected weighted F1 of about 0.26. Please clarify how the baseline was computed (e.g., uniform random versus stratified random).

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the VLM evaluation compares model outputs to externally assigned manual labels; the conclusion is empirical and self-contained.

full rationale

The paper makes no first-principles derivation and fits no parameter that is later renamed as a prediction. Its pipeline is: (1) manually label 1,243 JRDB images into four danger classes using Hall's proxemic zones; (2) prompt and QLoRA fine-tune three open-weight VLMs; (3) compare outputs to those external labels with accuracy, weighted F1, high-danger recall, and IoU. Each target value is defined by the human-assigned labels and by JRDB bounding boxes, which are independent of the models' outputs. No equation in the paper defines a model input in terms of the reported result, and no prompt, threshold, or training set is selected from test-set labels. The only self-referential element is the paper's own manual labeling, but that is a ground-truth target definition, not a quantity derived from the models under test. The conceded limitation that manual labels could be inaccurate is a measurement-validity caveat, not circularity. The conclusion that four-class proxemic reasoning remains unsolved for modern VLMs is an empirical negative result based on external black-box evaluations, not a theorem derived from its own assumptions. There are no load-bearing self-citations by the authors, and the evaluation is self-contained against external benchmarks and model outputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on human-assigned proxemic labels and on the assumption that the chosen models and dataset represent modern open-weight VLM capability. No new physical or conceptual entities are introduced; the four danger classes are a relabeling of Hall's zones. The QLoRA hyperparameters are hand-set, and the main free 'parameter' is the subjective boundary between danger levels.

free parameters (2)
  • Proxemic ground-truth labels = 1,243 images: high 222, moderate 407, low 299, minimum 315
    Manual annotation without measured distances; all model scores are computed against these labels, so label subjectivity directly propagates into every reported number.
  • QLoRA hyperparameters = r=8, alpha=16, dropout=0.05, two stages
    Hand-chosen without sensitivity analysis; the fine-tuning comparisons depend on these settings.
assumptions (4)
  • domain assumption Hall's proxemic zones map monotonically onto four discrete danger levels in egocentric robot images.
    Section 3.1 uses Hall's zones as the classification criterion without validating that visual appearance alone determines the proxemic zone.
  • domain assumption Manual annotations are reliable enough to serve as ground truth for evaluating VLMs.
    Section 3.1 acknowledges manual labeling without distance thresholds 'could potentially lead to inaccuracies'; no inter-annotator agreement is reported.
  • domain assumption The chosen subset of JRDB is representative of egocentric robot perception in human environments.
    The dataset is a modified subset of JRDB, and no domain shift or coverage analysis is provided.
  • domain assumption The three selected open-weight models represent the behavior of modern open-weight VLMs on this task.
    Section 3.2 selects models by size and openness, but generalization to other VLMs is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?." pith.science (2026). https://pith.science/paper/YB4DSTEF

@misc{pith2026260812515,
  author       = {Pith},
  title        = {Pith review of: Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YB4DSTEF}},
  note         = {Machine review of arXiv:2608.12515}
}
read the original abstract

Assessing proxemic danger from a robot's egocentric perspective is critical for safe embodied navigation in human environments and requires both visual and contextual reasoning. We evaluate three opensource vision-language models (VLMs) (\textit{InternVL}, \textit{Qwen-VL}, and \textit{SmolVLM}) on the classification of egocentric robot images into four danger levels, comparing three prompting strategies and two rounds of QLoRA fine-tuning against a stratified random baseline. Without fine-tuning, all models perform near the baseline, while fine-tuning yields only modest overall improvements. However, \textit{Qwen-VL} with an advanced prompt achieves substantially higher recall for high-danger cases than the other models. An analysis of person localization further shows that correct danger classification does not correspond to better spatial grounding, indicating that a model may produce a useful safety label without attending to the relevant region of the scene. These results show that current VLMs remain limited in fine-grained proxemic reasoning and spatial grounding, although targeted prompting and fine-tuning can improve high-danger detection in selected models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 7 canonical work pages

  1. [1]

    Alayrac, J.B., Donahue, J., Luc, P., et al.: Flamingo: a visual language model for few-shot learning (2022),https://arxiv.org/abs/2204.141983

    arXiv 2022

  2. [2]

    Bai, J., et al.: Qwen-vl: A versatile vision-language model for understanding (2023), https://arxiv.org/abs/2308.129664

    arXiv 2023

  3. [3]

    Chen, Z., et al.: Internvl: Scaling up vision foundation models (2024),https: //arxiv.org/abs/2404.168214

    arXiv 2024

  4. [4]

    Micromachines12(2), 193 (2021).https://doi.org/10.3390/mi120201932

    Daza, M., Barrios Aranibar, D., Diaz Amado, J.A., et al.: An approach of social navigation based on proxemics for crowded environments of humans and robots. Micromachines12(2), 193 (2021).https://doi.org/10.3390/mi120201932

    work page doi:10.3390/mi120201932 2021

  5. [5]

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetun- ing of quantized llms (2023),https://arxiv.org/abs/2305.143143

    arXiv 2023

  6. [6]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al.: An image is worth 16x16 words: Transformersforimagerecognitionatscale(2020),https://arxiv.org/abs/2010. 119293

    work page 2020

  7. [7]

    Ge, J., Luo, H., Qian, S., et al.: Chain of thought prompt tuning in vision language models (2023),https://arxiv.org/abs/2304.079193, 5

    arXiv 2023

  8. [8]

    Doubleday (1966) 2, 4

    Hall, E.T.: The Hidden Dimension. Doubleday (1966) 2, 4

    work page 1966

Show all 21 references
  1. [9]

    co / HuggingFaceTB / SmolVLM-Instruct4

    HuggingFace: Smolvlm (2024),https : / / huggingface . co / HuggingFaceTB / SmolVLM-Instruct4

    2024

  2. [10]

    Rudas and D

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in 10 V. Rudas and D. Kuzmenko neural networks...

    2017 doi

  3. [11]

    Kojima, T., Gu, S.S., Reid, M., et al.: Large language models are zero-shot rea- soners (2022),https://arxiv.org/abs/2205.119163, 5

    2022 arXiv

  4. [12]

    Lee, I.Y., D’Antoni, L., Berg-Kirkpatrick, T.: The format tax (2026),https:// arxiv.org/abs/2604.036163, 6

    2026 arXiv

  5. [13]

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023),https:// arxiv.org/abs/2304.084853

    2023 arXiv

  6. [14]

    IEEE Transactions on Pattern Analysis and Machine Intelligence45(6), 6748–6765 (2023).https://doi.org/10.1109/TPAMI.2021.30705432, 3

    Martín-Martín, R., Patel, M., Rezatofighi, H., et al.: Jrdb: A dataset and bench- mark of egocentric robot visual perception of humans in built environments. IEEE Transactions on Pattern Analysis and Machine Intelligence45(6), 6748–6765 (2023).https://doi.org/10.1109/TPAMI.202...

    2023

  7. [15]

    International Journal of Social Robotics12, 267–280 (2020).https://doi.org/10.1007/s12369-019-00560-92

    Patompak, P., Jeong, S., Nilkhamhang, I., Chong, N.Y.: Learning proxemics for personalized human–robot social interaction. International Journal of Social Robotics12, 267–280 (2020).https://doi.org/10.1007/s12369-019-00560-92

    2020 doi

  8. [16]

    Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision (2021),https://arxiv.org/abs/2103.000203

    2021 arXiv

  9. [17]

    Electronics11(16), 2490 (2022).https://doi.org/10

    Samarakoon, S.M.B.P., Muthugala, M.A.V.J., Jayasekara, A.G.B.P.: A review on human–robot proxemics. Electronics11(16), 2490 (2022).https://doi.org/10. 3390/electronics111624902

    2022

  10. [18]

    In: Advances in Neural Information Processing Systems (2017),https://arxiv.org/abs/1706.037623

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (2017),https://arxiv.org/abs/1706.037623

    2017 arXiv

  11. [19]

    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator (2015),https://arxiv.org/abs/1411.45553

    2015

  12. [20]

    Yu, S., Chen, Y., Ju, H., et al.: How far are vlms from visual spatial intelligence? a benchmark-driven perspective (2025),https://arxiv.org/abs/2509.189053

    2025

  13. [21]

    Zhang, S., Dong, L., Li, X., et al.: Instruction tuning for large language models: A survey (2023),https://arxiv.org/abs/2308.107925

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.