Pith. sign in

REVIEW 2 major objections 1 minor 44 references

Benchmarks for vision-language models in urban perception must treat disagreement as a measurement outcome and labels as negotiable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 18:44 UTC pith:ZSZP2LM2

load-bearing objection The paper shows model agreement tracking human reliability in one Montreal study and argues benchmarks should report disagreement and allow negotiable labels, but the broad prescription rests on limited evidence. the 2 major comments →

arxiv 2606.00871 v1 pith:ZSZP2LM2 submitted 2026-05-30 cs.CV cs.AI

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

Rashid Mushkani This is my paper
classification cs.CV cs.AI
keywords vision-language modelsurban perceptionbenchmarkinginter-annotator reliabilitystreetscape auditingdisagreementMontreal scenesgovernance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper contends that current ways of testing vision-language models on street images overlook how humans often disagree or choose not to answer on subjective aspects like overall impression. To fix this, benchmarks should record disagreement rates, measure how consistent human annotators are, and allow the categories and scoring rules to be adjusted when the results will guide city decisions. Evidence comes from twelve community members rating one hundred Montreal scenes on thirty dimensions and comparing that to seven models' zero-shot outputs. The study found that models align better with humans on dimensions where people agree more, and models differ in how often they say a scene does not apply to a category. This approach would make evaluations more honest about uncertainty in perception tasks.

Core claim

Benchmarking VLMs for urban perception should treat disagreement and abstention as measurement outcomes, report inter-annotator reliability alongside model alignment, and treat the label space and scoring policy as negotiable artifacts when outputs are intended to inform urban governance. This recommendation follows from the observed co-variation between model agreement and human reliability in the Montreal study and the distributional mismatch on Overall Impression.

What carries the argument

The 100-scene Montreal benchmark annotated by 12 participants from seven community organizations along 30 dimensions, used to evaluate seven VLMs in a deterministic zero-shot setting and to measure inter-annotator reliability.

Load-bearing premise

The patterns from the single Montreal study with its specific annotators and scenes are representative enough to justify changing benchmarking practices everywhere.

What would settle it

A replication study across multiple cities where model agreement with humans does not increase on dimensions with higher human reliability, or where treating labels as negotiable produces no change in downstream decisions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark creators will include inter-annotator agreement scores in reports.
  • Model evaluations will flag dimensions with low human reliability separately.
  • Institutions will negotiate label spaces before adopting model outputs for policy.
  • Evaluation reports will explicitly state assumptions about what counts as agreement.
  • Actions will be taken by benchmark creators, model developers, and institutions to make uncertainty visible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar principles could apply to other subjective AI tasks like content moderation where human judgments vary.
  • Future studies might test if negotiating labels improves model utility in real urban planning scenarios.
  • Extending the approach to other cities could reveal if the co-variation pattern holds beyond Montreal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript argues that benchmarks for vision-language models applied to urban perception should treat disagreement and abstention as measurement outcomes rather than noise, report inter-annotator reliability in addition to model alignment scores, and treat the choice of label space and scoring policy as negotiable when the model outputs are intended to inform urban governance decisions. The argument is grounded in an empirical study of 100 Montreal street scenes annotated along 30 dimensions by 12 participants from seven community organizations, together with a deterministic zero-shot evaluation of seven VLMs, which reveals that model agreement with human consensus co-varies with dimension-level human reliability and that models and humans show distributional mismatch on the appraisal dimension 'Overall Impression' including different rates of 'Not applicable'.

Significance. If the reported co-variation and mismatch generalize, the paper identifies a substantive gap between current VLM evaluation practices and the requirements of applications in streetscape auditing and public consultation, where human judgments are distributional and context-dependent. Explicitly naming actions for benchmark creators, model developers, and institutions could help make uncertainty visible in evaluation reports.

major comments (2)
  1. [Montreal benchmark (described in abstract)] The normative recommendation that all urban-perception benchmarks must treat disagreement as an outcome and labels as negotiable rests on the observed co-variation in one 100-scene Montreal study with 12 annotators from seven organizations; no additional datasets, cross-city replication, or formal argument is supplied to establish that the pattern is not idiosyncratic to the chosen scenes, dimensions, or annotator pool.
  2. [Evaluation setup] The abstract (and by extension the grounding for the central claim) provides no information on the annotation protocol, how inter-annotator reliability was computed, the statistical test for co-variation between model agreement and human reliability, or exclusion criteria, which are load-bearing for verifying whether the data support the broad recommendation.
minor comments (1)
  1. [Abstract] The phrase 'deterministic zero-shot evaluation' could be clarified to indicate whether temperature was set to zero or if any other sampling controls were applied.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback. We address the major comments point by point below, with planned revisions noted.

read point-by-point responses

  1. Referee: [Montreal benchmark (described in abstract)] The normative recommendation that all urban-perception benchmarks must treat disagreement as an outcome and labels as negotiable rests on the observed co-variation in one 100-scene Montreal study with 12 annotators from seven organizations; no additional datasets, cross-city replication, or formal argument is supplied to establish that the pattern is not idiosyncratic to the chosen scenes, dimensions, or annotator pool.

    Authors: We agree the empirical support is a single case study and does not include replication. The manuscript frames the Montreal results as an illustrative grounding for a normative argument about handling distributional judgments and reliability in urban-perception benchmarks, rather than as a universal empirical claim. We will revise the introduction and discussion to more explicitly qualify the scope, emphasize the relevance to governance applications where human labels are context-dependent, and add a limitations paragraph calling for multi-city replications. This keeps the recommendation while addressing the risk of overgeneralization. revision: partial

  2. Referee: [Evaluation setup] The abstract (and by extension the grounding for the central claim) provides no information on the annotation protocol, how inter-annotator reliability was computed, the statistical test for co-variation between model agreement and human reliability, or exclusion criteria, which are load-bearing for verifying whether the data support the broad recommendation.

    Authors: The full manuscript details the annotation protocol in Section 3, Krippendorff's alpha computation in Section 4.1, the Pearson correlation test for co-variation in Section 5.2, and exclusion criteria in Section 3.3. We accept that the abstract is insufficiently informative and will expand it to include concise statements on these elements so readers can assess the grounding without needing the full text. revision: yes

standing simulated objections not resolved
  • Lack of cross-city replication or additional datasets to confirm the co-variation and distributional mismatch are not idiosyncratic to the Montreal scenes, dimensions, or annotator pool.

Circularity Check

0 steps flagged

No significant circularity; argument is empirical observation supporting normative recommendation

full rationale

The paper grounds its recommendation in a new empirical benchmark (100 Montreal scenes, 12 annotators, 7 VLMs) showing co-variation between model agreement and human reliability plus distributional mismatch on Overall Impression. No equations, fitted parameters, self-definitional loops, or load-bearing self-citations appear in the provided text or abstract. The central claim is presented as a proposal informed by this single study rather than a derivation that reduces to its own inputs by construction. Generalization from one dataset is a question of evidence strength, not circularity per the specified patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are identifiable from the provided text.

pith-pipeline@v0.9.1-grok · 5709 in / 1132 out tokens · 58535 ms · 2026-06-28T18:44:59.716845+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing, mapping, and public consultation. These uses combine observable attributes with appraisal categories, and the human targets are often distributions of judgments with disagreement and explicit non-response. This paper argues that benchmarking VLMs for urban perception should treat disagreement and abstention as measurement outcomes, report inter-annotator reliability alongside model alignment, and treat the label space and scoring policy as negotiable artifacts when outputs are intended to inform urban governance. We ground the argument in a benchmark of 100 Montreal street scenes annotated along 30 dimensions by 12 participants from seven community organizations, and in a deterministic zero-shot evaluation of seven VLMs. Across dimensions, model agreement with human consensus co-varies with dimension-level human reliability, and for the appraisal dimension Overall Impression models and annotators exhibit distributional mismatch including different rates of Not applicable. We close with actions for benchmark creators, model developers, and institutions to make uncertainty and benchmark assumptions visible in evaluation reports.

Figures

Figures reproduced from arXiv: 2606.00871 by Rashid Mushkani.

Figure 1
Figure 1. Figure 1: Self-reported participant context (counts). Categories are not mutually exclusive and reflect intersectional identities; partici￾pants could select multiple identity markers, so counts exceed the number of participants. Collection. Each image received between one and three independent annotations, resulting in 230 completed forms. A shared subset ensured overlap across participants for reli￾ability analysi… view at source ↗
Figure 2
Figure 2. Figure 2: Overall agreement with human consensus by model. Macro-averaged accuracy (single-choice) and Jaccard (multi-label) across 30 dimensions [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean Jaccard overlap between model selections and the human consensus for the multi-label dimensions. 7. Results Overall alignment summary [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Difficulty by dimension. Each point is the mean model score for the dimension (accuracy or Jaccard depending on the item type). Human reliability and model performance [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Agreement by dimension and model. Warmer colors indicate higher agreement with human consensus (accuracy for single￾choice, Jaccard for multi-label). cludes abstentions without documentation can obscure how often a model declines to provide an appraisal. Reporting abstention rates alongside alignment therefore changes what a score implies about the model and about the target. Fourth, the source effect betw… view at source ↗
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Inter-annotator reliability by dimension. Bars show Krippendorff’s α (nominal). Treating negotiation as part of benchmark governance raises questions about representation, authority, and resource re￾quirements. These questions are not resolved by a single protocol. The position of this paper is that explicit proce￾dures and versioned specifications provide a basis for con￾testation and revision that is not… view at source ↗
Figure 8
Figure 8. Figure 8: Appraisal versus observable performance by model. Each line connects a model’s macro score on the appraisal subset to its macro score on the observable subset. tical power. Annotations reflect judgments from a specific community and may not generalize. Deterministic French￾to-English normalization and CSV parsing can under-score models if mapping errors occur. Evaluation is zero-shot under a fixed prompt a… view at source ↗
Figure 9
Figure 9. Figure 9: Overall Impression: distribution across annotators and models. Curves indicate the proportion of images assigned to each category [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Model agreement by image source. Real photos (p6–p10) and synthetic renders (p1–p5). D. Prompt and Parsing Details The inference script provides a deterministic system prompt that lists the 30 columns in the order shown in [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 8 canonical work pages

  1. [1]

    Proceedings of the 42nd International Conference on Machine Learning , series =

    Mushkani, Rashid and Nayak, Shravan and Berard, Hugo and Cohen, Allison and Koseki, Shin and Bertrand, Hadrien , title =. Proceedings of the 42nd International Conference on Machine Learning , series =. 2025 , publisher =

    2025

  2. [2]

    Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

    Sap, Maarten and Swayamdipta, Swabha and Vianna, Laura and Zhou, Xuhui and Choi, Yejin and Smith, Noah A. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022. doi:10...

    work page doi:10.18653/v1/2022.naacl-main.431 2022

  3. [3]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2019 , doi =

    2019

  4. [4]

    Significance , volume =

    To Predict and Serve? , author =. Significance , volume =. 2016 , doi =

    2016

  5. [5]

    Proceedings of the 1st Conference on Fairness, Accountability and Transparency , series =

    Runaway Feedback Loops in Predictive Policing , author =. Proceedings of the 1st Conference on Fairness, Accountability and Transparency , series =. 2018 , url =

    2018

  6. [6]

    Datasheets for Datasets

    Datasheets for Datasets , author =. arXiv preprint arXiv:1803.09010 , year =

    work page Pith review arXiv

  7. [7]

    Berard and A

    Rashid Mushkani and H. Berard and A. Cohen and S. Koseki , title =. Proceedings of the 42nd International Conference on Machine Learning (ICML) , year =. 2501.17899 , eprinttype =

    work page arXiv

  8. [8]

    Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume =

    Mushkani, Rashid and Berard, Hugo and Ammar, Toumadher and Chatonnier, Cassandre and Koseki, Shin , title =. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume =. 2025 , doi =

    2025

  9. [9]

    Habitat International , year =

    Rashid Mushkani and Shin Koseki , title =. Habitat International , year =. doi:10.1016/j.habitatint.2025.103536 , url =

    work page doi:10.1016/j.habitatint.2025.103536 2025

  10. [10]

    2025 , url=

    Prompt Commons: Collective Prompting as Governance for Urban AI , author=. 2025 , url=

    2025

  11. [11]

    2025 , url=

    Collective Recourse for Generative Urban Visualizations , author=. 2025 , url=

    2025

  12. [12]

    Journal of Urban Management , year =

    Rashid Mushkani and Hugo Berard and Toumadher Ammar and Shin Koseki , title =. Journal of Urban Management , year =. doi:10.1016/j.jum.2025.07.004 , url =

    work page doi:10.1016/j.jum.2025.07.004 2025

  13. [13]

    2025 , url=

    WeDesign: Generative AI-Facilitated Community Consultations for Urban Public Space Design , author=. 2025 , url=

    2025

  14. [14]

    2025 , url=

    Street Review: A Participatory AI-Based Framework for Assessing Streetscape Inclusivity , author=. 2025 , url=

    2025

  15. [15]

    2025 , url=

    Urban AI Governance Must Embed Legal Reasonableness for Democratic and Sustainable Cities , author=. 2025 , url=

    2025

  16. [16]

    2025 , url=

    Negotiative Alignment: Embracing Disagreement to Achieve Fairer Outcomes -- Insights from Urban Studies , author=. 2025 , url=

    2025

  17. [17]

    1960 , publisher =

    The Image of the City , author =. 1960 , publisher =

    1960

  18. [18]

    ACM Transactions on Intelligent Systems and Technology , volume =

    Urban computing: Concepts, methodologies, and applications , author =. ACM Transactions on Intelligent Systems and Technology , volume =. 2014 , doi =

    2014

  19. [19]

    PLOS ONE , volume =

    The Collaborative Image of the City: Mapping the Inequality of Urban Perception , author =. PLOS ONE , volume =. 2013 , doi =

    2013

  20. [20]

    2016 , eprint =

    Deep Learning the City: Quantifying Urban Perception at a Global Scale , author =. 2016 , eprint =

    2016

  21. [21]

    Proceedings of the National Academy of Sciences , volume =

    Urban Visual Intelligence: Uncovering Hidden City Profiles with Street View Images , author =. Proceedings of the National Academy of Sciences , volume =. 2023 , doi =

    2023

  22. [22]

    Science , volume =

    Combining Satellite Imagery and Machine Learning to Predict Poverty , author =. Science , volume =. 2016 , doi =

    2016

  23. [23]

    Nature Communications , volume =

    Using Publicly Available Satellite Imagery and Deep Learning to Understand Economic Well-Being in Africa , author =. Nature Communications , volume =. 2020 , doi =

    2020

  24. [24]

    Scientific Data , volume =

    WorldPop, Open Data for Spatial Demography , author =. Scientific Data , volume =. 2017 , doi =

    2017

  25. [25]

    2023 , note =

    GPT-4 , author =. 2023 , note =

    2023

  26. [26]

    Proceedings of the 40th International Conference on Machine Learning , series =

    BLIP-2: Bootstrapping Language--Image Pre-training with Frozen Image Encoders and Large Language Models , author =. Proceedings of the 40th International Conference on Machine Learning , series =. 2023 , month =

    2023

  27. [27]

    2023 , eprint =

    InstructBLIP: Towards General-Purpose Vision–Language Models with Instruction Tuning , author =. 2023 , eprint =

    2023

  28. [28]

    2024 , url =

    Liu, Haotian and Li, Chunyuan and Li, Yuheng and Li, Bo and Zhang, Yuanhan and Shen, Sheng and Lee, Yong Jae , title =. 2024 , url =

    2024

  29. [29]

    2023 , url =

    Pichai, Sundar and Hassabis, Demis , title =. 2023 , url =

    2023

  30. [30]

    Proceedings of the 32nd ACM International Conference on Multimedia , series =

    Duan, Haodong and Yang, Junming and Qiao, Yuxuan and Fang, Xinyu and Chen, Lin and Liu, Yuan and Agarwal, Amit and Chen, Zhe and Li, Mo and Ma, Yubo and Sun, Hailong and Zhao, Xiangyu and Cui, Junbo and Dong, Xiaoyi and Zang, Yuhang and Zhang, Pan and Wang, Jiaqi and Lin, Dahua and Chen, Kai , title =. Proceedings of the 32nd ACM International Conference ...

    work page doi:10.1145/3664647.3685520 2024

  31. [31]

    Journal of the Royal Statistical Society: Series B (Methodological) , volume =

    Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing , author =. Journal of the Royal Statistical Society: Series B (Methodological) , volume =. 1995 , doi =

    1995

  32. [32]

    Proceedings of the Twelfth International Conference on Learning Representations , year =

    Language Models Represent Space and Time , author =. Proceedings of the Twelfth International Conference on Learning Representations , year =

  33. [33]

    2023 , eprint =

    GPT4GEO: How a Language Model Sees the World's Geography , author =. 2023 , eprint =

    2023

  34. [34]

    ACM Transactions on Spatial Algorithms and Systems , volume =

    Mai, Gengchen and Huang, Weiming and Sun, Jin and Song, Suhang and Mishra, Deepak and Liu, Ninghao and Gao, Song and Liu, Tianming and Cong, Gao and Hu, Yingjie and Cundy, Chris and Li, Ziyuan and Zhu, Rui and Lao, Ni , title =. ACM Transactions on Spatial Algorithms and Systems , volume =. 2024 , doi =

    2024

  35. [35]

    URL:https://dl.acm.org/ doi/10.1145/3589132.3625625,doi:10.1145/3589132.3625625

    Bhandari, Prabin and Anastasopoulos, Antonios and Pfoser, Dieter , title =. Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems , series =. 2023 , publisher =. doi:10.1145/3589132.3625625 , url =

    work page doi:10.1145/3589132.3625625 2023

  36. [36]

    and Ermon, Stefano , title =

    Manvi, Rohin and Khanna, Samar and Burke, Marshall and Lobell, David B. and Ermon, Stefano , title =. Proceedings of the 41st International Conference on Machine Learning , series =. 2024 , publisher =

    2024

  37. [37]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  38. [38]

    Advances in Neural Information Processing Systems , volume =

    Learning to Navigate in Cities Without a Map , author =. Advances in Neural Information Processing Systems , volume =

  39. [39]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

  40. [40]

    Proceedings of the ACM Web Conference 2024 , series =

    Yan, Yibo and Wen, Haomin and Zhong, Siru and Chen, Wei and Chen, Haodong and Wen, Qingsong and Zimmermann, Roger and Liang, Yuxuan , title =. Proceedings of the ACM Web Conference 2024 , series =. 2024 , publisher =. doi:10.1145/3589334.3645378 , url =

    work page doi:10.1145/3589334.3645378 2024

  41. [41]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Hao, Xixuan and Chen, Wei and Yan, Yibo and Zhong, Siru and Wang, Kun and Wen, Qingsong and Liang, Yuxuan , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2025 , doi =

    2025

  42. [42]

    Nature Neuroscience , volume =

    The Cognitive Map in Humans: Spatial Navigation and Beyond , author =. Nature Neuroscience , volume =. 2017 , doi =

    2017

  43. [43]

    Educational and Psychological Measurement , volume =

    A Coefficient of Agreement for Nominal Scales , author =. Educational and Psychological Measurement , volume =. 1960 , doi =

    1960

  44. [44]

    Psychological Bulletin , volume =

    Measuring Nominal Scale Agreement Among Many Raters , author =. Psychological Bulletin , volume =. 1971 , doi =

    1971