{"id":"d8c65963-10ad-4e61-b941-48eedce94422","arxiv_id":"2505.08163","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Majority voting across three commercial LLMs achieves roughly 88 percent accuracy in detecting six neighborhood-environment indicators from Google Street View images, below a trained YOLOv11 detector's 99 percent mAP50.","lead":"Can general-purpose chatbots rate your neighborhood by looking at street photos? This paper tests four large language models on six visual signs like sidewalks and power lines, and finds that voting among the top three reaches about 88 percent accuracy without any model training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 88.5% majority-voting figure rests entirely on a single annotator's labels; without an inter-rater check the headline is an agreement rate, not validated accuracy.","rationale":"The reader identifies the single-labeler ground truth as the weakest assumption, and I agree. The main alternatives do not bite as hard. The mAP50-versus-accuracy terminology issue and the abstract's mention of fine-tuning that never appears in the methods are reporting flaws, but they do not directly invalidate the LLM comparison. The absence of released code and data is a reproducibility limitation, not a correctness flaw in the measured accuracies. The single-labeler concern is more fundamental: every accuracy table in Section IV-C, including the headline majority-voting result, is a comparison against one student's labels. If those labels are noisy or biased, the LLM scores and the YOLO baseline are both measured against an unreliable yardstick, and the claim that LLMs achieve 'over 88% accuracy' is not established. The proposed re-labeling check is feasible, inexpensive, and would settle the issue: high inter-rater agreement with stable recomputed accuracy would support the central claim in weakened form, while low agreement or a large accuracy shift would force the headline to be reworded as agreement with a single annotator. This matches the reader's CONDITIONAL verdict, so no adjustment is needed.","tokens_in":11560,"tokens_out":5221,"duration_ms":58892,"concrete_test":"Have two independent annotators re-label a random sample of 200-300 of the 1,200 images, stratified by indicator, using the same label definitions and protocol as Section IV-A. Compute per-indicator Cohen's kappa (or Fleiss' kappa) between the original labels and the adjudicated consensus labels, then recompute the majority-voting accuracies from Section IV-C2 against the adjudicated labels and compare with the reported 88.5%. If the recomputed average changes by less than 2-3 points and kappa is at least 0.7, the single-labeler concern does not undermine the conclusion. If the average moves by more than 5 points or kappa falls below 0.6, the paper should report accuracy as an interval or as 'agreement with a single annotator' until a multi-annotator study is performed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claim, 'over 88% accuracy' via majority voting, is computed by comparing every LLM response to a single undergraduate student's manual labels (Section IV-A: 'An undergraduate research student manually labeled images, including 1,927 indicator objects'). The only quality control reported is that 'the researcher checked the labels multiple times to ensure consistency and verified the label'; there is no second annotator, no adjudication, and no inter-rater reliability statistic such as Cohen's kappa. Because every per-indicator accuracy in Section IV-C2, and therefore the 88.5% average, is an agreement rate with this one set of labels, any systematic labeling bias or random labeling error directly changes the headline number. The Discussion (Section V) acknowledges that 'human error in labeling training data could impact the reliability of the model,' but this is not quantified, and it applies equally to the YOLOv11 baseline and to the LLM evaluation. This is the weakest load-bearing link because it affects all experiments at once: prompt strategy, language, temperature, and voting cannot rescue a result whose reference standard is unvalidated. Moreover, if the labeler and the LLM disagree, the reported 'accuracy' assumes the labeler is correct, and the paper supplies no evidence for that assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using off-the-shelf vision-capable large language models (ChatGPT 4o mini, Gemini 1.5 Pro, Claude 3.7, Grok-2) as zero-shot decoders of six neighborhood environmental indicators (streetlight, sidewalk, single-lane road, multilane road, powerline, apartment) from Google Street View images. It trains a YOLOv11 Nano baseline on 1,200 manually labeled images, reports high detection performance (99.1% average mAP50), and compares the four LLMs under parallel and sequential prompts, different prompt languages, and sampling-temperature/top-p settings. By majority voting across the three top-scoring LLMs, the paper claims 88.5% average accuracy, and concludes that LLMs can decode neighborhood environments without training effort.","tokens_in":11771,"tokens_out":8188,"duration_ms":74482,"significance":"The core idea is timely and practically relevant: if reliable, zero-shot LLM-based neighborhood auditing would substantially lower the barrier to large-scale built-environment assessment for public-health research. The paper is commendable for evaluating four commercial LLMs and per-class metrics, and the YOLO baseline provides a useful supervised reference. However, the headline numbers are vulnerable because all accuracy figures are agreements with a single annotator's labels and the majority-voting protocol is underspecified; the contribution currently reads as a promising feasibility study rather than a validated benchmark.","major_comments":[{"comment":"The ground truth for all experiments consists of labels produced by one undergraduate research student, with no second annotator, adjudication, or inter-rater reliability statistic (e.g., Cohen's kappa). The sentence in Section IV-A that 'the researcher checked the labels multiple times to ensure consistency' does not validate the labels. Consequently, every per-indicator accuracy in Tables III–VI and the 88.5% majority-voting average is an agreement rate with this single label set, and any labeling error or systematic bias directly shifts the reported numbers. An independent annotation subsample (e.g., 10–20% of images) with reported agreement is required before the word 'accuracy' can be used for the LLM results.","section":"Section IV-A and Section IV-C2"},{"comment":"The paper states that indicators are 'randomly split' 70/20/10, which suggests a split at the object level rather than the image level. Because the 1,200 images contain multiple indicator objects (1,927 objects total), object-level splitting can put instances from the same image in both training and test sets, causing data leakage and inflating the YOLO mAP50. The split must be performed at the image level, with all objects from one image confined to a single partition, and the reported detection results should be recomputed accordingly.","section":"Section IV-B1"},{"comment":"The majority-voting protocol is not fully specified. The text says the final prediction is reached 'when at least two models agree' but does not define the prediction for the three-way-disagreement case, nor does it state whether the reported accuracies are computed over all 1,200 images or only over the agreed subset. In addition, the 'top three' models are selected using the same test images on which the ensemble accuracy is then reported, so the 88.5% figure is not an out-of-sample estimate. Specify the complete voting rule and evaluate the ensemble on a held-out split or with a pre-registered model selection.","section":"Section IV-C2 and Figure 5"},{"comment":"The abstract's 'average accuracy of 99.13%' for YOLOv11 is in fact the average mAP50 from Table I (0.991; the per-class unrounded values average to 99.13%). Object-detection mAP50 is not classification accuracy, and the abstract's phrasing misrepresents the metric. Furthermore, the abstract and contribution list mention 'fine-tuning,' but no LLM fine-tuning is performed anywhere in the paper; the experiments cover prompt, language, and sampling-parameter tuning only. Revise the abstract and contributions to remove the fine-tuning claim and to label mAP50 as mAP50 (or 'detection accuracy' with a definition).","section":"Abstract and Section IV-B1"},{"comment":"The interpretation of the prompting experiment is internally inconsistent. The paper defines sequential prompting as asking about each indicator individually and parallel prompting as the concatenated 'and' version, but the discussion concludes that 'LLMs don't perform well with complex grammatical prompts when using sequential prompts.' Since Figure 4 shows parallel prompts outperforming sequential ones, it is the parallel prompt that is more complex grammatically; the sentence should refer to parallel prompts, and the connection to the cited work on simple sentence structure needs to be re-examined.","section":"Section IV-C1"}],"minor_comments":[{"comment":"The table heading 'Overall Accuracy' should be 'Overall Detection Performance' because the columns include precision, recall, F1, and mAP50; the abstract's 99.13% should be reconciled with the table's reported 99.1% average mAP50.","section":"Table I"},{"comment":"There are typos, including 'Genimi' in Figure 4 and Section IV-C2, and 'YOlOv11' in Section III-A; these should be corrected.","section":"Throughout"},{"comment":"The text says the appendix lists prompts in Spanish, simplified Chinese, and Bengali, but only the Spanish prompt is reproduced; the other two prompts should be added.","section":"Appendix B"},{"comment":"The statement that 'The samples for each indicator are evenly distributed' is contradicted by the label counts (streetlight 206, sidewalk 444, single-lane road 346, multilane road 505, powerline 301, apartment 125); clarify that class proportions were preserved in each split rather than implying balanced classes.","section":"Section IV-B1"},{"comment":"The per-class majority-voting accuracies listed (92.86, 84.91, 68.19, 97.07, 95.15, 95.15) average to approximately 88.9%, not the stated 88.5%; the rounding convention should be stated.","section":"Section IV-C2"},{"comment":"The LLM 'accuracy' metric is not defined in the main text; specify that it is (TP+TN)/(TP+FP+TN+FN) per indicator and clarify whether it is computed per image or per indicator.","section":"Section IV-C"},{"comment":"The comparison with prior work uses different indicators, datasets, and metrics (e.g., F1 versus accuracy); the claim of a 'significant improvement' over state-of-the-art should be tempered or supported by a controlled comparison on the same benchmark.","section":"Section IV-B3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable feasibility study, but as submitted it does not yet meet the evidentiary bar for a journal publication because the headline accuracy depends on unvalidated labels and an underspecified ensemble rule. I recommend major revision rather than rejection; the issues are addressable with additional annotation, a clarified voting protocol, and corrected reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper reports a genuinely new benchmark—four commercial LLMs on six neighborhood-environment indicators from Google Street View—and the YOLOv11 baseline is solid. But the headline \"over 88% accuracy\" for majority voting is really an agreement rate with a single undergraduate annotator. Until there is a second labeler or an inter-rater reliability check, that number is provisional.\n\nWhat's new and useful: this is the first study I know of that compares ChatGPT 4o mini, Gemini 1.5 Pro, Claude 3.7, and Grok-2 for detecting streetlights, sidewalks, powerlines, apartments, and road types in GSV images. The experiments cover parallel vs. sequential prompts, multilingual prompts, temperature/top-p, and majority voting. The findings that parallel prompts beat sequential, English beats Spanish/Chinese/Bengali, and voting lifts accuracy to 88.5% are practical for anyone building cheap screening tools. The YOLO baseline achieves 99.1 mAP50, which is impressive and honestly reported as mAP50 in the table, even though the abstract calls it \"average accuracy.\"\n\nThe main load-bearing issue is the ground truth. One undergraduate student labeled all 1,927 objects across 1,200 images; there is no second annotator, no Cohen's kappa, no adjudication. The stress-test note is correct: every LLM accuracy number is an agreement rate with those labels, and any systematic labeling bias moves the headline number. The Discussion acknowledges human labeling error but does not quantify it. This weakness affects the YOLO comparison too, though less so since the model is trained on the same labels.\n\nAlso, the abstract promises a focus on fine-tuning, but the paper only varies temperature and top-p; no fine-tuning appears. That is a mismatch between abstract and content. The YOLO result is 99.13% mAP50, not accuracy; calling it accuracy is misleading. No data or code are released, so independent replication is currently impossible. These are fixable, but they need to be fixed.\n\nOn the plus side, the paper is not circular: the LLM evaluation is an external benchmark against human labels, and the YOLO training is a standard supervised setup. The citation pattern looks fine, and the limitations section is honest about single-frame input and multilingual gaps.\n\nWho this is for: public-health and built-environment researchers who want a low-cost, training-free way to audit neighborhoods, and methodologists interested in how LLMs perform on fine-grained visual presence/absence tasks. If the ground-truth issue is resolved, this could become a useful reference.\n\nRecommendation: send it to peer review, but with major revision. Require a second annotator and an inter-rater reliability statistic, or at least an honest sensitivity analysis around label noise. Fix the abstract. Then it can be a solid contribution. I would not desk-reject this, but it is not ready as-is.","headline":"The 88% headline is an agreement rate with one annotator, so the central number is provisional; the benchmark is still genuinely new and referee-worthy.","tokens_in":12328,"tokens_out":2773,"would_cite":false,"duration_ms":27250,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Off-the-shelf LLMs, combined by majority vote, can decode neighborhood environmental indicators from street-view images at over 88% accuracy with no training, the paper argues.","keywords":["neighborhood environments","large language models","zero-shot image classification","Google Street View","majority voting","prompt engineering","YOLOv11","built environment health"],"falsifier":"Take a random subset of 300 labeled images, have three independent raters re-annotate them under a written rulebook, and measure agreement; if the original labels disagree with the majority re-annotation on more than a few percent of objects, recompute the 88.5% voting accuracy on the corrected labels, and the central claim would be falsified if the corrected accuracy falls clearly below the reported figure.","tokens_in":11356,"feed_emoji":"🏘️","tokens_out":7102,"duration_ms":66301,"temperature":0.7,"pith_summary":"This paper tests whether general-purpose vision-language models can do the work of trained object detectors in measuring neighborhood conditions from street-level imagery. The authors labeled 1,200 Google Street View images for six indicators—streetlight, sidewalk, powerline, apartment, single-lane road, and multilane road—and compared a YOLOv11 detector with four commercial LLMs using simple yes/no prompts. They report that majority voting across the three best models reaches 88.5% average accuracy without any training, while the detector reaches 99.1% mAP50. The result matters because neighborhood conditions are linked to health outcomes, and conventional assessment methods are too labor-intensive for large geographic areas.","feed_headline":"Off-the-shelf LLMs decode neighborhoods at 88%","feed_subtitle":"Majority voting across three models reads street-view photos for sidewalks and powerlines without retraining.","key_machinery":"The mechanism is a constrained parallel prompt plus majority voting. The parallel prompt concatenates six closed yes/no questions about the indicators into a single request with a fixed response format, so the model answers each question independently in one pass; this avoids the complex grammar that sequential follow-up prompts introduce. The yes/no answers from the three best-performing models are then fused by majority rule, and the paper reports this combination as the training-free decoder. The supervised reference point is a YOLOv11 (Nano) object detector trained on the same 1,927 labeled objects, which supplies the accuracy ceiling of 99.1% mAP50 against which the LLM pipeline is judged.","core_discovery":"The paper's central claim is that zero-shot inference from off-the-shelf LLMs, combined through majority voting, is a usable tool for decoding neighborhood environmental indicators at scale. On 1,200 street-view images, the best single models scored between 84% and 88% average accuracy; voting across Gemini, Claude, and Grok-2 lifted average accuracy to 88.5%, with per-indicator scores of 92.86% for streetlights, 84.91% for sidewalks, 68.19% for single-lane roads, 97.07% for multilane roads, and 95.15% for both powerlines and apartments. The parallel prompt, which asks all six yes/no questions in one request, outperformed sequential prompts, and English prompts outperformed Spanish, Chinese, and Bengali versions. The authors read this as evidence that LLMs can complement supervised models when no labeled training data are available.","pith_inferences":["A testable extension is a two-stage prompt for roads—first ask whether a road is present, then ask how many lanes—which would likely cure the single-lane-road failure mode that drags down the majority vote.","Because the paper's labels were produced by one annotator, a multi-annotator re-labeling of a subset would show whether the 88.5% figure is stable; both the YOLO baseline and the LLM numbers should be treated as provisional until then.","The reported language gap suggests the visual grounding of these models is English-centric; translating only the prompt words, not the visual concepts, loses recall, so future zero-shot pipelines should test bilingual prompts or image captions.","If LLM accuracy continues to improve, majority voting could eventually replace supervised detectors for routine screening, but the per-call cost and latency that the paper cites mean a practical large-scale pipeline would likely distill the LLM votes into a smaller local model."],"forward_implications":["Neighborhood audits for sidewalks, streetlights, and powerlines could be run on large image collections with API calls instead of a trained detection pipeline and human labelers.","English-only deployment would be most accurate; non-English prompts caused 15–20 percentage-point drops, so multilingual settings need prompt adaptation before the method is used there.","The method is already usable for multilane roads, powerlines, and apartments, with per-indicator accuracy above 95%, but single-lane road detection at 68% is not reliable enough without further prompting or multi-frame input.","A no-training LLM pipeline could make neighborhood-health studies easier to reproduce in new regions, since the same prompts can be applied to any street-view dataset."],"supporting_citations":[{"why":"Supplies ChatGPT 4o mini, one of the four evaluated models, and the temperature guidance used in parameter tuning.","marker":"[12]"},{"why":"Supplies Gemini 1.5 Pro, one of the evaluated models and part of the winning majority-vote set.","marker":"[14]"},{"why":"Supplies the YOLOv11 detector used to train the supervised baseline that sets the accuracy reference.","marker":"[17]"},{"why":"Supplies Claude 3.7, one of the evaluated models and part of the majority-vote set.","marker":"[18]"},{"why":"Supplies Grok-2, one of the evaluated models and part of the majority-vote set.","marker":"[19]"},{"why":"Provides the evidence that simple sentence structures outperform complex grammatical prompts, motivating the parallel prompt design.","marker":"[28]"},{"why":"Provides prior street-view indicator detection results that the YOLO baseline is compared against.","marker":"[11]"},{"why":"Establishes street-view-derived built environment indicators as predictors of health outcomes, motivating the six chosen labels and the study's purpose.","marker":"[6]"}],"fun_headline_variants":["LLM majority voting decodes neighborhoods at 88%","Zero-shot LLMs read street scenes with 88% accuracy","Off-the-shelf LLMs decode neighborhood indicators via voting","Majority-voted LLMs crack neighborhood decoding without training","LLMs hit 88% on street-view neighborhood decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole accuracy comparison rests on one undergraduate student's manual labels for 1,927 objects across 1,200 images being correct ground truth, with no second annotator or adjudication; if those labels contain systematic errors, both the YOLO baseline and the LLM scores inherit them.","fun_headline_variants_meta":{"raw":{"variants":["LLM majority voting decodes neighborhoods at 88%","Zero-shot LLMs read street scenes with 88% accuracy","Off-the-shelf LLMs decode neighborhood indicators via voting","Majority-voted LLMs crack neighborhood decoding without training","LLMs hit 88% on street-view neighborhood decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2102,"prompt_tokens":942,"completion_tokens":1160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":1078}},"tokens_in":558,"tokens_out":1160,"duration_ms":8552,"temperature":1.0,"reasoning_tokens":1078,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:02:01.231319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of 300 labeled images, have three independent raters re-annotate them under a written rulebook, and measure agreement; if the original labels disagree with the majority re-annotation on more than a few percent of objects, recompute the 88.5% voting accuracy on the corrected labels, and the central claim would be falsified if the corrected accuracy falls clearly below the reported figure.","supporting_citations":[{"cited_title":"Openai api reference - chat,","cited_arxiv_id":null,"evidence_quote":"Supplies ChatGPT 4o mini, one of the four evaluated models, and the temperature guidance used in parameter tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Gemini 1.5 Pro, one of the evaluated models and part of the winning majority-vote set."},{"cited_title":"Yolov11,","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv11 detector used to train the supervised baseline that sets the accuracy reference."},{"cited_title":"Claude ai,","cited_arxiv_id":null,"evidence_quote":"Supplies Claude 3.7, one of the evaluated models and part of the majority-vote set."},{"cited_title":"Grok 2 by xai,","cited_arxiv_id":null,"evidence_quote":"Supplies Grok-2, one of the evaluated models and part of the majority-vote set."},{"cited_title":"Decoding prompt syntax: Analysing its impact on knowledge retrieval in large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the evidence that simple sentence structures outperform complex grammatical prompts, motivating the parallel prompt design."},{"cited_title":"Multi- task classification for improved health outcome prediction based on environmental indicators,","cited_arxiv_id":null,"evidence_quote":"Provides prior street-view indicator detection results that the YOLO baseline is compared against."},{"cited_title":"Google street view images as predictors of patient health outcomes, 2017–2019,","cited_arxiv_id":null,"evidence_quote":"Establishes street-view-derived built environment indicators as predictors of health outcomes, motivating the six chosen labels and the study's purpose."}],"review_version":1}