{"id":"1ebdb87f-b3c0-4064-bcfa-571146a1065e","arxiv_id":"2411.16916","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"NYC DOT staff who tried a VR vision-loss simulation self-reported higher awareness and confidence afterward, but the small, uncontrolled study does not demonstrate a significant effect.","lead":"This paper describes a workshop in which New York City transportation staff tried virtual reality simulations of vision loss, then answered questionnaires about their awareness and confidence. The reported gains are based on 11 participants with no control group or statistical tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim that VR caused measured gains is unsupported: no control group, no inferential statistics, n=11 self-reports, and two pre/post items compare different question wording.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the pre/post differences are interpreted as causal effects of VR without controls, objective measures, or inferential statistics. My review confirms that this is the central soft spot. The paper is a well-structured workshop report with a thoughtfully designed VR simulation, and the qualitative focus-group and interview material has practical value, but the central claim of demonstrated effectiveness is not supported by the data as presented. The absence of raw data, the small volunteer sample, the lack of a control group, and the mismatched pre/post items for some outcomes all undermine the causal claim. The issue is not that VR cannot be effective; it is that this evidence does not establish that it is. Therefore I agree with the REJECT verdict and would not adjust it. The proposed replication and reanalysis would directly test whether the attribution to VR holds.","tokens_in":11694,"tokens_out":2660,"duration_ms":28898,"concrete_test":"Obtain the de-identified raw pre/post responses from the authors and run a preregistered replication with at least two arms: the full VR workshop versus an identical workshop without VR (or with a sham VR environment), using identical questionnaire items, an objective outcome measure such as a design-task scoring rubric, and a 3-month follow-up. If the VR arm does not outperform the control condition, or if the paired effects are not significant after correction, the central claim fails. As a minimal check with the existing data, compute paired Wilcoxon signed-rank tests and confidence intervals on the 11 responses for the identical items; with n=11, only very large and consistent shifts can reach significance, and if they do not, the asserted 'significant increase' is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the VR workshop produces significant increases in awareness, confidence, and intended practice (Sections 3.1, 4, 5)—rests entirely on pre/post self-report percentages from 11 self-selected DOT staff (Section 2.3), with no control condition, no random assignment, no inferential tests, and no raw data. Section 3.1 treats each percentage change as an intervention effect, but those changes can be fully explained by social desirability, demand characteristics, VR novelty, or the workshop's didactic framing alone. Moreover, two of the comparisons in Section 3.1 use non-identical items: pre-questionnaire Q4 asks whether participants currently consider VL needs when designing spaces, while post-questionnaire Q4 asks whether they would consider what they learned in future work; pre Q6 asks about awareness of VR for testing design guidelines, while post Q6 asks about awareness of VR for training mobility. Even for the identical items (Q1–Q3), the word 'significant' is asserted without p-values, confidence intervals, or paired tests. With n=11, chance fluctuations and response shift—where participants recalibrate their self-assessments after experiencing the simulation—are plausible. Thus the load-bearing assumption, that the self-reported changes are valid and caused by the VR simulation, is untested. The conclusion that 'the study demonstrates the effectiveness of VR' (Section 5) therefore overreaches; the evidence supports at most a pilot workshop report with promising subjective responses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a virtual reality (VR) workshop in which 11 New York City Department of Transportation staff experienced simulated vision loss (AMD, diabetic retinopathy, glaucoma) while navigating a virtual urban environment. Pre- and post-workshop Likert questionnaires, a focus group, and interviews with a Vision Zero Task Force are used to argue that the workshop significantly increased participants' awareness of vision-loss-related challenges, confidence in discussing mobility issues, and intention to consider visual impairment in future design work. The authors conclude that VR is an effective experiential learning platform for advancing accessibility in urban design.","tokens_in":11945,"tokens_out":2789,"duration_ms":27147,"significance":"If the study's central claims were methodologically sound, the work would be a useful contribution to accessibility education for urban planners, with a novel VR simulation design and a real-world stakeholder collaboration. The detailed description of the VR environment (shader-based simulation of three eye conditions, spatial audio, animated traffic) is a genuine strength, and the focus-group/interview material offers qualitative insight into how such workshops are received. However, the quantitative evidence is restricted to unanalyzed self-report percentages from 11 self-selected participants, with no control condition and no inferential statistics. As presented, the paper cannot support the claimed effectiveness of the VR intervention; at most it is a pilot workshop report with suggestive anecdotes.","major_comments":[{"comment":"The paper repeatedly uses 'significant' (e.g., 'Results showed a significant increase in awareness...') without any statistical test. Section 3.1 reports raw percentage changes from 11 participants, but provides no p-values, confidence intervals, or paired-test results. With n=11, these percentage changes are highly imprecise, and the word 'significant' is therefore not justified by the data. This is load-bearing because the paper's central claim, stated in the abstract and Section 5, is that the VR workshop 'demonstrates effectiveness.'","section":"Abstract and Section 3.1"},{"comment":"Two of the pre/post comparisons in Section 3.1 are based on non-identical items. Pre-questionnaire Q4 asks 'Do you consider persons with visual impairment needs when designing spaces?', whereas post-questionnaire Q4 asks 'Would you consider what you learnt in your future work?'. Pre Q6 asks about awareness of VR for 'testing design guidelines', whereas post Q6 asks about awareness of VR for 'training mobility'. The reported increases for these items conflate the intervention effect with changes in question wording, so the corresponding percentage changes (e.g., 40% to 80% for 'consider needs') cannot be interpreted as pre/post change in the same construct.","section":"Section 2.4 and Tables 1-2"},{"comment":"The causal claim that the VR experience 'effectively deepened comprehension' and 'fostered empathy and a long-term commitment' (Section 3.1) is unsupported by the absence of a control condition. Participants were self-selected DOT staff who were told the workshop was designed to raise awareness; social desirability, demand characteristics, and the novelty of VR are all plausible explanations for the observed self-reported improvements. The conclusion in Section 5 that 'this study demonstrates that VR technology holds significant promise' overreaches: the design cannot rule out these alternative explanations, and the manuscript does not acknowledge this limitation.","section":"Sections 2.3, 3.1, and 5"},{"comment":"The focus group and interview findings are presented as 'corroborating' the questionnaire results, but no systematic qualitative analysis (e.g., coding scheme, thematic analysis, inter-rater reliability) is reported. Quotations and paraphrases are used selectively, and the interviews were conducted with a different group (Vision Zero Task Force) than the workshop participants, making them even less informative as evidence for the workshop's effects. The manuscript should either provide a rigorous qualitative analysis or explicitly label these as anecdotal.","section":"Section 3.2 and 4"}],"minor_comments":[{"comment":"The abstract contains a typo: 'VL and and related immersive mobility challenges' should read 'VL and related immersive mobility challenges.'","section":"Abstract"},{"comment":"The text says the platform was designed 'to home a comprehensive understanding'; 'home' should be 'hone' (or 'develop').","section":"Section 2.2"},{"comment":"The text refers to 'Fig. 5c' ('see Fig. 5c') for the ABOUT button, but Figure 5 has only panels (a) and (b). Please correct the cross-reference or add the missing panel.","section":"Section 2.3 and Figure 5"},{"comment":"Several citations appear mismatched to their context: reference [40] (Giudice & Legge, 2008) is cited for the claim that VR offers unique advantages in educating users about urban accessibility, but the cited work concerns blind navigation and technology more broadly; reference [51] (Krull & Duart on mobile learning) is cited to support VR's ability to enhance urban planning processes. Please verify and replace with more directly relevant references.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript is essentially a workshop report with a suggestive but methodologically underpowered evaluation. The central claim of demonstrated effectiveness is not supported by the data, and the flaws (no control group, n=11, unmatched items, no inferential tests) are structural rather than cosmetic. While the authors might be able to reframe this as a pilot study with strong qualitative findings, that would require a substantial rewrite of the title, abstract, and conclusions, and even then the quantitative claims would need to be drastically qualified. This is more appropriate for a practice-oriented or work-in-progress venue than for a full research article in a journal in this field."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, you should know this upfront: the paper is a detailed workshop report, and the VR simulation is the best part. The claim that the workshop 'demonstrates' VR's effectiveness is not supported by the evidence. The design and practical detail are useful; the analysis is not.\n\nWhat's genuinely new: the collaboration with NYC DOT and the concrete implementation of three vision loss simulations (AMD, DR, glaucoma) at three severities using Unity shaders and culling masks. The description of obstacles, audio cues, and animated traffic is specific enough to be useful to someone building a similar tool. The focus group and Vision Zero interviews give real-world context. That's worth something.\n\nThe soft spot is the central claim. Numbers in Section 3.1 are raw pre/post percentages from 11 self-selected DOT staff, no control group, no p-values, no confidence intervals, no raw data. The word 'significant' is prose, not a test. Worse, two pre/post comparisons compare non-identical questions: pre-Q4 asks about current design consideration, post-Q4 asks about future intention; pre-Q6 asks about VR for testing design guidelines, post-Q6 asks about VR for training mobility. Those are not valid before/after measures. The discussion converts promising subjective responses into 'demonstrated effectiveness' — that's an overreach. Also, some citations don't support the claims they're attached to (e.g., Winters & Teschke is about cycling routes, not VR urban design; Giudice & Legge is about navigation technology, not VR education). That's sloppy.\n\nTo be fair, the paper doesn't hide its pilot nature entirely, and the focus group quotes are honest. But the abstract and conclusion overstate.\n\nWho is this for? People building VR empathy tools for accessibility training, or practitioners wanting a case study of a government agency workshop. They'll get useful implementation ideas. As evidence of VR efficacy, it's not credible.\n\nMy recommendation: I would not send this to a serious peer-reviewed journal as is. It needs to be reframed as a pilot/feasibility report with explicit limitations and no causal claims. If a venue exists for that — maybe a human factors workshop track — then it could be acceptable after heavy revision. But as a research paper, desk reject.","headline":"A well-built VR workshop report whose effectiveness claim outstrips the evidence; useful implementation detail, but no inferential support.","tokens_in":12523,"tokens_out":4056,"would_cite":false,"duration_ms":39777,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simulated vision loss in VR made urban planners far more aware of accessibility barriers.","keywords":["virtual reality","vision loss","accessibility","inclusive design","urban planning","experiential learning","empathy","workshop evaluation"],"falsifier":"Run the same workshop with two groups: one that experiences the immersive VR simulation and one that watches a recorded walkthrough of the same environment with the same explanations, then compare identical questionnaires; if the recorded group shows equal gains, immersion is not the causal ingredient, and the paper's central mechanism is falsified.","tokens_in":11497,"feed_emoji":"🥽","tokens_out":6233,"duration_ms":57185,"temperature":0.7,"pith_summary":"The paper reports on a workshop in which staff of the New York City Department of Transportation put on VR headsets and navigated a simulated NYC street while experiencing simulated versions of three common eye conditions: age-related macular degeneration, diabetic retinopathy, and glaucoma, at mild, moderate, and severe levels. The aim is to establish that this kind of first-person simulation can make accessibility problems tangible to the people who design streets, and thereby improve how vision-loss considerations enter urban design. Comparing questionnaires given before and after the session, the authors report large gains in participants' self-rated confidence about blindness and low vision, awareness of design guidelines, and willingness to consider vision-loss needs in future work. If those self-reports reflect real change, the workshop offers city agencies a low-cost way to build empathy and to test design interventions before any concrete is poured.","feed_headline":"VR vision-loss sims raise planners' accessibility awareness","feed_subtitle":"After one immersive session, 65% of planners felt confident on vision-loss basics, up from 10%.","key_machinery":"The load-bearing mechanism is the custom VR simulation itself: a Unity-built virtual urban environment modeled on real NYC locations, displayed through an Oculus Quest headset, with vision-loss effects produced by Gaussian-blur and distortion shaders combined with culling masks that selectively render parts of the scene. That setup reproduces central-vision loss for age-related macular degeneration, scattered dark spots and blur for diabetic retinopathy, and tunnel vision for glaucoma, while egocentric audio cues give users spatial information about traffic, cyclists, and people nearby. The argument runs through this mechanism: the embodied, first-person encounter with these visual conditions is what converts abstract accessibility guidelines into felt experience, which the questionnaires then measure.","core_discovery":"The central claim is that immersive VR simulation of vision loss is an effective experiential-learning tool for urban design professionals. Participants chose among three simulated conditions and three severity levels, then walked through a virtual environment modeled on challenging NYC streetscapes and stocked with bollards, construction sites, scaffolding, benches, traffic, cyclists, and spatialized audio cues. On pre/post Likert questionnaires, the share of participants rating themselves 'quite' or 'extremely' confident about the definitions of blindness and low vision rose from 10% to 65%; awareness of relevant design guidelines rose from under 15% to over 50%; confidence about mobility issues rose from under 20% to nearly 70%; and the share who said they would consider vision-loss needs in future work rose from 40% to over 80%. The paper reads these shifts as evidence that the experience deepened knowledge, empathy, and intention, and it presents focus-group and follow-up interview material in which participants endorsed VR as a tool for testing street designs, such as tactile paving, accessible pedestrian signals, and leading pedestrian intervals, before real-world implementation.","pith_inferences":["Beyond the paper, the same immersion could work in reverse: giving persons with vision loss the ability to walk through proposed designs and flag barriers before construction, turning them from reviewers into co-designers.","Beyond the paper, a control condition that watches a recorded walkthrough without wearing the headset would isolate whether embodiment, not just new information, drives the reported gains; without that, the active ingredient is unproven.","Beyond the paper, follow-up audits of actual design documents or street plans six to twelve months later would test whether self-reported commitment translates into changed practice.","Beyond the paper, co-designing the simulation with persons with vision loss could improve its validity, since simulated visual impairment may not capture the auditory and tactile skills that experienced blind pedestrians develop."],"forward_implications":["City agencies can trial accessibility interventions such as tactile paving, curb ramps, accessible pedestrian signals, and leading pedestrian intervals in VR before building them, avoiding the cost and risk of physical pilots.","VR workshops could be embedded in existing safety programs like Street Improvement Projects and the Bus Stops Under the Elevated initiative to train staff who make street-level decisions.","The same simulation could be extended to a continuum of vision loss, including gradual deterioration, and to changing conditions such as time of day and weather, making designs more resilient.","If the reported intention to consider vision-loss needs persists, professional practice in planning departments could shift toward more inclusive design as a routine default."],"supporting_citations":[{"why":"Supplies the epidemiological basis for choosing age-related macular degeneration, diabetic retinopathy, and glaucoma as the three simulated conditions.","marker":"[28]"},{"why":"Cited for the claim that immersive VR makes abstract concepts tangible and can create meaningful empathy.","marker":"[39]"},{"why":"Cited to support VR's unique advantages for educating about urban accessibility through simulated real-world design challenges.","marker":"[40]"},{"why":"Cited for VR as an experiential-learning tool that bridges theory and practice.","marker":"[41]"},{"why":"Cited for the idea that immersive VR can inspire realistic behavior and longer-term change.","marker":"[43]"},{"why":"Prior work using VR for empathic modeling of vision impairment, directly analogous to this workshop.","marker":"[44]"}],"fun_headline_variants":["VR sim lets planners experience vision loss, boosting design awareness","Immersive VR walk sharpens planners' view of blindness and low vision","VR vision-loss simulation lifts planners' confidence and awareness","Planners gain vision-loss insight via immersive VR streetscape","VR role-play makes planners 6.5x more confident on vision-loss basics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result rests on participants' self-ratings on a short questionnaire measuring true learning caused by the VR experience; without a control group, the same improvements could come from social desirability, the novelty of wearing a headset, or a general wish to please the workshop organizers.","fun_headline_variants_meta":{"raw":{"variants":["VR sim lets planners experience vision loss, boosting design awareness","Immersive VR walk sharpens planners' view of blindness and low vision","VR vision-loss simulation lifts planners' confidence and awareness","Planners gain vision-loss insight via immersive VR streetscape","VR role-play makes planners 6.5x more confident on vision-loss basics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000962,"raw_usage":{"total_tokens":4092,"prompt_tokens":935,"completion_tokens":3157,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":3068}},"tokens_in":551,"tokens_out":3157,"duration_ms":22857,"temperature":1.0,"reasoning_tokens":3068,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:44:26.000930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same workshop with two groups: one that experiences the immersive VR simulation and one that watches a recorded walkthrough of the same environment with the same explanations, then compare identical questionnaires; if the recorded group shows equal gains, immersion is not the causal ingredient, and the paper's central mechanism is falsified.","supporting_citations":[{"cited_title":"Ophthalmic Epidemiology 30(2), 129–141 (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the epidemiological basis for choosing age-related macular degeneration, diabetic retinopathy, and glaucoma as the three simulated conditions."},{"cited_title":"Computers & Electrical Engineering 93, 107272 (2021)","cited_arxiv_id":null,"evidence_quote":"Cited for the claim that immersive VR makes abstract concepts tangible and can create meaningful empathy."},{"cited_title":"The engineering handbook of smart technology for aging, disability, and independence, 479–500 (2008)","cited_arxiv_id":null,"evidence_quote":"Cited to support VR's unique advantages for educating about urban accessibility through simulated real-world design challenges."},{"cited_title":"Cyberpsychology & behavior 8(3), 220–230 (2005)","cited_arxiv_id":null,"evidence_quote":"Cited for VR as an experiential-learning tool that bridges theory and practice."},{"cited_title":"Philosophical Transactions of the Royal Society B: Biological Sciences 364(1535), 3549–3557 (2009)","cited_arxiv_id":null,"evidence_quote":"Cited for the idea that immersive VR can inspire realistic behavior and longer-term change."},{"cited_title":"In: Proceedings of the 33rd Australian Conference on Human- Computer Interaction, pp","cited_arxiv_id":null,"evidence_quote":"Prior work using VR for empathic modeling of vision impairment, directly analogous to this workshop."}],"review_version":1}