NavSight in the Wild: Understanding Real-World Use of a Mobile Augmented Reality Application for People with Low Vision in Outdoor Navigation

Ben Kosa; Kexin Zhang; Ru Wang; Sanbrita Mondal; Yuhang Zhao; Yuheng Wu

NavSight in the Wild: Understanding Real-World Use of a Mobile Augmented Reality Application for People with Low Vision in Outdoor Navigation

T0 review · 1 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports a seven-day diary study in which 12 people with low vision used NavSight, a mobile AR app that augments recognized outdoor objects, and finds that the augmentations support navigation by simplifying scenes, while…

desk verdict A solid week-long field study of AR low-vision navigation; the environmental-degradation finding is real but self-reported and should be framed as user-perceived, not model-verified. read the letter →

arxiv 2608.12759 v2 pith:JXGXOJOZ submitted 2026-08-13 cs.HC

Yuheng Wu , Kexin Zhang , Ben Kosa , Ru Wang , Sanbrita Mondal , Yuhang Zhao This is my paper · ORCID

classification cs.HC

keywords lowvisionaugmentedrealityoutdoornavigationdiarystudyvisualaugmentationsobjectrecognitionaccessibilityfielddeployment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that a mobile augmented-reality application can help people with low vision navigate uncontrolled outdoor environments, and to characterize how real-world use differs from lab evaluations. It presents NavSight, an app that recognizes 21 categories of outdoor objects and renders user-configurable visual augmentations, and reports a seven-day diary study with 12 low-vision participants. The study finds that augmentations simplified scenes into walkable and non-walkable regions, made tripping hazards and moving objects easier to notice, and extended visual reach, while also competing for attention and hands. Participants adapted their object selection, grouping, and augmentation styles across scenarios, and treated recognition errors differently depending on whether the error hid a hazard or merely added information. External conditions such as sun glare, wet surfaces, shadows, and nonstandard road markings degraded both recognition and augmentation visibility.

What carries the argument

The central object is NavSight, a smartphone AR app that runs a fine-tuned instance-segmentation model on the live camera feed and renders six user-configurable visual augmentations (contour enhancement, solid overlay, flashing, brightness adjustment, background darkening, and color removal) over two user-assigned object groups drawn from 21 outdoor object categories. The diary-study design—daily surveys, usage logs, screenshots, and a semi-structured exit interview—is the mechanism that moves the evaluation out of the lab and onto real sidewalks, driveways, and crossings. The load-bearing idea is that segmentation-based augmentations let users see environment structure directly, so even unrecognized hazards appear as holes in the augmented walkable surface.

What would settle it

Log the phone's camera feed, recognition outputs, and phone motion during the same diary protocol and compare them against participants' daily ratings and reported falls; if objective recognition failures and gait disruptions do not correlate with self-reported safety and helpfulness, or if falls are observed during sessions rated safe, the central characterization would need revision.

Watch

Extended reading notes

Core claim

NavSight's real-time visual augmentations can support real-world outdoor navigation for people with low vision by simplifying scenes into walkable and non-walkable regions, making tripping hazards and moving objects easier to notice, and extending visual reach, but external conditions such as sun glare, wet surfaces, shadows, and nonstandard markings degrade recognition and augmentation visibility, and users adapt by reconfiguring object selection, grouping, and effects over time. The paper derives this from a seven-day diary study in which 12 low-vision participants used the app in their own neighborhoods, parking lots, crossings, parks, and even indoor malls, with daily surveys, usage logs, screenshots, and exit interviews as evidence. A related finding is that segmentation-based augmentations can reveal unrecognized hazards as gaps in the augmented walkable surface, giving users a way to notice objects the model cannot name.

Load-bearing premise

The findings rest on participants' self-reported daily ratings and retrospective interview accounts being accurate measures of helpfulness, safety, accuracy, and distraction.

Editorial extensions

If this is right

Designers of low-vision AR navigation aids should expect users to configure augmentations dynamically rather than use fixed settings, and should support those changes with low overhead.
Recognition quality must be judged by consequence: errors that hide hazards (e.g., a step marked as sidewalk) are dangerous, while false positives that merely add information are tolerated and sometimes useful.
Environmental conditions, especially wet surfaces, shadows, and nonstandard markings, are first-order failure modes for AI recognition and need explicit mitigation before such aids can be relied on outdoors.
A phone-based form factor trades social acceptability and availability for divided visual attention and occupied hands; users adapt but still report distraction, especially while crossing streets.
Segmentation displays can multiply their value by exposing unrecognized hazards as gaps in the augmentation, suggesting that surface-level hazard detection should be a priority.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A wearable or gaze-aligned display would likely reduce the reported attention competition, but the paper's own social-acceptability findings suggest it could increase conspicuousness and the risk of being seen as filming; this trade-off is testable by running the same diary protocol across form factors.
The reliance on self-report could be tightened by logging camera frames, recognition confidence, and phone motion, then checking whether daily helpfulness and safety ratings track objective recognition failures and gait disruptions.
The asymmetry in error tolerance implies that evaluation metrics for assistive AR should weight false negatives on hazards more heavily than mAP-style averages do; a cost-sensitive metric derived from navigation outcomes would be a concrete next step.
The finding that false positives on out-of-list objects were sometimes helpful suggests a design where uncertain detections are deliberately rendered in a distinct 'possible hazard' style rather than suppressed.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is the first field deployment I know of real-time visual AR augmentations for low-vision outdoor navigation, and it is a solid piece of empirical HCI. The study tracks 12 low-vision participants for a week with usage logs, daily ratings, screenshots, and exit interviews. The headline finding is not that the augmentations work, it's how people actually use them: they simplify scenes into walkable/non-walkable regions, notice tripping hazards, scan for traffic, and they keep reconfiguring object selection and augmentation styles as lighting and context change. That behavioral material is new and credible.

The paper earns credit for transparency. Section 6.4 concedes the small sample, iOS-only deployment, self-report measures, and the absence of camera/recognition logs. The recognition model is evaluated on a fixed test split with per-class AP/FNR/FDR, and the weak classes (sidewalk, curb, crosswalk) line up with what participants complained about. The 'holes in the augmentation' finding, where users detect unrecognized hazards as gaps in the sidewalk overlay, is a genuinely nice observation. The discussion of trust calibration and consequence-based error metrics is thoughtful rather than speculative.

Now the soft spots, in proportion. The stress-test note is right: the paper states that weather, lighting/shadows, and nonstandard markings degrade recognition, but the evidence is participants' retrospective attributions plus a few screenshots, not logged model outputs. Section 6.4 explicitly says camera feeds and recognition results were not logged. So the causal link between environment and model failure is not established. Users could be misattributing errors to salient conditions. The abstract and Section 6.3.2 push the claim a bit harder than the evidence supports. That should be fixed either by logging or by reframing as perceived environmental challenges. It is a moderate limitation, not a fatal one. The central qualitative findings about usage and adaptation rest on the diaries and logs and hold up.

Minor items: no inter-rater reliability for the thematic coding; the 'no falls/injuries' safety line is weak at this sample size and duration, but the paper does not lean on it heavily. Code and data are promised but not yet public, so reproducibility is partial.

Bottom line: this deserves a serious referee and is likely a solid venue paper. I would send it to review with a request to soften or substantiate the environmental-degradation causal claim and add coding reliability. I would cite it as the main real-world AR low-vision deployment.

Referee Report

1 major / 5 minor

Summary. This paper presents NavSight, a mobile AR application for people with low vision that recognizes 21 outdoor object categories and renders six customizable visual augmentations in real time on an iPhone. The recognition component is a YOLO11l-seg model fine-tuned on Mapillary Vistas and evaluated on a static test split (mAP50=0.588, mAP75=0.305, mAP=0.318, with per-class AP/FNR/FDR reported in Appendix A). The authors deployed NavSight in a seven-day diary study with 12 participants with low vision, collecting daily 5-point ratings, privacy-preserving usage logs, user-captured screenshots, and semi-structured exit interviews. The findings characterize how participants used the app across outdoor scenarios, how augmentations changed scene perception (walkable versus non-walkable regions, tripping hazards, moving objects, visual reach), how the phone competed for attention and hands, how participants interpreted and coped with recognition errors, how they configured object selection and augmentation over time, and how they experienced social acceptability and usability in the wild. The paper derives design implications for context-aware configuration, debugging support, platform options, expanding recognition to surface-level hazards, supporting trust calibration, and consequence-aware evaluation metrics. Limitations, including self-reported measures and the absence of logged camera or recognition outputs, are acknowledged in Section 6.4.

Significance. If the findings hold, this is a valuable and timely contribution: it is one of the first in-the-wild studies of mobile AR visual augmentations for low vision outdoor navigation, and it provides ecological-validity evidence that short lab-based evaluations cannot supply. The paper is methodologically careful in several respects: the recognition model is quantitatively evaluated on a held-out test set with per-class metrics; the study triangulates daily ratings, usage logs, screenshots, and interviews; the logging design preserves bystander privacy; and the authors plan to open-source NavSight. The qualitative findings on configuration strategies, mental models of AI errors, and social acceptability are credible and generate concrete design hypotheses for future assistive AR systems. The main caveat is that the causal claims about environmental degradation of recognition rest on participant self-report rather than logged model outputs; this does not undermine the experiential findings, but the paper should label them as perceived effects and call for quantitative verification.

major comments (1)

[5.4.2 and 6.3.2 (also abstract)] The paper presents as a finding that weather conditions, lighting and shadows, and nonstandard road markings degrade NavSight's recognition, and Section 6.3.2 builds model-training implications (weather-robust datasets, image enhancement before inference) on this claim. The supporting evidence, however, consists of participants' retrospective attributions and a few illustrative screenshots; Section 6.4 explicitly states that camera feeds and recognition results were not logged to preserve privacy. Without logged recognition outputs or a controlled replication, the causal link between environmental conditions and model failures is not established, because users may attribute errors to salient conditions such as rain or shadows even when the true cause is viewpoint, distance, or object ambiguity. I recommend consistently framing these as user-perceived degradations, including in the abstract, and explicitly stating that the causal mechanism requires quantitative verification before model-training implications are drawn.

minor comments (5)

[Table 2 and Section 5.6] Please reconcile the day counts: P9 is listed as using NavSight for 11 days, but Section 5.6.1 says she rated comfort 5 on eight of her nine days; P7 is listed as 8 days, but Section 5.6.2 says she rated discomfort on all seven days.
[5.2] The sentence 'no participant reported a fall or injury while using NavSight' should be qualified by the small sample size and short deployment period, since it is not strong evidence of safety on its own.
[Section 3.4 and Table 4 note] The deployment thresholds (confidence = 0.4, IoU = 0.5) appear only in a table note; please state and briefly justify these choices in the main text, since FNR and FDR depend on them.
[Abstract and Section 1] The claim that no research has investigated real-world feasibility and challenges of AR systems for low vision is a strong universal statement; consider softening it to 'to our knowledge, no prior work has...' to avoid overclaiming.
[5.5.1] The claim that all participants converged on a stable set of augmented objects would be easier to assess with a small quantitative summary, such as the number of object-selection changes per participant per day, since the current support is narrative and spread across examples.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NavSight's findings are induced from diary data and its recognition model was evaluated on a held-out test set, independent of the deployment claims.

full rationale

This paper is an empirical field study, not a derivation. The central claims about how people with low vision used NavSight are induced from participant diaries, usage logs, screenshots, and interviews, and the recognition model was evaluated on a static 10% held-out test split of Mapillary Vistas (Section 3.4, Table 1, Appendix A) before deployment. No fitted parameter is renamed as a prediction, and no equation reduces to its own input. The six augmentations are described as 'inspired by prior research' (Section 3), including some of the authors' own earlier systems, but that citation is design provenance, not load-bearing evidence for the study's empirical findings. The paper's stated limitation in Section 6.4 that camera feeds and recognition results were not logged does mean the environmental-factor findings rely on participants' retrospective attributions, but that is an evidence-quality concern, not circularity: the claims are not equivalent to their inputs by construction. The safety line about no fall or injury is a self-report and is secondary. Accordingly, no circular step is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central result is observational, so the ledger is mostly domain assumptions about validity of self-report, representativeness of 12 participants, transfer of the trained recognition model to field conditions, and the coding process. One evaluation choice, the confidence and IoU thresholds used for FNR/FDR, is listed as a free parameter because the reported technical accuracy numbers depend on it. No invented entities are introduced; NavSight is a software system, not a theoretical construct.

free parameters (1)

deployment thresholds (confidence, IoU) = confidence = 0.4, IoU = 0.5
Chosen by the authors for the fine-tuned model; Appendix A's FNR/FDR values are conditional on this choice and no sensitivity analysis is reported. This is secondary to the diary findings.

assumptions (5)

domain assumption Self-reported diary surveys and interviews are valid evidence of real-world experience.
Most central findings, helpfulness, safety, accuracy, distraction, and comfort, come from 1 to 5 ratings and retrospective accounts; Section 6.4 acknowledges these measures rely on participants' impressions and memory.
domain assumption Participants' recognition-error attributions reflect actual system behavior.
The authors did not log camera feeds or recognition results, as stated in Section 6.4, so environmental causes of errors are inferred from participant reports rather than verified against model outputs.
domain assumption A 12-person convenience sample of iPhone-owning local volunteers is adequate to characterize PLV usage patterns.
The paper notes in Section 6.4 that the sample 'may not capture the full range of PLV's experiences, needs, and preferences.'
domain assumption The fine-tuned YOLO11l-seg-outdoor model's test-set performance transfers to the uncontrolled outdoor contexts in the diary study.
No egocentric field data was logged; model accuracy in the wild is only indirectly supported by participant perceptions and by the test-set evaluation in Section 3.4.
domain assumption Two-coder thematic analysis without a reported inter-rater reliability statistic yields reproducible themes.
Section 4.3 describes independent coding and consensus discussions, but no numeric agreement measure is reported, so theme stability is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NavSight in the Wild: Understanding Real-World Use of a Mobile Augmented Reality Application for People with Low Vision in Outdoor Navigation." pith.science (2026). https://pith.science/paper/JXGXOJOZ

@misc{pith2026260812759,
  author       = {Pith},
  title        = {Pith review of: NavSight in the Wild: Understanding Real-World Use of a Mobile Augmented Reality Application for People with Low Vision in Outdoor Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JXGXOJOZ}},
  note         = {Machine review of arXiv:2608.12759}
}

read the original abstract

The ability to navigate outdoors safely and independently is crucial yet challenging for people with low vision (PLV). While various augmented reality (AR) systems for low vision have been designed and evaluated in ideal lab environments, no research has investigated their real-world feasibility and challenges. We present NavSight, a mobile AR application that assists PLV in outdoor navigation by recognizing important outdoor objects (e.g., curb, vehicle) and rendering real-time visual augmentations. Through a seven-day diary study with 12 PLV in real-world settings, we characterize the impact of NavSight on scene perception, users' configuration strategies on what objects to augment and how to augment them across scenarios, how users made sense of and responded to recognition errors, and the social acceptability of using NavSight in public. We further identify environmental factors affecting recognition, such as weather conditions, lighting and shadows, and nonstandard road markings and textures, as well as usability issues in daily use. We discuss these real-world challenges and derive design implications for future AI-powered assistive AR systems for outdoor use.