Pith. sign in

REVIEW 3 major objections 4 minor 97 references

Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Showing blind and low-vision users where AI image descriptions disagree helps them spot unreliable claims and calibrate their trust in the model.

desk verdict Useful design space and user study, but the headline 4.9x effect is raw flagging, not validated error detection; the paper still deserves peer review. read the letter →

arxiv 2507.15692 v1 pith:MH6KP25O submitted 2025-07-21 cs.HC cs.CLcs.CV

Meng Chen , Akhil Iyer , Amy Pavel This is my paper
classification cs.HCcs.CLcs.CV
keywords accessibilityblindandlowvisionmultimodallargelanguagemodelsimagedescriptionsAIhallucinationtrustcalibrationscreenreadervariationsurfacing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that showing blind and low-vision (BLV) users the places where several multimodal large language models disagree about the same image is a practical way to help them tell reliable claims from unreliable ones without seeing the image. In a study with 15 BLV participants, people who read a variation summary and a variation-aware description built from nine model outputs identified 4.9 times as many unreliable claims as people who read a single description, and their average perceived reliability of the model output dropped from 5.78 to 3.93 on a 7-point scale. The paper contributes a design space for eliciting and presenting MLLM variations, a prototype that automatically aligns and aggregates descriptions from three models, and evidence that most participants (14 of 15) preferred seeing variations over a single description. The point that would matter if true is that disagreement among models is a usable error signal for people who cannot visually verify the image.

What carries the argument

The machinery is a design space plus an automatic aggregation pipeline. The design space covers how variations are elicited (multiple trials, prompts, or models), how they are compared (raw list, variation-aware description, or variation summary), the granularity of comparison (words, atomic facts, sentences, responses), and optional support and provenance indicators. The pipeline decomposes each model's description into atomic facts, clusters facts that refer to the same attribute, concatenates disagreeing variants with 'or', annotates which model or models support each variant, and then condenses the result into a variation summary that separates agreements, disagreements, and unique mentions. This aggregation is what lets a screen-reader user grasp disagreement without holding multiple long descriptions in memory.

What would settle it

Re-run the study on a fresh image set with claim-level ground truth produced independently by multiple sighted raters, and compute the condition effect separately for factual errors, speculative statements, and subjective opinions; if the 4.9x advantage mostly comes from opinionated claims or vanishes under stricter labels, the claimed calibration benefit is not about factual unreliability.

Watch

Extended reading notes

Core claim

The central claim is that surfacing variations across multiple MLLM-generated image descriptions lets BLV users detect unreliable information and calibrate their trust, and that aggregated presentations do this better than raw lists. The authors define unreliable claims as incorrect, speculative, or opinionated statements, and they show that when participants used their approach, a variation summary that highlights agreements, disagreements, and unique mentions together with a variation-aware description that merges atomic facts into one hierarchical text, the mean number of unreliable claims identified rose from 0.53 for a single description to 2.62, a 4.9x increase, while perceived reliability fell from 5.78 to 3.93. They also find that users rely on inconsistency as the main cue, with 94–96% of claims flagged in the variation conditions being attributed to inconsistencies, and that 11 of 15 participants ranked the variation summary as their favorite presentation style. The authors take this as evidence that systematic comparison across models can compensate for the absence of visual verification in high-stakes and subjective scenarios.

Load-bearing premise

The load-bearing premise is that the researchers' labels of which claims are 'unreliable' are accurate and consistent, since the category mixes factual errors with subjective opinions and noisy labels would make the measured detection rates a less clean measure of catching genuine errors.

Editorial extensions

If this is right

  • If variation surfacing is adopted in visual access tools, BLV users could get an explicit disagreement signal for high-stakes queries such as medication labels, charts, navigation, and tornado maps instead of a single confident answer.
  • Interfaces should prioritize aggregated presentations such as the variation summary and variation-aware description over raw lists, since participants found lists cognitively demanding and preferred summaries.
  • Perceived reliability of MLLM image descriptions will move closer to actual reliability when variations are shown, reducing the over-trust documented in earlier work.
  • The same comparison approach can be extended to other media, including video descriptions and computer-use agents, by surfacing divergent branches or frame descriptions.
  • Users may want to customize how support is shown, choosing between source counts, percentages, natural language, or no indicator, and to switch presentation styles depending on the task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the mechanism is model-agnostic disagreement, so it could work as a general trust-calibration layer for any black-box generative system, not just image descriptions, including documents, code, or medical summaries.
  • We infer that the paper's 'unreliable' category mixes factual errors with subjective opinions, so in subjective tasks a flagged disagreement may reflect legitimate diversity of viewpoints rather than model error; a deployed system may need to distinguish 'uncertain' from 'wrong.'
  • We infer that showing agreement counts could create a new failure mode: users may over-trust claims that all three models agree on, even when the models share the same hallucination or bias, a scenario the paper does not test.
  • We infer that the 4.9x effect could be tested with sighted users as well, since the comparison interface is not inherently tied to blindness, and a positive result would make variation surfacing a general interface technique for AI error detection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes that systematically surfacing variations across multiple MLLM-generated image descriptions helps blind and low vision (BLV) users detect unreliable information and calibrate their trust. The authors contribute a design space for eliciting and presenting variations, a prototype with three presentation styles (list of multiple descriptions, variation-aware description, and variation summary), and a within-subject user study with 15 BLV participants. The study compares a single MLLM description against a list of 9 descriptions and against the authors' aggregated approach, measuring the number of claims participants identified as unreliable and their perceived reliability ratings. The paper reports a 4.9x increase in identified unreliable claims for the aggregated approach relative to a single description, a significant decrease in perceived reliability, and a strong participant preference for the variation summary. The authors also report qualitative findings on why descriptions appear unreliable, design preferences, and potential use cases.

Significance. If the central claim holds, this is a useful and timely contribution to accessible AI: it addresses a real practice of BLV users cross-checking multiple tools, builds on prior work on LLM variation, and evaluates a concrete, screen-reader-friendly interface with actual BLV participants. The design space and the detailed preference data are valuable for future work. However, the headline quantitative result depends on how 'unreliable claims identified' is operationalized and on the unvalidated automatic aggregation pipeline. The paper does not yet establish that the observed increase reflects detection of actual errors rather than over-flagging of correct or legitimately subjective content. These issues are fixable with re-analysis and additional validation, so the contribution is defensible but requires revision.

major comments (3)
  1. [Section 5, Table 4] The headline 4.9x result is computed from the total number of claims participants flagged as unreliable, but this total includes False Positive and Opinionated counts, not only claims independently verified as incorrect or speculative. In Table 4, the overall totals are 24 (Single), 101 (List), and 118 (Ours), yet the True Positive (Incorrect) counts are 7, 42, and 63 respectively, and True Positive (Speculative) counts are 11, 27, and 28. The paper's own qualitative analysis in Section 5.1.2 reports that participants used uniqueness as a cue and flagged correct claims as unreliable, such as P11's 'Romania was seen in some models, but not in other models.' No inter-rater reliability or independent validation is reported for the researchers' true-positive/false-positive classification, and 'speculative' claims cannot be verified from the image alone. Please report the analysis restricted to true-positive incorrect claims, and validate the claim-level ground truth (e.g., dual coding with agreement statistics).
  2. [Section 3.2.2, Figure 1] The 'Ours' condition is not a pure presentation of the 9 raw model descriptions: the variation-aware description and variation summary are generated by Gemini 2.5 Pro using Chain-of-Thought and few-shot prompts. The paper states that the prompt instructs the model to retain all distinct claims, but no evaluation is reported of whether the aggregation pipeline actually preserves all atomic facts, avoids introducing new errors, and correctly attributes facts to source models. If the summary omits or alters claims, the observed difference between conditions reflects content differences rather than the effect of surfacing variations. This is a load-bearing internal-validity concern. A small validation study measuring recall and precision of the aggregated facts against the 9 source descriptions would address it.
  3. [Sections 4.3 and 5.1] The operational definition of 'unreliable' includes 'opinionated' claims, and the study applies this definition to subjective tasks such as outfit matching and room aesthetics. The variation summary explicitly lists disagreements and unique mentions, and participants reported using inconsistency as their primary cue in the List and Ours conditions (96% and 94% of reported claims). Table 4 shows Opinionated counts rising from 1 in Single to 13 in Ours, concentrated in the Subjectivity category. Under this design, a model's differing subjective opinion is counted as an unreliable claim, and the intervention itself supplies many of the items participants then report. The paper therefore does not yet demonstrate that users' perceived reliability is calibrated to actual reliability; it shows that surfacing disagreements lowers trust and increases flags, which may include over-flagging of legitimate variation. Please report factual error detection separately from subjective-disagreement detection, and discuss calibration with respect to independently established answer correctness.
minor comments (4)
  1. [Section 4.4] The counterbalancing description is vague: 'counterbalanced description conditions across the three ambiguity sources' does not specify whether the assignment of images to conditions was a Latin square, randomized, or another scheme, nor whether condition order was counterbalanced. Please clarify and report any order effects.
  2. [Figure 4] The caption and axis area contain apparent placeholder strings such as '/gid00011'; these should be removed or replaced with proper labels.
  3. [Throughout] There are several typos and small errors: 'percieved' in RQ1, 'Tabe 4' in Section 5, 'splited' in Section 5.1.2, 'an another representative' in Section 3.2.2, and 'Participant ranged' in Section 4.1.
  4. [Table 8] The image I6 is labeled 'Bottle' but the task prompt and description refer to a baseball card; please verify whether the label or the prompt is correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 4.9x result is a measured behavioral outcome, not a fitted value or self-citation.

full rationale

The paper's central claim is an empirical behavioral result: a within-subject study with 15 BLV participants measured the number of claims participants flagged as unreliable under three presentation conditions. No quantity in the study is fitted to the outcome it is then said to predict, and no load-bearing result is imported by self-citation. The design space and prototype are explicitly built from prior work (including the authors' own GenAssist and long-form-description papers), but those citations establish context and design rationale, not the empirical outcome; the 4.9x finding is measured participant behavior, and the paper transparently reports false positives and opinionated claims (Table 4), which are validity or construct concerns rather than circular reductions. The concern that the variation summary supplies items participants later flag may reduce the construct validity of the 'ability to identify unreliable claims' measure, but the outcome is not equivalent to the input by definition or by any equation; therefore no circularity step meets the evidentiary bar.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The main assumptions are domain-level: that inconsistencies across MLLM responses are useful reliability signals, that the automatic aggregation pipeline preserves information, and that the small-sample user study reflects real-world use. The hand-chosen study parameters (3 models, 3 trials, 9 images, 4-minute time limit) are free parameters of the evaluation, not fitted constants.

free parameters (2)
  • Number of models and trials (3 models x 3 trials = 9 descriptions) = 3 models, 3 trials, 9 descriptions
    The authors chose this configuration to 'harness intra-model and inter-model variations while balancing computational costs and minimizing cognitive load.' This is a hand-chosen setting of the intervention, not fitted to the outcome, but it directly affects the variation signal shown to participants.
  • Ground-truth label of 'unreliable' claims = Not specified as a numeric parameter; categories: incorrect, speculative, opinionated
    Labeling of which claims are (un)reliable is a human-coded judgment used as the basis for counting true positives. Its calibration affects the 4.9x result and is not supplied as a separate validated measurement.
assumptions (3)
  • domain assumption Inconsistency among multiple MLLM responses is a reliable cue to potential unreliability of a specific claim.
    This is the central premise of the intervention, stated in the introduction and evaluated in the study. It is supported by the study's qualitative results (e.g., 94-96% of flagged claims were based on inconsistency) but is not always true: unique mentions can be correct, and unanimity can be confidently wrong.
  • ad hoc to paper The MLLM-generated summaries produced by Gemini 2.5 Pro preserve the information content of the 9 source descriptions without introducing new errors.
    The variation-aware descriptions and summaries are automatically generated by Gemini 2.5 Pro using Chain-of-Thought and few-shot prompts (Section 3.2.2). The paper does not evaluate the fidelity of this aggregation step against the original 9 descriptions.
  • domain assumption BLV users' self-reported identification of unreliable claims, and their Likert ratings, reflect genuine changes in their ability and trust rather than task demand characteristics.
    The study is a within-subject design where participants can infer the hypothesis (variations help). The verbal protocol and ranking questions may be partly influenced by the visible presence of different presentation conditions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions." pith.science (2026). https://pith.science/paper/MH6KP25O

@misc{pith2026250715692,
  author       = {Pith},
  title        = {Pith review of: Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MH6KP25O}},
  note         = {Machine review of arXiv:2507.15692}
}
read the original abstract

Multimodal large language models (MLLMs) provide new opportunities for blind and low vision (BLV) people to access visual information in their daily lives. However, these models often produce errors that are difficult to detect without sight, posing safety and social risks in scenarios from medication identification to outfit selection. While BLV MLLM users use creative workarounds such as cross-checking between tools and consulting sighted individuals, these approaches are often time-consuming and impractical. We explore how systematically surfacing variations across multiple MLLM responses can support BLV users to detect unreliable information without visually inspecting the image. We contribute a design space for eliciting and presenting variations in MLLM descriptions, a prototype system implementing three variation presentation styles, and findings from a user study with 15 BLV participants. Our results demonstrate that presenting variations significantly increases users' ability to identify unreliable claims (by 4.9x using our approach compared to single descriptions) and significantly decreases perceived reliability of MLLM responses. 14 of 15 participants preferred seeing variations of MLLM responses over a single description, and all expressed interest in using our system for tasks from understanding a tornado's path to posting an image on social media.

Figures

Figures reproduced from arXiv: 2507.15692 by the authors.

Figure 1
Figure 1. Automatic variation-aware description and varia [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (A) Input image and prompt. (B) Raw image descriptions from 3 MLLMs ( [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Images used in the study with corresponding name, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Left: average identified unreliable claims reported by participants overall and in each image category. Right: average [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Participants’ preference on variation presentation [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Participants’ preference on support indicator styles. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

97 extracted references · 40 canonical work pages

  1. [1]

    Privacy - Labels

    2022. Privacy - Labels. https://www.apple.com/privacy/labels/

    2022

  2. [2]

    Be My AI

    2024. Be My AI. https://www.bemyeyes.com/be-my-ai. A visual assistance tool for blind and low-vision users, part of the Be My Eyes app

    2024

  3. [3]

    Envision AI

    2024. Envision AI. https://www.letsenvision.com. AI-powered tool for assisting visually impaired users with real-world tasks through a smartphone app

    2024

  4. [4]

    2024. Gemini. https://www.google.com/gemini. An AI tool developed by Google for advanced conversational capabilities

    2024

  5. [5]

    Aira: Connecting People Who Are Blind with Remote Agents

    2025. Aira: Connecting People Who Are Blind with Remote Agents. https://aira.io

    2025

  6. [6]

    Be My Eyes: Connecting Blind and Low Vision Users with Volunteers

    2025. Be My Eyes: Connecting Blind and Low Vision Users with Volunteers. https://www.bemyeyes.com Accessed: 2024-11-13

    2025

  7. [7]

    Claude 3.7 Sonnet and Claude Code

    2025. Claude 3.7 Sonnet and Claude Code. https://www.anthropic.com/news/ claude-3-7-sonnet

    2025

  8. [8]

    Introducing Operator

    2025. Introducing Operator. https://openai.com/index/introducing-operator/

    2025

Show all 97 references
  1. [9]

    2025. Taxy AI. https://taxy.ai/ Accessed: 2025-07-13

    2025

  2. [10]

    Ali Abdolrahmani, William Easley, Michele Williams, Stacy Branham, and Amy Hurst. 2017. Embracing Errors: Examining How Context of Use Impacts Blind Individuals’ Acceptance of Navigation Aid Errors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing System...

    2017

  3. [12]

    I look at it as the king of knowledge

    Rudaiba Adnin and Maitraye Das. 2024. “I look at it as the king of knowledge”: How Blind People Use and Understand Generative AI Tools. ACM ASSETS 16, 54 (2024), 92

    2024

  4. [13]

    Rahaf Alharbi, Pa Lor, Jaylin Herskovitz, Sarita Schoenebeck, and Robin N. Brewer

  5. [14]

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016)

    2016 arXiv

  6. [15]

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L Glassman. 2024. Chainforge: A visual toolkit for prompt engineering and llm hypothesis testing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1–18

    2024

  7. [16]

    Akari Asai and Hannaneh Hajishirzi. 2020. Logic-Guided Data Augmentation and Regularization for Consistent Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 5642–5...

    2020 doi

  8. [17]

    Mauro Avila, Katrin Wolf, Anke Brock, and Niels Henze. 2016. Remote assistance for blind users in daily life: A survey about be my eyes. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments. 1–2

    2016

  9. [18]

    Karim Benharrak, Tim Zindulka, Florian Lehmann, Hendrik Heuer, and Daniel Buschek. 2024. Writer-defined AI personas for on-demand feedback generation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

    2024

  10. [19]

    Cynthia L Bennett, Jane E, Martez E Mott, Edward Cutrell, and Meredith Ringel Morris. 2018. How teens with visual impairments take, edit, and share photos on social media. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12

    2018

  11. [20]

    It’s Complicated

    Cynthia L. Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P. Bigham, Anhong Guo, and Alexandra To. 2021. “It’s Complicated”: Negotiating Accessibil- ity and (Mis)Representation in Image Descriptions of Race, Gender, and Disability. In Proceedings of the 2021 CHI Confe...

    2021

  12. [21]

    Jeffrey P Bigham and Patrick Carrington. [n. d.]. Learning from the Front: People with Disabilities as Early Adopters of AI. ([n. d.])

  13. [22]

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User ...

    2010

  14. [23]

    Burton, Erin Brady, Robin Brewer, Callie Neylan, Jeffrey P

    Michele A. Burton, Erin Brady, Robin Brewer, Callie Neylan, Jeffrey P. Bigham, and Amy Hurst. 2012. Crowdsourcing subjective fashion advice using VizWiz: challenges and opportunities. In Proceedings of the 14th International ACM SIGAC- CESS Conference on Computers and Accessib...

    2012

  15. [24]

    Ruei-Che Chang, Yuxuan Liu, and Anhong Guo. 2024. WorldScribe: Towards Context-Aware Live Visual Descriptions. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA)(UIST ’24). Association for Computing Machinery, New Yo...

    2024

  16. [25]

    Ruei-Che Chang, Yuxuan Liu, Lotus Zhang, and Anhong Guo. 2024. EditScribe: Non-Visual Image Editing with Natural Language Verification Loops. https: //doi.org/10.1145/3663548.3675599 arXiv:2408.06632 [cs]

    2024

  17. [26]

    Cochran, and Yuhang Zhao

    Ruijia Chen, Junru Jiang, Pragati Maheshwary, Brianna R. Cochran, and Yuhang Zhao. 2025. VisiMark: Characterizing and Augmenting Landmarks for People with Low Vision in Augmented Reality to Support Indoor Navigation. https: //doi.org/10.1145/3706598.3713847 arXiv:2502.10561 [cs]

    2025

  18. [27]

    Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. 2024. Relic: Investigating large language model responses using self-consistency. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–18

    2024

  19. [28]

    Yoonseo Choi, Eun Jeong Kang, Seulgi Choi, Min Kyung Lee, and Juho Kim. 2025. Proxona: Supporting Creators’ Sensemaking and Ideation with LLM-Powered Audience Personas. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25) . Association for ...

    2025

  20. [29]

    Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. LM vs LM: Detect- ing Factual Errors via Cross Examination. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for C...

    2023 doi

  21. [30]

    Microsoft Corporation. 2017. Seeing AI: An App for the Visually Impaired. https://www.microsoft.com/en-us/ai/seeing-ai Accessed: 2024-11-13

    2017

  22. [31]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics 9 (2021), 1012–1031

    2021

  23. [32]

    Rachel Gardner, Maya Varma, Clare Zhu, and Ranjay Krishna. 2020. Determining question-answer plausibility in crowdsourced datasets using multi-task learning. arXiv preprint arXiv:2011.04883 (2020)

    2020 arXiv

  24. [33]

    Kummerfeld, and Elena L

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. http://arxiv.org/abs/2401.13726 arXiv:2401.13726 [cs]

    2024 arXiv

  25. [34]

    Chilton, and Jeffrey P

    Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia B. Chilton, and Jeffrey P. Bigham. 2019. Making Memes Accessible. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’19). Association for Computing Machinery, ...

    2019

  26. [35]

    Ricardo E Gonzalez Penuela, Jazmin Collins, Cynthia Bennett, and Shiri Azenkot

  27. [36]

    Kummerfeld, and Elena L

    Ziwei Gu, Ian Arawjo, Kenneth Li, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. An AI-Resilient Text Rendering Technique for Reading and Skimming Documents. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). As...

    2024

  28. [37]

    In Proceedings of the CHI Conference on Human Factors in Computing Systems

    Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–21

  29. [38]

    Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

    Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. arXiv:1802.08218 [cs.CV] https://arxiv.org/ abs/1802.08218

    2018 arXiv

  30. [39]

    Ananya Gubbi Mohanbabu and Amy Pavel. 2024. Context-Aware Image Descrip- tions for Web Accessibility. InThe 26th International ACM SIGACCESS Conference on Computers and Accessibility . 1–17

    2024

  31. [40]

    Jonggi Hong and Hernisa Kacorri. 2024. Understanding How Blind Users Handle Object Recognition Errors: Strategies and Challenges. arXiv preprint arXiv:2408.03303 (2024)

    2024 arXiv

  32. [41]

    Jaylin Herskovitz, Andi Xu, Rahaf Alharbi, and Anhong Guo. 2023. Hacking, switching, combining: understanding and supporting DIY assistive technology design by blind people. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–17

    2023

  33. [42]

    Mina Huh, Fangyuan Xu, Yi-Hao Peng, Chongyan Chen, Hansika Murugu, Danna Gurari, Eunsol Choi, and Amy Pavel. 2024. Long-Form Answers to Visual Ques- tions from Blind and Low Vision People. arXiv preprint arXiv:2408.06303 (2024). ASSETS ’25, October 26–29, 2025, Denver, CO, USA...

    2024 arXiv

  34. [43]

    Mina Huh, Yi-Hao Peng, and Amy Pavel. 2023. GenAssist: Making image gen- eration accessible. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–17

    2023

  35. [45]

    Mina Huh, Saelyne Yang, Yi-Hao Peng, Xiang’Anthony’ Chen, Young-Ho Kim, and Amy Pavel. 2023. AVscript: Accessible Video Editing with Audio-Visual Scripts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17

    2023

  36. [46]

    Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W. Reeder

  37. [47]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran- Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fo...

    2022 doi arXiv

  38. [48]

    Satwik Ram Kodandaram, Utku Uckun, Xiaojun Bi, IV Ramakrishnan, and Vikas Ashok. 2024. Enabling uniform computer interaction experience for blind users through large language models. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibili...

    2024

  39. [49]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Lin- guistic Invariances for Uncertainty Estimation in Natural Language Generation. (2023)

    2023

  40. [50]

    Matthias Kellner and Michele Ceriotti. 2024. Uncertainty quantification by direct propagation of shallow ensembles. https://doi.org/10.48550/arXiv.2402.16621 arXiv:2402.16621 [physics] version: 1

    2024 doi arXiv

  41. [51]

    Jaewook Lee, Jaylin Herskovitz, Yi-Hao Peng, and Anhong Guo. 2022. Image- Explorer: Multi-Layered Touch Exploration to Encourage Skepticism Towards Imperfect AI-Generated Image Captions. In Proceedings of the 2022 CHI Confer- ence on Human Factors in Computing Systems (New Orl...

    2022

  42. [52]

    Yoonjoo Lee, Kihoon Son, Tae Soo Kim, Jisu Kim, John Joon Young Chung, Eytan Adar, and Juho Kim. 2024. One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations. In The 2024 ACM Conference on Fairness, Accountability, and Transpar...

    2024

  43. [53]

    Masaki Kuribayashi, Kohei Uehara, Allan Wang, Shigeo Morishima, and Chieko Asakawa. 2025. WanderGuide: Indoor Map-less Robotic Guide for Exploration by Blind People. https://doi.org/10.1145/3706598.3713788 arXiv:2502.08906 [cs]

    2025

  44. [54]

    Lipton, and Liu Leqi

    Xinyu Li, Ruiyang Zhou, Zachary C. Lipton, and Liu Leqi. 2024. Personalized Language Modeling from Personalized Human Feedback. https://doi.org/10. 48550/arXiv.2402.05133 arXiv:2402.05133 [cs]

    2024 doi arXiv

  45. [55]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (E...

    2023 doi

  46. [56]

    Tomer Levinboim, Ashish V Thapliyal, Piyush Sharma, and Radu Soricut. 2019. Quality estimation for image captions based on large-scale human evaluations. arXiv preprint arXiv:1909.03396 (2019)

    2019 arXiv

  47. [57]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 (2024)

    2024 arXiv

  48. [58]

    Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell

  49. [59]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models. http://arxiv. org/abs/2305.19187 arXiv:2305.19187 [cs, stat]

    2024 arXiv

  50. [60]

    Meredith Ringel Morris. 2020. AI and Accessibility: A Discussion of Ethical Considerations. Commun. ACM 63, 6 (May 2020), 35–37. https://doi.org/10.1145/ 3356727 arXiv:1908.08939 [cs]

    2020 arXiv

  51. [61]

    Vishnu Nair, Hanxiu ’Hazel’ Zhu, and Brian A. Smith. 2023. ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23)....

    2023

  52. [62]

    Zheng Ning, Brianna L Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, and Toby Jia-Jun Li. 2024. SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers. In Proceedings of the CHI Conference on Hu...

    2024

  53. [63]

    Jennifer Mankoff, Janice Light, James Coughlan, Christian Vogler, Abraham Glasser, Gregg Vanderheiden, and Laura Rice. 2025. Accessibility Considerations in the Development of an AI Action Plan. https://doi.org/10.48550/arXiv.2503. 14522 arXiv:2503.14522 [cs]

    2025 doi arXiv

  54. [64]

    Amy Pavel, Gabriel Reyes, and Jeffrey P. Bigham. 2020. Rescribe: Authoring and Automatically Editing Audio Descriptions. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machine...

    2020

  55. [65]

    Yi-Hao Peng, Ding Li, P Jeffrey Bigham, and Amy Pavel. 2025. Morae: Proactively Pausing UI Agents for User Choices. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology

    2025

  56. [66]

    Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 6174–6184

    2019

  57. [67]

    OpenAI. 2024. ChatGPT. https://www.openai.com/chatgpt. Accessed: 2024-11-11

    2024

  58. [68]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. InProceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). A...

    2023

  59. [69]

    Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. 2024. Quanti- fying uncertainty in natural language explanations of large language models. In International Conference on Artificial Intelligence and Statistics. PMLR, 1072–1080

    2024

  60. [70]

    Tess Van Daele, Akhil Iyer, Yuning Zhang, Jalyn C Derry, Mina Huh, and Amy Pavel. 2024. Making Short-Form Videos Accessible with Hierarchical Video Summaries. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–17

    2024

  61. [71]

    Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolu...

    2024

  62. [72]

    Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael Bernstein, and Ranjay Krishna. 2023. Explanations Can Reduce Overreliance on AI Systems During Decision-Making. https://doi.org/ 10.48550/arXiv.2212.06823 arXiv:2212.06823 [cs]

    2023 doi arXiv

  63. [73]

    W3C. 2024. Web Accessibility Initiative (WAI). https://www.w3.org/WAI/

    2024

  64. [74]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    2022 arXiv

  65. [75]

    Vera Liao, and Jen- nifer Wortman Vaughan

    Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, and Jen- nifer Wortman Vaughan. 2024. Generation Probabilities Are Not Enough: Un- certainty Highlighting in AI Code Completions. https://doi.org/10.48550/arXiv. 2302.07248 arXiv:2302.07248

    2024 doi arXiv

  66. [76]

    Justin D Weisz, Jessica He, Michael Muller, Gabriela Hoefer, Rachel Miles, and Werner Geyer. 2024. Design principles for generative AI applications. In Proceed- ings of the 2024 CHI Conference on Human Factors in Computing Systems

    2024

  67. [77]

    World Health Organization. 2012. WHO releases new global estimates on visual impairment. https://www.emro.who.int/control-and-preventions-of-blindness- and-deafness/announcements/global-estimates-on-visual-impairment.html

    2012

  68. [78]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review 11, 12 (Nov. 2024). https://doi.org/10.1093/nsr/nwae403

    2024 doi

  69. [79]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://doi.org/10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs]

    2023 doi arXiv

  70. [80]

    Sukhai, and Sowmya Somanath

    Yichun Zhao, Miguel A Nacenta, Mahadeo A. Sukhai, and Sowmya Somanath

  71. [81]

    Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2018. A face recognition application for people with visual impairments: Understanding use beyond the lab. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–14. Surfacing Variations ...

    2018

  72. [83]

    Vera Liao, and Rachel K

    Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20). Association for C...

    2020

  73. [85]

    In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24)

    TADA: Making Node-link Diagrams Accessible to Blind and Low-Vision People. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3613904.3642222

    2024

  74. [87]

    – Combine atomic facts about the same subject into a single coherent sentence

    GROUP FACTS – Each description contains multiple atomic facts (self-contained claims). – Combine atomic facts about the same subject into a single coherent sentence. – If variant statements describe the same fact, concatenate using “or”

  75. [88]

    – Include all single and unique claims

    PARAGRAPH FORMATION – Merge grouped facts into comprehensive paragraphs. – Include all single and unique claims

  76. [89]

    below is

    MODEL DIFFERENCES – Annotate differences with counts: (n_A of N_A ModelA, n_B of N_B ModelB). – Example: (2 of 3 GPT, 3 of 3 Gemini). – If a model does not support a fact, omit it from the parentheses. # INPUT FORMAT Input: list of descriptions; each has atomic facts, response...

  77. [90]

    Synthesize all key observations across models in a coherent paragraph form

  78. [91]

    Start with high-level observations (image type, layout, purpose) before detailed attributes (counts, colors, labels)

  79. [92]

    Identify and group statements agreed upon across models

  80. [93]

    the shirt is blue

    In the agreements section, do not include alternate variants; choose the common canonical form (e.g., “the shirt is blue”; NOT “blue, possibly cyan”)

  81. [94]

    Clearly highlight disagreements with inline references to the differing model outputs

  82. [95]

    Note any uniquely mentioned information; attribute to the specific model(s)

  83. [96]

    Mention model names only when discussing disagreements or unique points

  84. [97]

    Provide a Markdown bullet list summarizing each section

  85. [98]

    Be as comprehensive as possible across agreements, disagreements, uniqueness

  86. [99]

    similarity

    Use only information explicitly present in the input. No inference. # REQUIRED OUTPUT CONTENT Return both a narrative summary (hierarchical markdown paragraphs with inline model agreement annotations) and a JSON object: { “similarity”: “Summary of similar points across models....

    2025

  87. [2009]

    nutrition label

    A "nutrition label" for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security (SOUPS ’09) . Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/1572532.1572538

  88. [2017]

    In proceedings of the 2017 CHI conference on human factors in computing systems

    Understanding blind people’s experiences with computer-generated cap- tions of social media images. In proceedings of the 2017 CHI conference on human factors in computing systems . 5988–5999

    2017

  89. [2024]

    In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (St

    Misfitting With AI: How Blind People Verify and Contest AI Errors. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (St. John’s, NL, Canada)(ASSETS ’24). Association for Computing Machinery, New York, NY, USA, Article 61, 17 page...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.