Pith. sign in

REVIEW 4 major objections 9 minor 89 references

Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development

T0 review · 4 major / 9 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning GPT-4o on partial DOM context and OCR text yields icon alt-text that matches human-written quality and beats full-screen vision-language models.

desk verdict Solid engineering contribution with a real deployment-time niche, but the SOTA claim leans on a noisy gold standard and the human-study evidence is too thin. read the letter →

arxiv 2504.13069 v1 pith:4PP6UFZO submitted 2025-04-17 cs.SE cs.HC

Sabrina Haque , Christoph Csallner This is my paper · ORCID
classification cs.SEcs.HC
keywords alt-textgenerationUIiconaccessibilityscreenreadersfine-tunedGPT-4ovision-languagemodelsAndroiddevelopmentDOMcontextOCR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AltIcon aims to move accessibility fixes for mobile apps from post-launch cleanup to the moment a developer adds an icon. The paper claims that two fine-tuned GPT-4o variants, one text-only and one multi-modal, generate alt-text for UI icons from partial screen information: the icon's DOM entry, its parent and siblings, and OCR text inside the icon. The generated descriptions score higher than existing icon-labeling tools and full-screen vision-language models on standard captioning metrics. A small human study rated the multi-modal variant on par with human-written labels. If right, screen reader users would get meaningful icon descriptions earlier, and developers would not need to wait for a complete screen.

What carries the argument

The load-bearing mechanism is the DOM-derived icon context packed into a structured prompt: activity name, icon class and resource ID, parent node, sibling nodes, OCR-extracted in-icon text, and either a zero-shot icon-only label (for the text variant) or the icon image itself (for the multi-modal variant). The prompt asks the fine-tuned model for a short, descriptive alt-text that conveys functionality to a vision-impaired user and avoids generic words like 'button' or 'icon.' This local context substitutes for the full-screen screenshot that prior models need, and the ablation shows that local DOM context, not pixels, carries most of the signal.

What would settle it

Take a set of icons whose true function is confirmed by the app's developer or by the app's source code, especially the vague cases where WC20 annotators disagreed, and ask screen-reader users which generated description lets them complete the intended action. If AltIcon-MMT no longer beats PaliGemma on that function-confirmed set, the claim that it generates higher-quality alt-text for users would be refuted, even if it still matches crowd labels.

Watch

Extended reading notes

Core claim

On a filtered subset of 1,635 WC20 icons, AltIcon-MMT, which fine-tunes GPT-4o with the icon image plus DOM and OCR context, reaches CIDEr 138.3 and SPICE 23.2, ahead of PaliGemma in full-screen mode (127.1 CIDEr, 21.8 SPICE) and all other baselines. AltIcon-TextT, the cheaper text-only variant, still beats every baseline with CIDEr 134.3 and SPICE 22.3. In the RQ4 user study, AltIcon-MMT averaged 4.1 on a 1-5 scale, statistically indistinguishable from the human-written ground truth, while PaliGemma scored 3.6. The paper argues that the key is context: identical icons have different functions, and the DOM tree plus in-icon text supplies the function during development when no full screenshot exists. Ablations show parent and sibling DOM information contributes the most, followed by the icon's resource ID and then OCR text.

Load-bearing premise

The evaluation treats WC20's crowd-written reference captions as the correct alt-text for every icon, even though the paper finds those captions sometimes conflict and sometimes disagree with the icon's DOM meaning; if the references are wrong, the reported rankings may not reflect real accessibility benefit.

Editorial extensions

If this is right

  • Developers can receive a usable alt-text suggestion at the moment they add an icon, rather than after a screen is finished or after release, reducing context switching and technical debt.
  • Static analysis tools such as Android Lint could call AltIcon to propose a concrete contentDescription fix instead of only flagging the missing attribute.
  • The text-only variant offers a low-cost path: roughly $1.51 to $2.37 inference cost on the 1,635-icon test set and cheaper fine-tuning than the multi-modal variant, making high-quality labels feasible for budget-constrained teams.
  • Because AltIcon works from partial screens, it applies during wireframing and in-progress UI code, the stage that 44% of surveyed developers said they preferred.
  • AltIcon can also run in batch mode to retro-inject alt-text into existing layout files, giving a migration path for apps that already shipped without labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same context-packaging recipe could transfer to Jetpack Compose and to non-icon widgets such as buttons, text inputs, and images, since the prompt structure is framework-agnostic; the paper lists these as future engineering, not evaluated results.
  • If resource IDs are as informative as the ablation suggests, developer naming conventions are an untapped signal: projects with meaningful IDs may need little or no image input, while projects with opaque IDs such as 'button1' would depend more on OCR and pixels.
  • The paper does not resolve whether 'on par with human annotations' understates true quality when the crowd is wrong and the DOM context is right; a developer-confirmed oracle would settle which source is more trustworthy.
  • A direct test of real accessibility benefit would measure whether injecting AltIcon labels early improves screen-reader task completion times, not just agreement with reference captions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. The paper proposes AltIcon, a development-time approach for generating alt-text for mobile UI icons. AltIcon extracts textual context from the Android DOM tree (activity name, resource IDs, parent and sibling node properties), adds OCR text from the icon image, and optionally the icon image itself, then feeds this into a fine-tuned GPT-4o model. Two variants are presented: a text-only fine-tuned model (AltIcon-TextT) and a multimodal fine-tuned model (AltIcon-MMT). The authors conduct a formative survey on developer preferences, evaluate the method against LabelDroid, Coala, Pix2Struct, PaliGemma, and zero-shot GPT-4o on a filtered subset of the WC20 dataset using standard captioning metrics, and run two additional user studies (RQ4 usability rating and RQ5 plugin perception). The central claim is that AltIcon generates higher-quality alt-text than the baselines, especially in partial-screen settings, and that AltIcon-MMTi is rated on par with human-written alt-text.

Significance. If the results hold, the paper makes a useful contribution by showing that off-the-shelf large language models can be adapted to generate context-aware icon alt-text during development, addressing an accessibility gap that current tools handle only post-hoc. The strengths are the focus on partial-screen inputs, the direct comparison of multiple baselines on the same test icons, the inclusion of component ablations and cost analysis, and the public replication package. The main risk is that the reported superiority is partly due to giving AltIcon DOM text that the pixel-only baselines cannot see, and the human-study evidence for parity with human-written alt-text is based on a small, confounded evaluation. The claims are plausible but need additional controls and more careful qualification.

major comments (4)
  1. [Section 5.1, Tables 3 and 4] The comparison against baselines is not feature-controlled. AltIcon receives DOM tree text (resource IDs, parent/sibling text) and OCR text, while the pixel-only baselines (LabelDroid, Pix2Struct, PaliGemma) see only images. Table 4 shows that removing parent/sibling DOM information reduces CIDEr by about 8 points for both AltIcon variants, indicating that the DOM text is a major contributor to the reported advantage. The claim that AltIcon is 'SOTA' is therefore substantially a feature-engineering result. To support the state-of-the-art claim, the paper should add a baseline that also has access to the same textual features (e.g., a text-only LLM with DOM context, or a VLM fed with OCR output) and report the comparison under matched input features.
  2. [Section 5.5, RQ4] The user study does not support the claim that AltIcon-MMTi is 'on par with human-written alt-text.' The study uses only four raters, one of whom is an author, and 50 screens. The 'ground truth' option is one of three WC20 crowd labels selected at random, and the paper itself notes that these labels can conflict (Figure 4c, Section 5.7.4). The Wilcoxon p-values (0.47 and 0.39) only show that no significant difference was detected in this small sample; they are not equivalence tests. The survey also provided the app name and a red bounding box, giving raters additional context that the original WC20 annotators lacked, which may favor the context-aware AltIcon outputs. This evidence is too weak to carry the parity claim; either the claim should be downgraded or the study should be replicated with a larger, independent panel and an equivalence design.
  3. [Section 5.1, Table 2 and RQ1] The automated metrics are computed against multiple ground-truth labels per icon (up to three, totaling 4,419 labels for 1,635 test icons), but the paper does not specify how multi-reference aggregation was performed for BLEU, ROUGE, METEOR, CIDEr, and SPICE. Different implementations handle multiple references differently (e.g., averaging per-reference scores vs. pooling all references), which materially affects the numerical results and the comparability with the baselines' originally published scores. This should be stated explicitly.
  4. [Section 4.1.2 and RQ2] AltIcon-TextT is not truly text-only: it relies on a zero-shot GPT-4o label-inference step to convert the icon image into a text label. The paper reports the accuracy of the earlier EfficientNet classifier (94.4% test accuracy) but does not evaluate the accuracy or reliability of the GPT-4o labeler, which is the component actually used in the reported results. Since a wrong inferred label propagates into the text-only fine-tuned model, the cost-performance trade-off in RQ2 is not fully interpretable without this information.
minor comments (9)
  1. [Abstract and throughout] The abstract uses 'ALTICON' in one place while the rest of the paper uses 'AltIcon'; please standardize the spelling.
  2. [Section 4.3] The text contains a typo: 'wastes time and and reduces adoption' should be 'wastes time and reduces adoption.'
  3. [References] Reference [41] duplicates reference [40] (both are the ILuvUI paper); the duplicate should be removed or replaced with a distinct reference.
  4. [Table 3] The table header includes 'Fine-tune' and 'Infer' columns with values in USD, but the caption does not explain that these are cost columns or how the costs were computed; please clarify.
  5. [Section 5.5] The survey is described as 'double-anonymous,' but one of the four raters is an author of the paper. Please clarify what blinding was actually applied and how the author's participation was handled in the analysis.
  6. [Section 3 and Section 4] The paper states that 'six out of 10 developers' requested a plugin for the current development screen, but Table 1 reports only 44% selecting 'Screen UI code in-progress.' These numbers should be reconciled or the statement should be reworded to match the data.
  7. [Figure 4 caption] The caption says 'green box, not marked in survey or tool input,' which is confusing in the context of the figure; clarify whether the green box is part of the figure annotation only.
  8. [Section 5.1] The paper should note whether the HuggingFace Pix2Struct and PaliGemma baselines, which are fine-tuned on 40k WC20 elements, may have seen the same test screens during their fine-tuning; a brief statement about this potential overlap would help interpret the comparison.
  9. [Section 5.7.4] The sentence 'human annotators where confused' should be 'human annotators were confused.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AltIcon is fine-tuned on WC20 training labels and evaluated on held-out WC20 test labels; comparisons and ablations are self-contained.

full rationale

The paper's central derivation is an empirical pipeline: AltIcon extracts DOM/OCR/icon inputs, fine-tunes GPT-4o variants on WC20 training labels, and evaluates on the held-out WC20 test split (Table 2: 18,176 training icon labels vs. 4,419 test labels). No fitted parameter is renamed as a prediction; the test outputs are generated from inputs the model did not see during fine-tuning. The ablations (Table 4) remove components and measure score drops, which is a genuine causal decomposition, not a circular reduction. The RQ4 user study compares outputs of held-out models against WC20 test labels; although the gold standard is the same annotation style the model was trained to imitate and the study is small (four raters, one author), this is a benchmarking-validity concern, not a derivation-to-fit circularity under the rules: the human ratings are not used to fit AltIcon. Self-citations are limited to the replication package reference; no load-bearing claim reduces to an unverified self-citation. The 'first formative study' and 'first method' novelty statements are scope claims, not derivations. Therefore no circular step can be exhibited by quoting an equation or construction that equates output to input.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new physical or mathematical entities. Its claim rests on dataset assumptions (WC20 as ground truth), on the informativeness of DOM strings, on proprietary model stability, and on several author-chosen hyperparameters (epochs, resolution, per-class cap, length constraint) that are not independently justified.

free parameters (4)
  • Maximum icons per class for fine-tuning = 15
    Training set of 1,425 icons sampled with up to 15 per class from 99 icon classes; this cap affects model performance and was chosen by the authors.
  • Fine-tuning epoch count = 3
    All AltIcon variants fine-tuned for 3 epochs; no early stopping or epoch tuning is reported, and this affects final quality.
  • Icon input resolution after super-resolution = 128x128
    Icons resized with Real-ESRGAN to 128x128; affects both the multimodal model input and OCR results.
  • Output length constraint in prompt = 2-7 words
    The prompt instructs the model to produce 2-7 words; this length choice interacts with n-gram metrics such as BLEU, ROUGE, and CIDEr, and with what screen reader users need.
assumptions (3)
  • domain assumption WC20 third-party human annotations are a valid gold standard for icon alt-text quality.
    Section 5 states that alignment with WC20 annotations is the measurement of quality; the paper itself notes these annotations can conflict or be confused (Figure 4c, Section 5.7.4), so the assumption is load-bearing.
  • domain assumption Developer-written resource IDs, parent, and sibling DOM text in real Android layouts carry reliable semantic signal about icon function.
    AltIcon's main inputs are these strings; the ablation in Table 4 shows the largest performance drop when parent and sibling DOM info is removed, so the method depends on this premise.
  • domain assumption GPT-4o fine-tuning via the OpenAI API is a stable, reproducible experimental object.
    All training and inference use gpt-4o-2024-08-06 via API; the paper does not address model updates, nondeterminism, or API deprecation, yet the exact scores depend on this external service.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development." pith.science (2026). https://pith.science/paper/4PP6UFZO

@misc{pith2026250413069,
  author       = {Pith},
  title        = {Pith review of: Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PP6UFZO}},
  note         = {Machine review of arXiv:2504.13069}
}
read the original abstract

Alt-text is essential for mobile app accessibility, yet UI icons often lack meaningful descriptions, limiting accessibility for screen reader users. Existing approaches either require extensive labeled datasets, struggle with partial UI contexts, or operate post-development, increasing technical debt. We first conduct a formative study to determine when and how developers prefer to generate icon alt-text. We then explore the ALTICON approach for generating alt-text for UI icons during development using two fine-tuned models: a text-only large language model that processes extracted UI metadata and a multi-modal model that jointly analyzes icon images and textual context. To improve accuracy, the method extracts relevant UI information from the DOM tree, retrieves in-icon text via OCR, and applies structured prompts for alt-text generation. Our empirical evaluation with the most closely related deep-learning and vision-language models shows that ALTICON generates alt-text that is of higher quality while not requiring a full-screen input.

Figures

Figures reproduced from arXiv: 2504.13069 by the authors.

Figure 1
Figure 1. Example zoom out (left) vs. lower volume (right) minus buttons in Rico. Icons’ widespread use in mobile apps can be attributed to their ability to convey in￾formation effectively while consuming min￾imal screen space. Inferring meaning￾ful alt-text for icons is challenging be￾cause identical vi￾sual representations can serve different functions depending on their context. With￾out considering sur￾rounding UI element… view at source ↗
Figure 3
Figure 3. shows two example icons containing text that would be helpful for alt-text generation but is not available in the DOM tree. AltIcon successfully extracts the texts (“Live” and “Quote”) and adds them to the respective icon contexts [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Sample double-anonymous user survey screens (RQ4): target icon (red box), icon’s parent (green box, not marked in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Screens where AltIcon infers sub-optimal alt-text; target icon (red box), icon’s parent (green box, not marked for tools), app name: ground truth (ref), PaliGemma-448c (PG), AltIcon-TextT (TextT), AltIcon-MMTi (MMT), TextT & MMT (AltIcon). 5.7.1 Icon Context Misleads I…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 61 canonical work pages

  1. [1]

    Josh Achiam et al . 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774

    arXiv 2024

  2. [2]

    ADA.gov. 2023. The Americans with Disabilities Act (ADA). https://www.ada. gov/

    2023

  3. [3]

    Ali S Alotaibi, Paul T Chiou, Fazle M Tawsif, and William GJ Halfond. 2023. ScaleFix: An Automated Repair of UI Scaling Accessibility Issues in Android Applications. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 147–159

    2023

  4. [4]

    Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020. Accessibility issues in Android apps: State of affairs, sentiments, and ways forward. InProc. ACM/IEEE 42nd International Conference on Software Engineering (ICSE) . 1323–1334

    2020

  5. [5]

    Abdulaziz Alshayban and Sam Malek. 2022. AccessiText: Automated detection of text accessibility issues in Android apps. In Proc. 30th Symposium on the Foundations of Software Engineering (FSE) . 984–995

    2022

  6. [6]

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic Propositional Image Caption Evaluation. In Proc. 14th European Conference on Computer Vision (ECCV) . Springer, 382–398

    2016

  7. [7]

    Android. 2024. Lint:Improve your code with lint checks. Accessed March 1, 2024. https://developer.android.com/studio/write/lint

    2024

  8. [8]

    Android Studio. 2025. Improve your code with lint checks. https://developer. android.com/studio/write/lint

    2025

Show all 89 references
  1. [9]

    Android Studio. 2025. Layouts in views. https://developer.android.com/develop/ ui/views/layout/declaring-layout

    2025

  2. [10]

    Rohan Anil et al. 2024. Gemini: A family of highly capable multimodal models. arXiv:2312.11805 [cs.CL] https://arxiv.org/abs/2312.11805

    2024 arXiv

  3. [11]

    Anonymous. 2025. Replication package: Code and data for this paper. https: //figshare.com/s/6452b23b9589f747c540

    2025

  4. [12]

    Gilles Baechler et al. 2024. ScreenAI: A vision-language model for UI and info- graphics understanding. arXiv preprint arXiv:2402.04615 (2024)

    2024 arXiv

  5. [13]

    Aleksander Bai, Heidi Camilla Mork, and Viktoria Stray. 2017. A cost-benefit analysis of accessibility testing in agile software development: results from a multiple case study. International Journal on Advances in Software 10, 1&2 (2017), 96–107

    2017

  6. [14]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computation...

    2005

  7. [15]

    Lucas Beyer et al . 2024. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726 (2024)

    2024 arXiv

  8. [16]

    Tingting Bi, Xin Xia, David Lo, John Grundy, Thomas Zimmermann, and Denae Ford. 2022. Accessibility in software practice: A practitioner’s perspective. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 4 (2022), 1–26

    2022

  9. [17]

    Yuxiang Chai et al. 2024. AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents. arXiv:2407.17490 [cs.HC] https://arxiv.org/abs/2407.17490

    2024 arXiv

  10. [18]

    Jieshan Chen et al . 2020. Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. In Proc. ACM/IEEE 42nd International Conference on Software Engineering (ICSE) . ACM, 322–334

    2020

  11. [19]

    Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. 2022. Towards complete icon labeling in mobile applications. In Proc. CHI Conference on Human Factors in Computing Systems . ACM, 1–14. https://doi.org/10.1145/3491102.3502073

    2022

  12. [20]

    Sen Chen, Chunyang Chen, Lingling Fan, Mingming Fan, Xian Zhan, and Yang Liu

  13. [21]

    Xi Chen et al. 2023. Pali-3 vision language models: Smaller, faster, stronger.arXiv preprint arXiv:2310.09199 (2023)

    2023 arXiv

  14. [22]

    Pei-Yu Chi, Sen-Po Hu, and Yang Li. 2018. Doppio: Tracking ui flows and code changes for app development. In Proc. 2018 CHI Conference on Human Factors in Computing Systems. 1–13

    2018

  15. [23]

    Biplab Deka et al. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proc. 30th annual ACM symposium on User Interface Software and Technology (UIST). 845–854

    2017

  16. [24]

    Giovanni Delnevo, Manuel Andruccioli, and Silvia Mirri. 2024. On the Interaction with Large Language Models for Web Accessibility: Implications and Challenges. In Proc. 21st IEEE Consumer Communications & Networking Conference (CCNC) . IEEE, 1–6

    2024

  17. [25]

    Marianna Di Gregorio, Dario Di Nucci, Fabio Palomba, and Giuliana Vitiello

  18. [26]

    DigitalA11Y. 2025. Shift Left Accessibility in Design, Development and Testing. https://www.digitala11y.com/connecting-dots-of-an-accessibility-audit/

    2025

  19. [27]

    Sidong Feng, Suyu Ma, Han Wang, David Kong, and Chunyang Chen. 2024. Mud: Towards a large-scale and noise-filtered ui dataset for modern style ui modeling. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1–14

    2024

  20. [28]

    Sidong Feng, Suyu Ma, Jinzhong Yu, Chunyang Chen, Tingting Zhou, and Yankun Zhen. 2021. Auto-Icon: An automated code generation tool for icon designs assisting in US development. In Proc. 26th International Conference on Intelligent User Interfaces (IUI). 59–69

    2021

  21. [29]

    FLATICON. 2024. Access 14.8M+ vector icons & stickers. https://www.flaticon. com/

    2024

  22. [30]

    Raymond Fok, Mingyuan Zhong, Anne Spencer Ross, James Fogarty, and Jacob O Wobbrock. 2022. A Large-Scale Longitudinal Analysis of Missing Label Accessi- bility Failures in Android Apps. In Proc. 2022 CHI Conference on Human Factors in Computing Systems. 1–16

    2022

  23. [31]

    GitHub. 2023. EasyOCR. https://github.com/JaidedAI/EasyOCR

    2023

  24. [32]

    GitHub. 2023. Microsoft COCO Caption Evaluation. https://github.com/tylin/ coco-caption

    2023

  25. [33]

    Google. 2024. Get started on Android with TalkBack. https://support.google. com/accessibility/android/answer/6283677

    2024

  26. [34]

    Google. 2025. Jetpack Compose. https://developer.android.com/jetpack/compose Accessed: March 2025

    2025

  27. [35]

    Samine Hadadi. 2021. Adee: Bringing accessibility right inside design tools. In Proc. 23rd International ACM SIGACCESS Conference on Computers and Accessibil- ity. 1–4

    2021

  28. [36]

    Samine Hadadi, Zhanna Sarsenbayeva, and Judy Kay. 2023. Starting well on design for accessibility: analysis of W3C’s 167 accessibility evaluation tools for the design phase. In Proc. 25th International ACM SIGACCESS Conference on Computers and Accessibility. 1–7

    2023

  29. [37]

    Calista Huang et al. 2024. ACCESS: Prompt Engineering for Automated Web Accessibility Violation Corrections. arXiv preprint arXiv:2401.16450 (2024)

    2024 arXiv

  30. [38]

    IntelliJ. 2024. IntelliJ Platform Plugin SDK: Code Inspections. https://plugins. jetbrains.com/docs/intellij/code-inspections.html#creating-an-inspection

    2024

  31. [39]

    iOS. 2024. VoiceOver on iPhone. https://support.apple.com/guide/iphone/turn- on-and-practice-voiceover-iph3e2e415f/ios

    2024

  32. [41]

    Yue Jiang, Eldon Schoop, Amanda Swearngin, and Jeffrey Nichols. 2023. Iluvui: Instruction-tuned language-vision modeling of UIs from machine conversations. arXiv preprint arXiv:2310.04869 (2023)

    2023 arXiv

  33. [42]

    Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2016. Re-evaluating automatic metrics for image captioning. arXiv preprint arXiv:1612.07600 (2016)

    2016 arXiv

  34. [43]

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. OCR-Free Document Understanding Transformer. In Proc. 17th European Conference on Computer Vision (ECCV) . Springer, 498–517....

    2022 doi

  35. [44]

    Urvashi Kokate, Kristen Shinohara, and Garreth W Tigwell. 2022. Exploring Accessibility Features and Plug-ins for Digital Prototyping Tools. In Proc. 24th International ACM SIGACCESS Conference on Computers and Accessibility . 1–4

    2022

  36. [45]

    Kenton Lee et al. 2023. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In Proc. International Conference on Machine Learning (ICML). PMLR, 18893–18912

    2023

  37. [46]

    Tianshi Li, Yuvraj Agarwal, and Jason I Hong. 2018. Coconut: An IDE plugin for developing privacy-friendly apps. Proc. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1–35

    2018

  38. [47]

    Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020. Widget captioning: Generating natural language description for mobile user interface elements. arXiv preprint arXiv:2010.04295 (2020)

    2020 arXiv

  39. [48]

    Yichen Li, Yun Peng, Yintong Huo, and Michael R Lyu. 2024. Enhancing LLM- based coding tools through native integration of IDE-derived static context. In Proc. 1st International Workshop on Large Language Models for Code . 70–74

    2024

  40. [49]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Sum- maries. In Proc. Text summarization branches out . Association for Computational Linguistics, 74–81

    2004

  41. [50]

    Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning design semantics for mobile apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 569–579

    2018

  42. [51]

    Zhe Liu et al . 2023. Chatting with GPT-3 for zero-shot human-like mobile automated GUI testing. arXiv preprint arXiv:2305.09434 (2023)

    2023 arXiv

  43. [52]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yuekai Huang, Jun Hu, and Qing Wang. 2024. Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–20

    2024

  44. [53]

    Cláudia Mamede, Eduard Pinconschi, and Rui Abreu. 2022. A transformer-based IDE plugin for vulnerability detection. In Proc. 37th IEEE/ACM International Conference on Automated Software Engineering (ASE) . 1–4

    2022

  45. [54]

    Forough Mehralian. 2021. fmehralian/COALA: one of the first releases . https: //doi.org/10.5281/zenodo.5100853

    2021 doi

  46. [55]

    Forough Mehralian, Titus Barik, Jeff Nichols, and Amanda Swearngin. 2024. Automated Code Fix Suggestions for Accessibility Issues in Mobile Apps. arXiv preprint arXiv:2408.03827 (2024)

    2024 arXiv

  47. [56]

    Forough Mehralian, Ziyao He, and Sam Malek. 2024. Automated Accessibility Analysis of Dynamic Content Changes on Mobile Apps. In Proc. IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE, 482–494

    2024

  48. [57]

    Forough Mehralian, Navid Salehnamadi, and Sam Malek. 2021. Data-driven accessibility repair revisited: On the effectiveness of generating labels for icons in Android apps. In Proc. 29th Symposium on the Foundations of Software Engineering (FSE). ACM, 107–118

    2021

  49. [58]

    George A Miller. 1995. WordNet: a lexical database for English. Commun. ACM 38, 11 (1995), 39–41

    1995

  50. [59]

    Darliane Miranda and João Araujo. 2022. Studying industry practices of accessi- bility requirements in agile development. In Proc. 37th ACM/SIGAPP Symposium on Applied Computing. 1309–1317

    2022

  51. [60]

    Peya Mowar, Yi-Hao Peng, Aaron Steinfeld, and Jeffrey P Bigham. 2024. Tab to autocomplete: The effects of AI coding assistants on web accessibility. In Proc. 26th International ACM SIGACCESS Conference on Computers and Accessibility . 1–6

    2024

  52. [61]

    Peya Mowar, Yi-Hao Peng, Jason Wu, Aaron Steinfeld, and Jeffrey P Bigham. 2025. CodeA11y: Making AI Coding Assistants Useful for Accessible Web Development. arXiv preprint arXiv:2502.10884 (2025)

    2025 arXiv

  53. [62]

    Liming Nie et al. 2024. SoK: Detection and Repair of Accessibility Issues. arXiv preprint arXiv:2411.19727 (2024)

    2024 arXiv

  54. [63]

    OpenAI. 2024. Fine-Tuning with OpenAI Models. https://platform.openai.com/ docs/guides/fine-tuning. Accessed February 1, 2025

    2024

  55. [64]

    OpenAI. 2024. OpenAI developer platform. https://platform.openai.com/docs/ overview. Accessed February 1, 2025

    2024

  56. [65]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting of the Association for Computational Linguistics (ACL) . 311–318

    2002

  57. [66]

    Neha Patil, Dhananjay Bhole, and Prasanna Shete. 2016. Enhanced UI Automator Viewer with improved Android accessibility evaluation features. In 2016 Inter- national Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT). IEEE, 977–983

    2016

  58. [67]

    Stephen Robertson. 2004. Understanding inverse document frequency: on theo- retical arguments for IDF. Journal of documentation 60, 5 (2004), 503–520

    2004

  59. [68]

    Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O Wobbrock. 2018. Examining image-based button labeling for accessibility in Android apps through large-scale analysis. In Proc. 20th International ACM SIGACCESS Conference on Computers and Accessibility. 119–130. Confe...

    2018

  60. [69]

    Patrick Royston. 1992. Approximating the Shapiro-Wilk W-test for non-normality. Statistics and computing 2 (1992), 117–119

    1992

  61. [70]

    Navid Salehnamadi, Ziyao He, and Sam Malek. 2023. Assistive-technology aided manual accessibility testing in mobile apps, powered by record-and-replay. In Proc. CHI Conference on Human Factors in Computing Systems . 1–20

    2023

  62. [71]

    Navid Salehnamadi, Forough Mehralian, and Sam Malek. 2022. Groundhog: An automated accessibility crawler for mobile apps. In Proc. 37th IEEE/ACM International Conference on Automated Software Engineering . 1–12

    2022

  63. [72]

    Amanda Swearngin et al. 2024. Towards automated accessibility report generation for mobile apps. ACM Transactions on Computer-Human Interaction 31, 4 (2024), 1–44

    2024

  64. [73]

    Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2024. Axnav: Replaying accessibility tests from natural language. In Proc. CHI Conference on Human Factors in Computing Systems . 1–16

    2024

  65. [74]

    Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning . PMLR, 6105–6114

    2019

  66. [75]

    The Noun Project. 2024. Icons and Photos For Everything. https://thenounproject. com/

    2024

  67. [76]

    Access Board

    U.S. Access Board. 2018. About the ICT Accessibility 508 Standards and 255 Guidelines. https://www.access-board.gov/ict/

    2018

  68. [77]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proc. IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) . 4566–4575

    2015

  69. [78]

    Christopher Vendome, Diana Solano, Santiago Liñán, and Mario Linares-Vásquez

  70. [79]

    Bryan Wang, Gang Li, and Yang Li. 2023. Enabling Conversational Interaction with Mobile UI using Large Language Models. In Proc. Conference on Human Factors in Computing Systems (CHI) . ACM, 432:1–432:17

    2023

  71. [80]

    Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. 2021. Real-esrgan: Train- ing real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision . 1905–1914

    2021

  72. [81]

    WebAIM. 2024. Screen Reader User Survey. https://webaim.org/projects/ screenreadersurvey10/

    2024

  73. [82]

    Hao Wen et al. 2024. AutoDroid: LLM-powered Task Automation in Android. In Proc. 30th Annual International Conference on Mobile Computing and Networking (MobiCom). ACM, 543–557

    2024

  74. [83]

    Frank Wilcoxon. 1992. Breakthroughs in statistics. Individual comparisons by ranking methods (1992), 196–202

    1992

  75. [84]

    Jason Wu et al. 2024. UICLIP: A data-driven model for assessing user interface design. In Proc. 37th Annual ACM Symposium on User Interface Software and Technology (UIST). 1–16

    2024

  76. [85]

    Xiaoxue Zang, Ying Xu, and Jindong Chen. 2021. Multimodal icon annotation for mobile applications. In Proc. 23rd International Conference on Mobile Human- Computer Interaction (MobileHCI). ACM, 1–11. https://doi.org/10.1145/3447526. 3472064

    2021 doi

  77. [86]

    Yuxin Zhang, Sen Chen, Lingling Fan, Chunyang Chen, and Xiaohong Li. 2023. Automated and Context-Aware Repair of Color-Related Accessibility Issues for Android Apps. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations ...

    2023

  78. [87]

    Yaroslav Zharov, Yury Khudyakov, Evgeniia Fedotova, Evgeny Grigorenko, and Egor Bogomolov. 2024. Tool-Augmented LLMs as a Universal Interface for IDEs. In Proc. 1st ACM/IEEE Workshop on Integrated Development Environments . 40–42

    2024

  79. [2019]

    In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)

    Can everyone use my app? an empirical study on accessibility in android apps. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 41–52

    2019

  80. [2021]

    IEEE Transactions on Software Engineering 48, 10 (2021), 3954–3968

    Accessible or not? an empirical investigation of android app accessibility. IEEE Transactions on Software Engineering 48, 10 (2021), 3954–3968

    2021

  81. [2022]

    Empirical Software Engineering 27, 6 (2022), 145

    The making of accessible android applications: an empirical study on the state of the practice. Empirical Software Engineering 27, 6 (2022), 145

    2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.