Pith. sign in

REVIEW 4 major objections 6 minor 30 references

Multi-Tool Analysis of User Interface & Accessibility in Deployed Web-Based Chatbots

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multi-tool audit of 106 live web chatbots finds that more than 80% have at least one critical accessibility failure.

desk verdict Useful new corpus of chatbot accessibility, but the headline 80% figure is unsupported and 'critical' is never defined; fix that and the paper earns its place. read the letter →

arxiv 2506.04659 v1 pith:NGUH4QOW submitted 2025-06-05 cs.HC

Mukesh Rajmohan , Smit Desai , Sanchari Das This is my paper
classification cs.HC
keywords ConversationalagentsChatbotaccessibilityWCAGARIArolesAutomatedtestingUserinterfacedesignMulti-toolevaluationAssistivetechnology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that deployed web-based chatbots, despite advances in conversational ability, pervasively fail basic accessibility requirements. Auditing 106 live chatbots across healthcare, education, and customer service with four industry tools, the authors find over 80% exhibit at least one critical issue, and 45% have missing semantic structure or ARIA-role misuse. They also show that accessibility scores agree strongly across automated tools while performance scores do not, which they offer as evidence that multi-tool evaluation is necessary. The point matters because accessibility failures exclude screen-reader, keyboard-only, and other assistive-technology users from services that increasingly route through chatbots.

What carries the argument

The machinery is a standardized multi-tool auditing pipeline applied to a curated corpus of 106 operational web-based chatbots (standalone full-page apps and embedded floating widgets). Four tools carry the measurement: Google's Lighthouse and PageSpeed Insights for automated scoring, SiteImprove for rule-based WCAG violation detection, and Microsoft Accessibility Insights for guided manual audits of focus order, ARIA semantics, and modal behavior. The protocol records performance, best practices, SEO, and accessibility scores plus WCAG violation counts, then compares tools via correlation and relates violation counts to metric scores. The multi-tool design is the load-bearing choice: the paper's argument that accessibility failures are widespread depends on converging evidence from tools with different detection mechanisms.

What would settle it

Run the same four-tool protocol on a new sample of 106 live web chatbots drawn from a defined sampling frame—for instance, a random crawl of sites embedding chat widgets—and count how many show at least one critical accessibility issue. If that rate falls well below 80%, the paper's headline claim does not hold for the broader population; alternatively, a user study where screen-reader users attempt scripted tasks on the flagged chatbots would directly test whether the automated 'critical' flags correspond to real interaction barriers.

Watch

Extended reading notes

Core claim

The central claim is that the front-end layer of real-world web chatbots is broadly noncompliant with WCAG. Of the 106 chatbots audited, over 80% had at least one critical accessibility issue—missing ARIA roles, unlabeled buttons, inaccessible modals, broken keyboard navigation—and about 45% of all violations stem from missing semantic structures or incorrect ARIA roles. The paper further reports that automated accessibility scores are highly correlated across tools (Lighthouse vs PageSpeed Insights $r = 0.861$) while performance scores are not ($r = 0.436$), and that chatbots with more WCAG violations tend to score lower on performance, best practices, and SEO. The authors conclude that accessibility neglect is systemic rather than an edge case, and that a single-tool audit is insufficient to capture it.

Load-bearing premise

The load-bearing premise is that the 106 chatbots, gathered from industry directories and 'top 10' search queries rather than a randomized sample, are representative of deployed web-based chatbots generally.

Editorial extensions

If this is right

  • If the 80% figure holds, most organizations offering web-based chatbots are currently excluding assistive-technology users from core services.
  • Single-tool accessibility audits are insufficient: because performance scores disagreed across tools while accessibility scores agreed, relying on one tool can miss categories of failure.
  • Accessibility violations are not isolated: chatbots with more WCAG violations also scored lower on performance, best practices, and SEO, making accessibility a proxy for general UI robustness.
  • Embedded widgets and standalone apps can show distinct failure patterns, motivating deployment-context-specific testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the pattern generalizes, accessibility compliance for chatbots will likely become a regulatory liability as digital-service rules are enforced, since the same failures that break WCAG also break legal accessibility duties.
  • Inference: the high correlation between automated accessibility scores suggests that a cheap single automated scan could serve as a triage step, but the paper's own low performance-tool correlation implies that no single tool should be trusted for a final pass—an extension the authors do not state.
  • Inference: a testable next step is pairing the same audit pipeline with actual screen-reader user tests on a subset of the flagged chatbots to calibrate automated flags against real barriers; the paper identifies this need but does not run it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a multi-tool accessibility and UI evaluation of 106 deployed web-based chatbots. The authors combined Google Lighthouse, PageSpeed Insights, SiteImprove, and Microsoft Accessibility Insights to score performance, best practices, SEO, and accessibility, and to count WCAG violations. They report that over 80% of the chatbots exhibit at least one critical accessibility issue, that 45% suffer from missing semantic structures or ARIA role misuse, and that accessibility scores correlate strongly across tools (r = 0.861) while performance scores do not (r = 0.436). They argue these findings motivate a multi-tool evaluation approach and offer design recommendations for accessible chatbot interfaces.

Significance. The paper addresses a timely and under-examined question: whether deployed web-based chatbots meet basic WCAG accessibility requirements. Its design has genuine strengths: a relatively large corpus of 106 operational chatbots spanning standalone applications and embedded widgets; a multi-layered evaluation using three automated tools plus a manual audit method; and a focus on concrete violation types (ARIA roles, labels, focus management, modals, keyboard traps) that matter for assistive technology users. If the reported findings are substantiated, the paper would be a valuable empirical contribution for CUI and accessibility researchers, and its recommendations in Section 5 are actionable. However, the central quantitative claims are currently not verifiable from the manuscript, and the tool-agreement analysis is weakened by the shared engine between Lighthouse and PageSpeed Insights. The paper is not yet in a publishable state without addressing these issues.

major comments (4)
  1. [Section 3.3, Section 4.4, Section 7] The headline claim that 'over 80% of the chatbots we analyzed exhibit critical accessibility issues' is not supported by any result presented in Section 4. Section 3.3 describes the evaluation procedure but does not define what constitutes a 'critical' issue, how severity was rated, or how outputs from the four tools were aggregated into a per-chatbot binary outcome. Section 4.4 reports only that 'almost 45% of the issues stem from missing semantic structures or incorrect ARIA role applications,' which is a proportion of issues, not of chatbots, and Figure 6 shows severity data for only ten chatbots. The Conclusion also asserts that chatbots with more than ten WCAG violations 'score 24% lower on performance metrics,' but no supporting analysis appears in the results. The authors need to provide an explicit severity framework, per-chatbot severity distributions, and the calculations behind the 80% and 24% figures.
  2. [Abstract and Section 4.4] The abstract states that '45% suffer from missing semantic structures or ARIA role misuse,' but Section 4.4 states that 'almost 45% of the issues stem from missing semantic structures or incorrect ARIA role applications.' The first is a percentage of chatbots, the second a percentage of issues; these are not equivalent. The abstract's phrasing is unsupported by the reported result and should be corrected to match the actual unit of analysis.
  3. [Section 4.1] The paper uses the high correlation between Google Lighthouse and PageSpeed Insights (accessibility r = 0.861) as evidence for the value of a multi-tool approach. However, PageSpeed Insights is built on the Lighthouse engine, so the two tools are not independent; a high correlation is expected and partly measures self-consistency. The validation claim should be reframed, for example by comparing Lighthouse or PageSpeed Insights against SiteImprove and Microsoft Accessibility Insights, or by explicitly acknowledging this shared dependency and its implications.
  4. [Section 3.1 and Section 6] The sample of 106 chatbots was assembled from directories and 'top 10' search queries without a defined sampling frame or randomization. The paper does not address this in Section 6, which lists only the desktop-browser limitation and the absence of user testing. Without evidence that the sample is representative of deployed web-based chatbots, the population-level generalization of the 80% figure is not supported. The authors should either qualify the claim to the convenience sample or provide a sampling rationale and, ideally, a comparison of sample characteristics to a broader inventory of chatbots.
minor comments (6)
  1. [Abstract] 'We offer a replicable evaluation insights' should be 'We offer replicable evaluation insights.'
  2. [Section 3.1] The footnote marker in '322 unique chatbots 1 candidates' is misplaced and the footnote is incomplete.
  3. [Section 4.2] The sentence 'Appendix Figure 7 shows the average scores ( 80−𝐺𝐿, 82−𝑃𝑆))' contains an extra closing parenthesis and inconsistent spacing.
  4. [Section 4.4] A comma or em dash is needed after 'incorrect ARIA role applications' before 'core elements for screen reader compatibility.'
  5. [Section 5.2] There is a stray space before the period in 'implementation alone .'
  6. [Section 4.1] The correlation coefficients are reported without confidence intervals or significance tests, which would help readers judge the strength of the agreement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all central claims are external measurements with no fitted-parameter or self-citation chain.

full rationale

The paper's central claims—that over 80% of the 106 sampled chatbots exhibit at least one critical accessibility issue, and that accessibility scores correlate across tools—are empirical measurements of external web pages, not derivations from the paper's own inputs. There is no parameter fitted to a subset of data and then renamed as a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. The self-citations to Kishnani et al. [15] and Tazi et al. [25] appear only to note that the selected tools have been used previously in mobile-app accessibility evaluations; they do not bear the weight of any conclusion about chatbot accessibility. The correlation between Lighthouse and PageSpeed Insights (r=0.861) is reported as an observation; even if PageSpeed Insights internally uses Lighthouse, that would weaken the claim that tool agreement validates independence, but it is not a circular derivation—nothing in the paper defines the correlation in terms of itself. Concerns about sample representativeness, the undefined 'critical' threshold, and the mismatch between '45% of issues' and '45% suffer' are validity and reproducibility issues, not circularity. The derivation chain is therefore self-contained with respect to the paper's stated inputs and methods.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central claims rest on empirical choices: a convenience sample, unverifiable tool independence, and an arbitrary violation threshold, all of which limit the strength of the quantitative findings.

free parameters (1)
  • WCAG violation count threshold = 10 violations
    Used in the conclusion to compare performance: chatbots with more than ten WCAG violations score 24% lower on performance metrics. The cutoff is arbitrary and no sensitivity analysis is provided.
assumptions (3)
  • domain assumption Lighthouse and PageSpeed Insights provide independent measurements.
    Section 3.2 presents both as separate tools and Section 4.1 uses their correlation to justify a multi-tool approach; however, PageSpeed Insights is built on the same Lighthouse engine, so the two are not independent.
  • domain assumption The 106-chatbot sample is representative of deployed web-based chatbots.
    Section 3.1 uses convenience sampling from directories and 'top 10' searches; generalizing the 80% claim to all chatbots assumes representativeness without a sampling frame.
  • domain assumption Automated accessibility scores capture user-facing accessibility barriers.
    The paper relies on automated scanners for the main claims; authors state in limitations that these cannot replicate the lived experiences of users with disabilities, yet the headline percentages are based on these scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Tool Analysis of User Interface & Accessibility in Deployed Web-Based Chatbots." pith.science (2026). https://pith.science/paper/NGUH4QOW

@misc{pith2026250604659,
  author       = {Pith},
  title        = {Pith review of: Multi-Tool Analysis of User Interface & Accessibility in Deployed Web-Based Chatbots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGUH4QOW}},
  note         = {Machine review of arXiv:2506.04659}
}
read the original abstract

In this work, we present a multi-tool evaluation of 106 deployed web-based chatbots, across domains like healthcare, education and customer service, comprising both standalone applications and embedded widgets using automated tools (Google Lighthouse, PageSpeed Insights, SiteImprove Accessibility Checker) and manual audits (Microsoft Accessibility Insights). Our analysis reveals that over 80% of chatbots exhibit at least one critical accessibility issue, and 45% suffer from missing semantic structures or ARIA role misuse. Furthermore, we found that accessibility scores correlate strongly across tools (e.g., Lighthouse vs PageSpeed Insights, r = 0.861), but performance scores do not (r = 0.436), underscoring the value of a multi-tool approach. We offer a replicable evaluation insights and actionable recommendations to support the development of user-friendly conversational interfaces.

Figures

Figures reproduced from arXiv: 2506.04659 by the authors.

Figure 1
Figure 1. Methodological Overview to Test UI and Accessibility [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Line Graph: Lighthouse vs PageSpeed Insights Met [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Comparison of chatbot performance across evalu [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (4 more)
Figure 2
Figure 2. Figure 2: Scatter Plots: Lighthouse vs PageSpeed Insights [PITH_FULL_IMAGE:figures/full_fig_p004_2.png]
Figure 5
Figure 5. Figure 5: Relationship between WCAG violations and key [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Proportional Severity of Violations by Chatbot (Top 10) [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Bar Graph: Average scores across Google Lighthouse and PageSpeed Insights [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 19 canonical work pages

  1. [1]

    Eleni Adamopoulou and Lefteris Moussiades. 2020. An Overview of Chatbot Technology. InArtificial Intelligence Applications and Innovations, Ilias Maglogian- nis, Lazaros Iliadis, and Elias Pimenidis (Eds.). Springer International Publishing, Cham, 373–383. doi:10.1007/978-3-030-49186-4_31

    work page doi:10.1007/978-3-030-49186-4_31 2020

  2. [3]

    Petter Bae Brandtzaeg and Asbjørn Følstad. 2018. Chatbots: changing user needs and motivations. Interactions 25, 5 (Aug. 2018), 38–43. doi:10.1145/3236669

    work page doi:10.1145/3236669 2018

  3. [4]

    Petter Bae Brandtzaeg and Asbjørn Følstad. 2017. Why People Use Chatbots. In Internet Science: 4th International Conference, INSCI 2017, Thessaloniki, Greece, November 22-24, 2017, Proceedings . Springer-Verlag, Berlin, Heidelberg, 377–392. doi:10.1007/978-3-319-70284-1_30

    work page doi:10.1007/978-3-319-70284-1_30 2017

  4. [5]

    Guendalina Caldarini, Sardar Jaf, and Kenneth McGarry. 2022. A Literature Survey of Recent Advances in Chatbots. Information 13, 11 (Jan. 2022), 41. doi:10. 3390/info13010041

    work page 2022

  5. [6]

    Myers, Benjamin Cowan, Joel Fischer, Stephan Schlögl, Christine Murad, and Stuart Reeves

    Heloisa Candello, Cosmin Munteanu, Leigh Clark, Jaisie Sin, María Inés Torres, Martin Porcheron, Chelsea M. Myers, Benjamin Cowan, Joel Fischer, Stephan Schlögl, Christine Murad, and Stuart Reeves. 2020. CUI@CHI: Mapping Grand Challenges for the Conversational User Interface Community. In Extended Ab- stracts of the 2020 CHI Conference on Human Factors in...

    arXiv 2020

  6. [7]

    Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. SuperAgent: A Customer Service Chatbot for E-commerce Websites. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, System Demonstra- tions, Mohit Bansal and Heng Ji (Eds.). Association ...

    doi:10.18653/v1/p17-4017 2017

  7. [8]

    João Antônio Desidério de Moraes, Geovana Ramos Sousa Silva, and Edna Dias Canedo. 2024. Towards Enabling Inclusive Conversations: Bridging Accessibility Gaps for the Visually Impaired in a Chatbot Web Chat. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems (IHC ’24) . Association for Computing Machinery, New York, NY, ...

    work page arXiv 2024

  8. [9]

    Jide Edu, Cliona Mulligan, Fabio Pierazzi, Jason Polakis, Guillermo Suarez-Tangil, and Jose Such. 2022. Exploring the security and privacy risks of chatbots in mes- saging services. In Proceedings of the 22nd ACM Internet Measurement Conference (Nice, France) (IMC ’22). Association for Computing Machinery, New York, NY, USA, 581–588. doi:10.1145/3517745.3561433

    arXiv 2022

Show all 30 references
  1. [10]

    Google. 2024. Lighthouse. https://developers.google.com/web/tools/lighthouse

    2024

  2. [11]

    Google. 2024. PageSpeed Insights. https://pagespeed.web.dev

    2024

  3. [12]

    Francisco Iniesto, Tim Coughlan, Kate Lister, Peter Devine, Nick Freear, Richard Greenwood, Wayne Holmes, Ian Kenny, Kevin McLeod, and Ruth Tudor. 2023. Creating ‘a Simple Conversation’: Designing a Conversational User Interface to Improve the Experience of Accessing Support f...

    2023 doi

  4. [13]

    Borna Kalhor and Sanchari Das. 2023. Evaluating the Security and Privacy Risk Postures of Virtual Assistants. arXiv preprint arXiv:2312.14633 (2023)

    2023 arXiv

  5. [14]

    Andrew Kirkpatrick, Joshue O’Connor, Alastair Campbell, and Michael Cooper

  6. [15]

    Urvashi Kishnani, Naheem Noah, Sanchari Das, and Rinku Dewri. 2023. Assessing security, privacy, user interaction, and accessibility features in popular e-payment applications. In Proceedings of the 2023 European Symposium on Usable Security . 143–157

    2023

  7. [16]

    Cowan, Tad Hirsch, and Gary Hsieh

    Raina Langevin, Ross J Lordon, Thi Avrahami, Benjamin R. Cowan, Tad Hirsch, and Gary Hsieh. 2021. Heuristic Evaluation of Conversational Agents. In Pro- ceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21) . Association for Computing Machinery, N...

    2021

  8. [17]

    Kate Lister, Tim Coughlan, Francisco Iniesto, Nick Freear, and Peter Devine

  9. [18]

    Mekler and Kasper Hornbæk

    Elisa D. Mekler and Kasper Hornbæk. 2019. A Framework for the Experience of Meaning in Human-Computer Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19) . Association for Computing Machinery, New York, ...

    2019

  10. [19]

    Microsoft. 2024. Accessibility Insights. https://accessibilityinsights.io

    2024

  11. [20]

    Christopher Power, André Freire, Helen Petrie, and David Swallow. 2012. Guide- lines are only half of the story: accessibility problems encountered by blind users on the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12) . Association ...

    2012

  12. [21]

    Suleiman Saka and Sanchari Das. 2024. Evaluating Privacy Measures in Health- care Apps Predominantly Used by Older Adults. In 2024 Conference on Building a Secure & Empowered Cyberspace (BuildSEC) . IEEE, 79–86

    2024

  13. [22]

    SiteImprove. 2024. SiteImprove Accessibility Checker. https://siteimprove.com

    2024

  14. [23]

    Aishwarya Surani and Sanchari Das. 2022. Understanding privacy and security postures of healthcare chatbots. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. Presented at: CHI , Vol. 22. 1–7

    2022

  15. [24]

    We Have No Security Concerns

    Faiza Tazi, Josiah Dykstra, Prashanth Rajivan, and Sanchari Das. 2024. “We Have No Security Concerns”: Understanding the Privacy-Security Nexus in Telehealth for Audiologists and Speech-Language Pathologists. In Proceedings of the 2024 CHI Conference on Human Factors in Comput...

    2024

  16. [25]

    Faiza Tazi, Suleiman Saka, Griffin Opp, Shradha Neupane, Sanchari Das, Lorenzo De Carli, and Indrakshi Ray. 2023. Accessibility evaluation of IoT Android mobile companion APPs. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems . 1–7

    2023

  17. [26]

    Markel Vigo, Justin Brown, and Vivienne Conway. 2013. Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A ’13). Association for...

    2013

  18. [27]

    Lu Wang, Chaomei Chen, and Jina Huh-Yoo. 2023. Investigating the Synonyms of Conversational Agents to Aid Cross-Disciplinary CA Research. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). Association for Com...

    2023

  19. [28]

    Christina Ziying Wei, Young-Ho Kim, and Anastasia Kuzminykh. 2023. The Bot on Speaking Terms: The Effects of Conversation Architecture on Perceptions of Conversational Agents. In Proceedings of the 5th International Conference on Conversational User Interfaces (Eindhoven, Neth...

    2023

  20. [29]

    Qingxiao Zheng, Yiliu Tang, Yiren Liu, Weizi Liu, and Yun Huang. 2022. UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems (New Orleans, LA, USA) (...

    2022

  21. [2018]

    https://www.w3.org/ TR/WCAG21/

    Web Content Accessibility Guidelines (WCAG) 2.1. https://www.w3.org/ TR/WCAG21/. W3C Recommendation

  22. [2020]

    In Proceedings of the 17th International Web for All Conference (Taipei, Taiwan) (W4A ’20)

    Accessible conversational user interfaces: considerations for design. In Proceedings of the 17th International Web for All Conference (Taipei, Taiwan) (W4A ’20). Association for Computing Machinery, New York, NY, USA, Article 5, 11 pages. doi:10.1145/3371300.3383343

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.