Pith. sign in

REVIEW 4 major objections 5 minor 72 references

Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a prompted multimodal LLM can find more usability issues in two mobile apps than five experienced human evaluators.

desk verdict The paper's core claim that GPT-4 beats five expert evaluators is unsupported because the benchmark includes GPT-4's own findings, but the prompt engineering and failure-mode analysis are worth a look. read the letter →

arxiv 2507.02306 v1 pith:OGFAI2PG submitted 2025-07-03 cs.HC cs.AI

Ruican Zhong , David W. McDonald , Gary Hsieh This is my paper
classification cs.HCcs.AI
keywords heuristicevaluationusabilitytestingmultimodalLLMGPT-4synthetichuman-AIcomparisonNielsen'sheuristicsUI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that heuristic evaluation, the practice of checking a user interface against a fixed set of design principles, can be carried out by a multimodal large language model prompted with screenshots. Its central result is that GPT-4, given ordered screenshots and a carefully engineered prompt, identified 73% and 77% of the usability issues on two mobile apps, while the combined findings of five experienced UX evaluators covered 57% and 63%. The method stayed stable across two accounts and a three-month period, and it performed especially well on layout and aesthetic problems, but it missed many violations that only appear when comparing multiple screens and sometimes misread standard UI components. A sympathetic reader would care because usability testing is expensive and slow, and a reliable low-cost first pass could change how widely it is used.

What carries the argument

The load-bearing mechanism is a prompting protocol combined with a comparison benchmark. The prompt gives the model a user scenario and ordered screenshots, asks it to evaluate the first five or the second five of Nielsen's heuristics, and for each violation requires a specific location, a rationale tied to the heuristic, a severity rating from 0 to 4, and a reason for that rating. This decomposition converts a broad 'critique the design' request into the structured violation-reporting format of heuristic evaluation and fits within output token limits. The benchmark is a master set built by pooling issues from five expert evaluators, five trained research assistants, and GPT-4 itself, deduplicating them, and manually coding severity; each evaluator's coverage is then computed as the share of master-set issues they reported.

What would settle it

Construct a master set for two new apps using only think-aloud user testing or evaluations from experts who never see LLM output, and run the same GPT-4 prompt against it; if GPT-4's coverage falls to the five-expert aggregate level or below, the claimed advantage collapses. A simpler version is available in the paper's own data: recompute GPT-4's coverage using only the master-set issues that human experts and research assistants reported, excluding the issues that only GPT-4 contributed; if that coverage is no better than 57-63%, the headline result is an artifact of self-scoring.

Watch

Extended reading notes

Core claim

The paper's central claim is that an off-the-shelf multimodal LLM can perform a genuine heuristic evaluation rather than merely produce vague design praise. The authors developed a prompting procedure, chain-of-thought instructions that ask for the issue, a rationale, and a severity rating, with Nielsen's 10 heuristics split into two passes so output stays within token limits, and applied it to screenshots of a rental app and a language-learning app. Measured against a master set of unique usability issues, GPT-4 covered 73% (97/133) and 77% (87/113) of the issues, versus 57% (76/133) and 63% (71/113) for the aggregated output of five expert evaluators. The paper also reports that synthetic evaluation's coverage stayed roughly constant across evaluation tasks while human coverage declined by about 32%, that GPT-4 beat Gemini-1.5-pro and Claude 3.5 Sonnet under the same prompt, and that the main failure modes were misreading UI components, not understanding app conventions, and missing across-screen violations.

Load-bearing premise

The benchmark is treated as a complete and correct list of every real usability problem, but it was assembled partly from the very model being scored; if the union of expert findings, research-assistant findings, and GPT-4 findings is incomplete, or if some of GPT-4's false positives survived the severity coding, the 73% and 77% coverage numbers are not an independent measurement.

Editorial extensions

If this is right

  • Off-the-shelf GPT-4 can be turned into a heuristic evaluator by prompt design alone, without fine-tuning or retraining.
  • At the observed coverage levels, a synthetic pass could serve as an inexpensive first sweep that finds most usability issues before human experts invest time in detailed review.
  • Coverage of roughly 70-77% can be expected to repeat across reruns and accounts over at least a three-month window, making the method usable in longitudinal design processes.
  • The method is complementary rather than a full replacement: it is strongest on aesthetic and layout violations and weakest on consistency across screens and on recognizing standard app conventions, so human review should be aimed at those categories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because GPT-4 helped build the master set against which it was scored, the reported 73% and 77% should be read as an upper bound; an independently constructed ground truth would likely place the model closer to, though possibly still above, the human aggregate.
  • The split into two five-heuristic passes suggests the model's recall is constrained by output length, not just perception; longer-context models or structured output could reduce duplicate reports and recover some across-screen issues.
  • A practical division of labor follows from the paper's failure analysis: let the LLM sweep for spacing, hierarchy, and visual-consistency problems, while humans focus on app-specific conventions and multi-screen flows.
  • The reliability result is model- and time-specific; the same protocol on a future GPT version or a different off-the-shelf model would need a fresh reliability check before being trusted.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a method for conducting heuristic evaluation of mobile user interfaces by prompting a multimodal large language model (GPT-4) with screenshots and Nielsen's 10 heuristics. Across two apps, the authors report that the synthetic evaluation identified 73% and 77% of the issues in an aggregated master set, compared with 57% and 63% for the union of five expert evaluators. They also test the consistency of outputs over a three-month period and across two accounts, and compare GPT-4 with Gemini 1.5 Pro and Claude 3.5 Sonnet under the same prompting protocol. The paper further provides qualitative analyses of the kinds of issues that synthetic evaluation detects well (e.g., small layout differences) and poorly (e.g., recognizing UI components, understanding app conventions, and aggregating across screens).

Significance. If the reported coverage numbers were valid, the claim that a prompted off-the-shelf multimodal LLM can outperform a panel of five expert human evaluators would be of substantial practical interest to the HCI community, given the cost and time of heuristic evaluation. The paper also makes useful contributions by testing LLM output stability over time and across accounts and by comparing multiple models under a fixed prompting procedure. However, the central comparison is compromised by the construction of the ground-truth master set, which includes GPT-4's own findings without independent validation. As a result, the headline performance numbers are not trustworthy, and the paper's main claim is unsupported by the current study design.

major comments (4)
  1. [4.1.3 and 4.3] The master set used as the benchmark is constructed by unioning the issues reported by the expert evaluators, the local research assistants, and GPT-4, after which severity-0 items are removed. GPT-4's coverage (73% and 77%) is then computed against this same master set, so the model receives direct credit for its own previously unseen outputs. The paper never reports how many of the 133/113 non-zero-severity master issues were found exclusively by GPT-4, nor does it provide any external validation of those issues; the severity coding performed by the first author and two researchers assigns 0-4 ratings but does not independently establish that a GPT-4-only item is a genuine usability problem. This circularity makes the headline claim that synthetic evaluation outperforms the aggregation of five expert evaluators unsupported by the design.
  2. [4.1.3 and 4.3] The same design also biases the human comparison downward: the aggregated 5-expert coverage of 57% and 63% is computed against a master set that includes GPT-4-only and research-assistant-only issues, so the experts are penalized for missing items that no human ever identified and for which no external validation exists. A valid comparison requires a ground-truth set that is constructed independently of the systems being compared, for instance from user-based usability testing or from a separate panel of experts whose findings are not part of the coverage calculation. The evaluation should either re-analyze the data with GPT-4's exclusive findings excluded from the master set, or report coverage separately for master items with and without GPT-4 provenance.
  3. [4.3] The statement that 'synthetic evaluation reported more usability issues than the aggregated 5-expert evaluation for both the rental app (p < 0.001) and the language learning app (p < 0.001) across all severity ratings' is contradicted by Table 3, which reports that for severity 4 the 5-expert evaluation achieved 71% (10/14) coverage versus GPT-4's 64% (9/14). The authors should correct this statement and clarify whether any statistical test was performed at the per-severity level; the current wording appears to describe the overall comparison rather than a per-severity one.
  4. [7] The limitation section acknowledges the small number of apps and possible model drift, but it does not mention the fundamental validity threat posed by the master set construction. Given that Section 4.3.3 documents that a substantial fraction of GPT-4's severity-0 issues stem from UI misrecognition, the manuscript should at minimum quantify how many non-severity-0 GPT-4-only issues survived the authors' severity coding and discuss what independent evidence, if any, supports treating those items as genuine usability problems.
minor comments (5)
  1. [1] The CCS Concepts and ACM Reference Format still contain 'Do Not Use This Code' placeholders and the copyright year '2018'; these template artifacts should be removed or corrected before publication.
  2. [4.3.1] The interpretation of the logistic regression is imprecise: an estimated coefficient of -0.316 on the log-odds scale corresponds to an odds ratio of approximately 0.73, not a '31.6% decrease' in performance; the authors should revise the wording and, ideally, report the odds ratio with its confidence interval.
  3. [4.3 and 5.3.2] There is a numerical inconsistency in the reported language-learning-app coverage for GPT-4: Section 4.3 states 77% (87/113), Section 5.3.1 states 77% (87/113), but Section 5.3.2 states 78% (88/113); the same GPT-4 run should not yield two different counts.
  4. [4.3.1 and Figure 5] The text and figure would benefit from a clearer definition of 'evaluation task' and of the units on the x-axis; it is not evident whether the four tasks per app are pooled across both apps in the regression, or whether some other pooling is used.
  5. [Table 2] The 'Error prevention' row for the rental app has a denominator of 1 issue; the authors may want to note explicitly that this cell provides very limited information rather than implying a meaningful comparison.

Circularity Check

1 steps flagged · score 6.0 of 10

GPT-4 is scored against a master set that includes GPT-4's own findings, so the 73%/77% coverage advantage over human evaluators is partly built into the benchmark.

  1. self definitional [Section 4.1.3 (Master set) and Section 4.3 (Results)]
    "The master set of heuristic issues represents the complete list of heuristic violations of the two apps uncovered in our study. ... To curate the master set, we first combined all heuristic violations found by our expert evaluators and all issues found by GPT-4. ... The aggregated 5-expert evaluation found 57% (76/133) and 63% (71/113) for the rental and language learning apps respectively, compared to the aforementioned 73% and 77% uncovered by the synthetic evaluation."

    The master set used as ground truth is defined as a union that includes GPT-4's own heuristic violations. Every GPT-4 issue that survives duplicate removal and the authors' severity coding is therefore a member of the denominator, and it automatically matches GPT-4 when coverage is computed. Since the reported numbers imply at least 21 (rental) and 16 (language) nonzero master-set items were found by GPT-4 but by no expert, the human aggregate is scored against GPT-4-derived items that may never have been independently verified as real usability problems. The severity coding assigns 0-4 ratings but does not provide an external ground truth for GPT-4-only issues, so the 73%/77% versus 57%/63% advantage is partly self-referential rather than an independent measurement.

full rationale

The central comparison in this paper is partially circular because the benchmark master set is constructed by unioning GPT-4's findings with human and research-assistant findings (Section 4.1.3), and then GPT-4's coverage is computed against that same set (Section 4.3). Any GPT-4 finding that is not a duplicate and is rated severity > 0 is added to the master set, so GPT-4 is guaranteed credit for its own outputs, while human evaluators are penalized for missing those GPT-4-derived items. The authors' severity coding filters some false positives, but it does not independently validate GPT-4-only issues. No other formal circularity was found: self-citations are used only as standard references, and the reliability and cross-platform analyses inherit the same master-set issue rather than adding a new one. Because the headline claim of exceeding five experienced human evaluators is substantially an artifact of benchmark construction, the score is elevated, though the comparison is not entirely forced since human findings and research-assistant findings also contribute to the master set.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a master-set benchmark that contains GPT-4's own outputs. There are no fitted physical parameters, but several hand-chosen thresholds and prompt parameters affect the main numbers, and the validity of the benchmark is assumed rather than demonstrated.

free parameters (3)
  • Minimum issues per heuristic in prompt = 2
    The prompt instructs the LLM to identify at least 2 problems for each block of heuristics (Section 3, Table 1). This hand-chosen value affects the number and coverage of reported issues, and no ablation is reported.
  • Heuristic block split for token limit = 5/5
    The authors split Nielsen's 10 heuristics into two blocks of five to avoid output truncation (Section 3). This choice changes how across-screen reasoning is applied and is not systematically varied.
  • Severity threshold for counting real issues = severity != 0
    The paper excludes severity 0 issues from performance calculations, but severity labels were assigned and re-coded by the first author (Section 4.1.3). Changing this threshold changes all reported coverage rates.
assumptions (5)
  • ad hoc to paper The union of expert, research assistant, and GPT-4 findings is a complete and correct master set of usability issues.
    Section 4.1.3. This is the load-bearing assumption that makes coverage a meaningful performance metric. It is violated because the evaluated model is also a source for the benchmark, and no external ground truth is provided.
  • domain assumption Nielsen's 10 heuristics are appropriate and sufficient for evaluating these two mobile apps.
    Section 2.1 and Section 4.1. The paper relies on Nielsen's heuristics as valid criteria and does not compare against user testing or another usability method.
  • domain assumption UpWork freelancers with self-reported UX experience represent expert heuristic evaluators.
    Section 4.1.2. Participants were recruited on UpWork and screened by self-report; no portfolio verification or standardized expertise test is reported, and only five rated each app.
  • domain assumption Screenshots plus task descriptions provide enough information for heuristic evaluation.
    Section 2.1 and Section 4.1. The paper assumes static screenshots suffice, but later finds that across-screen violations are difficult, which partially undermines this assumption.
  • domain assumption Issues generated at different times and from different accounts can be reliably matched by researcher coding.
    Section 5.1.1. The first author coded and cross-checked matching, but no inter-rater reliability or codebook is provided, so the consistency results rest on subjective matching.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation." pith.science (2026). https://pith.science/paper/OGFAI2PG

@misc{pith2026250702306,
  author       = {Pith},
  title        = {Pith review of: Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OGFAI2PG}},
  note         = {Machine review of arXiv:2507.02306}
}
read the original abstract

Usability evaluation is crucial in human-centered design but can be costly, requiring expert time and user compensation. In this work, we developed a method for synthetic heuristic evaluation using multimodal LLMs' ability to analyze images and provide design feedback. Comparing our synthetic evaluations to those by experienced UX practitioners across two apps, we found our evaluation identified 73% and 77% of usability issues, which exceeded the performance of 5 experienced human evaluators (57% and 63%). Compared to human evaluators, the synthetic evaluation's performance maintained consistent performance across tasks and excelled in detecting layout issues, highlighting potential attentional and perceptual strengths of synthetic evaluation. However, synthetic evaluation struggled with recognizing some UI components and design conventions, as well as identifying across screen violations. Additionally, testing synthetic evaluations over time and accounts revealed stable performance. Overall, our work highlights the performance differences between human and LLM-driven evaluations, informing the design of synthetic heuristic evaluations.

Figures

Figures reproduced from arXiv: 2507.02306 by the authors.

Figure 2
Figure 2. These two figures are example screens in the language [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. This figure shows expert evaluators’ perfor [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 5
Figure 5. This figure shows expert evaluators’ performance and synthetic heuristic evaluation’s performance across user tasks for the [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 55 canonical work pages

  1. [1]

    [n. d.]. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    work page

  2. [3]

    Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training LLMs for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering . 1–5

    work page 2022

  3. [4]

    Sai Pankaj Akula. 2021. A critical evaluation on SRK STORE APP by using the Heuristic Principles of Usability

    work page 2021

  4. [5]

    Hend S Al-Khalifa, Bashayer Al-Twaim, and Bedour AlHarbi. 2016. A heuristic checklist for usability evaluation of Saudi government mobile applications. In Proceedings of the 18th International Conference on Information Integration and Web-based Applications and Services . 375–378

    work page 2016

  5. [6]

    Ghassan Al-Qaimari and Darren McRostie. 1999. KALDI: a computer-aided usability engineering tool for supporting testing and analysis of human-computer interaction. In Computer-Aided Design of User Interfaces II: Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces, 21–23 October, 1999, Louvain-la-Neuve, Belgium . S...

    work page 1999

  6. [7]

    Anthony. 2010. When to Use a Button or Link. https://uxmovement.com/buttons/when-to-use-a-button-or-link/

    work page 2010

  7. [8]

    American Economic Association, Royal Economic Society, and Herbert A Simon. 1966. Theories of decision-making in economics and behavioural science. Springer

    work page 1966

  8. [9]

    Ahmad Azizi, Mahmood Maniati, Hadis Ghanbari-Adivi, Zeinab Aghajari, Sedigheh Hashemi, Bahareh Hajipoor, Asma Rabiee Qolami, Maryam Qolami, and Amirabbas Azizi. 2021. Usability evaluation of hospital information system according to heuristic evaluation. Frontiers in Health Informatics 10, 1 (2021), 69

    work page 2021

Show all 72 references
  1. [10]

    Sandrine Balbo. 1995. Automatic evaluation of user interface usability: Dream or reality. In Proceedings of the Queensland Computer-Human Interaction Symposium, Vol. 7. Bond University Queensland

    1995

  2. [11]

    Lynn K Baumeister, Bonnie E John, and Michael D Byrne. 2000. A comparison of tools for building GOMS models. In Proceedings of the SIGCHI conference on Human factors in computing systems . 502–509

    2000

  3. [12]

    Neil Bowers. 1996. Weblint: quality assurance for the World Wide Web. Computer Networks and ISDN Systems 28, 7-11 (1996), 1283–1290

    1996

  4. [13]

    André Castello Branco, Eveline Sacramento, Eliza Oliveira, Oksana Tymoshchuk, Maria Antunes, Margarida Almeida, Luís Pedro, Fernando Ramos, and Daniel Carvalho. 2022. Usability Evaluation of a Community-led Innovation Mobile App. (2022)

    2022

  5. [14]

    Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg. 2024. What is the Visual Cognition Gap between Humans and Multimodal LLMs? arXiv preprint arXiv:2406.10424 (2024)

    2024 arXiv

  6. [15]

    P Chandler and J Sweller. 1991. Congnitive load theory and the format of instruction. Congnition and Instruc tion. (1991). Manuscript submitted to ACM 20 Ruican Zhong, David W. McDonald, and Gary Hsieh

    1991

  7. [16]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1–45

    2024

  8. [17]

    Ed H Chi, Peter Pirolli, and James Pitkow. 2000. The scent of a site: A system for analyzing and predicting information scent, usage, and usability of a web site. In Proceedings of the SIGCHI conference on Human factors in computing systems . 161–168

    2000

  9. [18]

    WebTrends Coorporation. 2000. http://www.webtrendslive.com/de-fault.htm

    2000

  10. [19]

    John Cugini and Jean Scholtz. 1999. VISVIP: 3D visualization of paths through web sites. In Proceedings. Tenth International Workshop on Database and Expert Systems Applications. DEXA 99 . IEEE, 259–263

    1999

  11. [20]

    Alexiei Dingli and Sarah Cassar. 2014. An intelligent framework for website usability. Advances in Human-Computer Interaction 2014 (2014), 5–5

    2014

  12. [21]

    Peitong Duan, Chin-Yi Cheng, Gang Li, Bjoern Hartmann, and Yang Li. 2024. UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology . 1–17

    2024

  13. [22]

    Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–20

    2024

  14. [23]

    François Fleuret, Ting Li, Charles Dubout, Emma K Wampler, Steven Yantis, and Donald Geman. 2011. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences 108, 43 (2011), 17621–17625

    2011

  15. [24]

    Camilo Fosco, Vincent Casser, Amish Kumar Bedi, Peter O’Donovan, Aaron Hertzmann, and Zoya Bylinskii. 2020. Predicting visual importance across graphic design types. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology . 249–260

    2020

  16. [25]

    Reese Hoi Yin Fung, Dickson KW Chiu, Eddie HT Ko, Kevin KW Ho, and Patrick Lo. 2016. Heuristic usability evaluation of university of Hong Kong libraries’ mobile website. The Journal of Academic Librarianship 42, 5 (2016), 581–594

    2016

  17. [26]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

    2023 arXiv

  18. [27]

    FA Glenn, SM Schwartz, and LV Ross. 1992. Development of a human operator simulator version v (HOS-V): Design and implementation. US Army Research Institute for the Behavioral and Social Sciences, PERI-POX, Alexandria, V A (1992)

    1992

  19. [28]

    Neilsen Normal Group. 2020. Remote Usability-Testing Costs: Moderated vs. Unmoderated. https://www.nngroup.com/articles/remote-usability- testing-costs/

    2020

  20. [29]

    Nielsen Norman Group. 2023. How to Conduct a Heuristic Evaluation. https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/#: ~:text=A%20heuristic%20evaluation%20is%20a,make%20systems%20easy%20to%20use

    2023

  21. [30]

    Monty L Hammontree, Jeffrey J Hendrickson, and Billy W Hensley. 1992. Integrated data capture and analysis tools for research and testing on graphical user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems . 431–432

    1992

  22. [31]

    Brian Helfrich and James A Landay. 1999. QUIP: quantitative user interface profiling. Unpublished manuscript. A vailable at http://home. earthlink. net/ bhelfrich/quip/index. html (1999)

    1999

  23. [32]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6700–6709

    2019

  24. [33]

    Michael Hughes. 1999. Rigor in usability testing. Technical communication 46, 4 (1999), 488

    1999

  25. [34]

    Melody Y Ivory and Marti A Hearst. 2001. The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys (CSUR) 33, 4 (2001), 470–516

    2001

  26. [35]

    Raina Langevin, Ross J Lordon, Thi Avrahami, Benjamin R Cowan, Tad Hirsch, and Gary Hsieh. 2021. Heuristic evaluation of conversational agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–15

    2021

  27. [36]

    Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020. GUIComp: A GUI design assistant with real-time, multi-faceted feedback. In Proceedings of the 2020 CHI conference on human factors in computing systems . 1–13

    2020

  28. [37]

    Weisi Lin, Li Dong, and Ping Xue. 2005. Visual distortion gauge based on discrimination of noticeable contrast changes. IEEE transactions on circuits and systems for video technology 15, 7 (2005), 900–909

    2005

  29. [38]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2023. Chatting with gpt-3 for zero-shot human-like mobile automated gui testing. arXiv preprint arXiv:2305.09434 (2023)

    2023 arXiv

  30. [39]

    George D Magoulas, Sherry Y Chen, and Kyparissia A Papanikolaou. 2003. Integrating layered and heuristic evaluation for adaptive learning environments. In Proceedings of the second workshop on empirical evaluation of adaptive systems, held at the 9th international conference o...

    2003

  31. [40]

    Jennifer Mankoff, Anind K Dey, Gary Hsieh, Julie Kientz, Scott Lederer, and Morgan Ames. 2003. Heuristic evaluation of ambient displays. In Proceedings of the SIGCHI conference on Human factors in computing systems . 169–176

    2003

  32. [41]

    Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. 2023. Sources of Hallucination by Large Language Models on Inference Tasks. arXiv preprint arXiv:2305.14552 (2023)

    2023 arXiv

  33. [42]

    MeasuringU. 2018. How Much Does a Usability Test Cost? https://measuringu.com/usability-cost/

    2018

  34. [43]

    Rolf Molich and Jakob Nielsen. 1990. Improving a human-computer dialogue. Commun. ACM 33, 3 (1990), 338–348

    1990

  35. [44]

    Meredith Ringel Morris. 2023. Scientists’ Perspectives on the Potential for Generative AI in their Fields. arXiv preprint arXiv:2304.01420 (2023)

    2023 arXiv

  36. [45]

    Meredith Ringel Morris. 2024. Prompting Considered Harmful. Commun. ACM 67, 12 (2024), 28–30

    2024

  37. [46]

    Abdallah Namoun, Ahmed Alrehaili, and Ali Tufail. 2021. A Review of Automated Website Usability Evaluation Tools: Research Issues and Challenges. In International Conference on Human-Computer Interaction . Springer, 292–311. Manuscript submitted to ACM Synthetic Heuristic Eval...

    2021

  38. [47]

    Jakob Nielsen. 1992. Finding usability problems through heuristic evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems. 373–380

    1992

  39. [48]

    Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 249–256

    1990

  40. [49]

    Dan R Olsen and Bradley W Halversen. 1988. Interface usage measurements in a user interface management system. In Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software . 102–108

    1988

  41. [50]

    OpenAI. 2023. GPT-4 Model. https://platform.openai.com/docs/guides/gpt

    2023

  42. [51]

    OpenAI. 2023. GPT-4 Technical Report. ArXivabs/2303.08774

    2023 arXiv

  43. [52]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023. LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. arXiv preprint arXiv:2308.02828 (2023)

    2023 arXiv

  44. [53]

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology . 1–22

    2023

  45. [54]

    Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Techno...

    2022

  46. [55]

    Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil˙e Lukoši¯ut˙e, et al. 2023. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11...

    2023 arXiv

  47. [56]

    Matthias Rauterberg. 1995. From Novice to Expert Decision Behaviour: a Qualitative Modelling Approach with Petri Nets. In Advances in Human Factors/Ergonomics. Vol. 20. Elsevier, 449–454

    1995

  48. [57]

    Janice Ginny Redish. 2007. Expanding usability testing to evaluate complex systems. Journal of usability studies 2, 3 (2007), 102–111

    2007

  49. [58]

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries 23, 3 (2022), 289–301

    2022

  50. [59]

    Albrecht Schmidt, Passant Elagroudy, Fiona Draxler, Frauke Kreuter, and Robin Welsch. 2024. Simulating the Human in HCD with ChatGPT: Redesigning Interaction Design with AI. Interactions 31, 1 (2024), 24–31

    2024

  51. [60]

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 2998–3009

    2023

  52. [61]

    Moritz Staudinger, Wojciech Kusa, Florina Piroi, Aldo Lipani, and Allan Hanbury. 2024. A Reproducibility and Generalizability Study of Large Language Models for Query Generation. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development i...

    2024

  53. [62]

    John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257–285

    1988

  54. [63]

    Space-O Technologies. [n. d.]. 20 Most Popular Mobile App Categories You Must Know in 2024. https://www.spaceo.ca/blog/popular-categories-of- apps/

    2024

  55. [64]

    Janet Patton Tracy and Michael J Albers. 2006. Measuring cognitive load to test the usability of web sites. In Annual Conference-society for technical communication, Vol. 53. 256

    2006

  56. [65]

    Dana L Uehling and Karl Wolf. 1995. User action graphing effort (UsAGE). InConference companion on human factors in computing systems . 290–291

    1995

  57. [66]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837

    2022

  58. [67]

    Jason Wu, Yi-Hao Peng, Amanda Li, Amanda Swearngin, Jeffrey P Bigham, and Jeffrey Nichols. 2024. UIClip: A Data-driven Model for Assessing User Interface Design. arXiv preprint arXiv:2404.12500 (2024)

    2024 arXiv

  59. [68]

    Sherry Wu, Hua Shen, Daniel S Weld, Jeffrey Heer, and Marco Tulio Ribeiro. 2023. ScatterShot: Interactive In-context Example Curation for Text Transformation. In Proceedings of the 28th International Conference on Intelligent User Interfaces . 353–367

    2023

  60. [69]

    Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. 2023. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959 (2023)

    2023 arXiv

  61. [70]

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. 2022. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 (2022)

    2022 arXiv

  62. [71]

    Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. 2024. LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Automation Task Evaluation. arXiv preprint arXiv:2404.16054 (2024)

    2024 arXiv

  63. [72]

    Xiaohui Zhang, Weisi Lin, and Ping Xue. 2008. Just-noticeable difference estimation with pixels in images. Journal of Visual Communication and Image Representation 19, 1 (2008), 30–41

    2008

  64. [73]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 Manuscript submitted to ACM

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.