Pith. sign in

REVIEW 1 major objections 2 minor 35 references

Usability Analysis of Configurator User Interfaces with Multimodal Large Language Models

T0 review · 1 major / 2 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multimodal large language models can reliably detect configurator-specific usability issues and suggest improvements when using 18 domain criteria.

desk verdict They run MLLMs on 16 configurator screenshots against 18 literature-derived criteria and conclude from their own review that the outputs are reliable. read the letter →

arxiv 2605.29456 v2 pith:2YMMCX5U submitted 2026-05-28 cs.SE cs.HC

Sebastian Lubos , Alexander Felfernig , Damian Garber , Adnan Kraljić , Tarik Kraljić , Viet-Man Le , Thi Ngoc Trang Tran , Gerhard Leitner
show 6 more authors
Julian Schwazer Doris Suppan Reinhard Willfort Ivan Dukic Jeremias Fuchs Manuel Henrich
This is my paper · ORCID
classification cs.SEcs.HC
keywords configuratorusabilitymultimodallargelanguagemodelsuserinterfaceanalysiscriteriasoftwareconfigurationsemi-automatedevaluationdomain-specificheuristics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper examines whether multimodal large language models can perform usability analysis on configurator user interfaces. It synthesizes 18 specific criteria from existing literature and tests the approach on 16 real-world configurators. For each criterion, the models assign severity levels to issues and propose fixes. The results indicate that these models produce assessments that align with human judgment in identifying problems and offering relevant recommendations. This method could make evaluating and improving configurator usability more efficient by handling much of the initial analysis automatically.

What carries the argument

The set of 18 configurator-specific usability criteria, each evaluated separately by the MLLM to produce severity ratings and suggestions.

What would settle it

Independent expert evaluations of the same 16 configurators using the 18 criteria, measuring the level of agreement with MLLM severity ratings and recommendation quality.

Watch

Extended reading notes

Core claim

By applying 18 configurator-specific usability criteria to screenshots or descriptions of 16 real-world configurators, multimodal large language models generate individual severity ratings and actionable improvement recommendations for each criterion, and a subsequent review finds these outputs to be reliable and domain-aware.

Load-bearing premise

A qualitative human review of the MLLM outputs sufficiently establishes reliability, and the 18 criteria capture configurator usability without major omissions or overlaps.

Editorial extensions

If this is right

  • Analysis effort for configurator usability decreases because MLLMs handle initial assessments.
  • Improvement suggestions are tailored to the configurator domain.
  • The approach works across multiple real-world examples.
  • Human oversight is still required for final validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Integration with existing UI design tools could automate parts of the review process.
  • Expanding the criteria list might cover additional aspects of configurator interaction.
  • Similar techniques could apply to usability analysis in other specialized software domains.
  • Larger-scale studies with more configurators would test consistency across different MLLM versions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper proposes using multimodal large language models (MLLMs) for semi-automated usability analysis of configurator user interfaces. It synthesizes 18 configurator-specific criteria from the literature, applies them via MLLMs to evaluate 16 real-world configurators (producing per-criterion severity ratings and improvement suggestions), and asserts that a qualitative review of the outputs confirms the MLLMs can reliably detect issues and offer domain-aware recommendations (while still requiring human validation).

Significance. If the reliability claim were supported by objective evidence, the work would represent a useful empirical demonstration of applying existing MLLMs to a specialized software-engineering domain where general heuristics are insufficient. The synthesis of 18 domain criteria and the concrete application to 16 configurators could lower the effort for systematic UI analysis, provided the criteria are shown to be non-redundant and the MLLM outputs are validated against expert baselines.

major comments (1)
  1. [Abstract, §4, and §5] Abstract, §4, and §5: The assertion that MLLMs 'can reliably identify configurator-specific usability issues' rests entirely on an unspecified qualitative review of the generated severity ratings and suggestions. No quantitative validation is reported (no inter-rater agreement metrics such as Cohen/Fleiss kappa, no precision/recall against ground-truth issues identified by experts, no ablation on criterion overlap/omission, and no baseline comparison). Because this review is the sole support for the central claim of reliability, the claim is unsupported by objective evidence.
minor comments (2)
  1. [§5] The paper should clarify the exact procedure used for the 'review of the results' (who performed it, how many configurators were inspected in detail, and what rubric was applied) to allow readers to assess its scope.
  2. [§3] Details on the literature synthesis process for the 18 criteria (search strategy, inclusion criteria, and any redundancy or coverage checks) would strengthen the methodological transparency.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. We agree that the reliability claim in the abstract, §4, and §5 rests on qualitative review alone and lacks objective quantitative support, which is a limitation of the current exploratory study. We will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract, §4, and §5] Abstract, §4, and §5: The assertion that MLLMs 'can reliably identify configurator-specific usability issues' rests entirely on an unspecified qualitative review of the generated severity ratings and suggestions. No quantitative validation is reported (no inter-rater agreement metrics such as Cohen/Fleiss kappa, no precision/recall against ground-truth issues identified by experts, no ablation on criterion overlap/omission, and no baseline comparison). Because this review is the sole support for the central claim of reliability, the claim is unsupported by objective evidence.

    Authors: We acknowledge that the central claim relies solely on an author-conducted qualitative review of the MLLM outputs without quantitative metrics, inter-rater agreement, expert baselines, or ablation studies. This is a genuine limitation of the work, which is positioned as an initial demonstration rather than a validated method. In the revision we will: (1) tone down the language in the abstract, §4, and §5 to state that the outputs 'appeared consistent with domain expectations upon qualitative review' instead of claiming 'reliability'; (2) add an explicit description of the review process (authors with configurator expertise examined a sample of outputs for plausibility and actionability); and (3) expand the limitations and future-work sections to highlight the absence of quantitative validation and the need for expert ground-truth comparisons. We cannot retroactively add the requested quantitative analyses to this study but will treat the point as a clear direction for follow-on research. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; empirical application with no self-referential reductions

full rationale

The paper synthesizes 18 criteria from external literature, applies existing MLLMs to 16 configurators, and performs a qualitative review of outputs. No equations, fitted parameters, predictions, or self-citation chains appear. The central claim rests on author inspection of MLLM results rather than any derivation that reduces to its own inputs by construction. This is a standard empirical domain application with no load-bearing self-referential steps.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No mathematical derivations, fitted parameters, or new postulated entities are introduced; the work rests on the empirical claim that MLLM outputs align with human judgment after a review.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Usability Analysis of Configurator User Interfaces with Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/2YMMCX5U

@misc{pith2026260529456,
  author       = {Pith},
  title        = {Pith review of: Usability Analysis of Configurator User Interfaces with Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YMMCX5U}},
  note         = {Machine review of arXiv:2605.29456}
}
read the original abstract

Configuration is a key technology for tailoring complex software systems, services, and products. A successful application of configurators not only depends on technical correctness, performance, and domain modeling but also on their usability. While general usability heuristics are widely used, configurator-specific criteria and tool support for systematic user interface (UI) analysis are limited. This paper explores the use of multimodal large language models (MLLMs) for scalable and semi-automated usability analysis of configurator UIs. We synthesize 18 configurator-specific usability criteria from the literature and apply these criteria in an MLLM-based analysis of 16 real-world configurators. Each criterion is assessed individually to generate severity ratings for usability issues and actionable improvement suggestions. A review of the results confirms that MLLMs can reliably identify configurator-specific usability issues and provide domain-aware improvement recommendations. Although human validation remains necessary, this approach has the potential to significantly reduce the required effort to analyze configurator usability.

Figures

Figures reproduced from arXiv: 2605.29456 by the authors.

Figure 1
Figure 1. System prompt for the usability analysis, providing general instructions. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Prompt template for the usability analysis, where variable placeholders [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Identified issues per configurator across all usability criteria. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Identified issues per usability criterion across all configurators. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 22 canonical work pages

  1. [1]

    In: Proceedings of the 25th International Conference on Advanced Information Systems En- gineering

    Abbasi, E.K., Hubaux, A., Acher, M., Boucher, Q., Heymans, P.: The anatomy of a sales configurator: an empirical study of 111 cases. In: Proceedings of the 25th International Conference on Advanced Information Systems En- gineering. p. 162–177. CAiSE’13, Springer-Verlag, Berlin, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38709-8_11

    work page doi:10.1007/978-3-642-38709-8_11 2013

  2. [2]

    In: Meiselwitz, G

    Castro, J.W., Garnica, I., Rojas, L.A.: Automated tools for usability evaluation: A systematic mapping study. In: Meiselwitz, G. (ed.) Social Computing and Social Media: Design, User Experience and Impact. pp. 28–46. Springer International Publishing, Cham (2022)

    2022

  3. [3]

    In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

    Duan, P., Warner, J., Li, Y., Hartmann, B.: Generating automatic feedback on ui mockups with large language models. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. CHI ’24, Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3613904.3642782

    work page doi:10.1145/3613904.3642782 2024

  4. [4]

    Morgan Kaufmann (2014)

    Felfernig, A., Hotz, L., Bagley, C., Tiihonen, J.: Knowledge-based Configuration – From Research to Business Cases. Morgan Kaufmann (2014)

    2014

  5. [5]

    IEEE Transactions on Engineering Man- agement54(1), 41–56 (2007)

    Felfernig, A.: Standardized configuration knowledge representations as technolog- ical foundation for mass customization. IEEE Transactions on Engineering Man- agement54(1), 41–56 (2007). https://doi.org/10.1109/TEM.2006.889066 Usability Analysis of Configurator UIs with Multimodal LLMs 19

    work page doi:10.1109/tem.2006.889066 2007

  6. [6]

    Felfernig, A., Falkner, A., Benavides, D.: Analysis of Feature Models, pp. 45–72. Springer International Publishing, Cham (2024). https://doi.org/10.1007/978-3- 031-61874-1_3

    work page doi:10.1007/978-3- 2024

  7. [7]

    Psychological bulletin76(5), 378 (1971)

    Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological bulletin76(5), 378 (1971)

    1971

  8. [8]

    Gemini Team Google: Gemini: A family of highly capable multimodal models (2024),https://arxiv.org/abs/2312.11805

    work page Pith review arXiv 2024

Show all 35 references
  1. [9]

    In: Human-Computer Interaction – INTERACT 2025:20thIFIPTC13InternationalConference,BeloHorizonte,Brazil,September 8–12, 2025, Proceedings, Part III

    Guerino, G., Rodrigues, L., Capeleti, B., Mello, R.F., Freire, A., Zaina, L.: Can gpt-4o evaluate usability like human experts? a comparative study on issue iden- tification in heuristic evaluation. In: Human-Computer Interaction – INTERACT 2025:20thIFIPTC13InternationalConfer...

    2025 doi

  2. [10]

    British Journal of Mathematical and Statistical Psychology61(1), 29–48 (2008)

    Gwet, K.L.: Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology61(1), 29–48 (2008). https://doi.org/https://doi.org/10.1348/000711006X126600

    2008 doi

  3. [11]

    Hewett, T.T., Baecker, R., Card, S., Carey, T., Gasen, J., Mantei, M., Perlman, G., Strong, G., Verplank, W.: Human-Computer Interaction, pp. 5–29. ACM, New York, NY, USA (1992)

    1992

  4. [12]

    ISO/IEC/IEEE 9241-11:2018(E) (2018)

    International Organization for Standardization: ISO/IEC/IEEE International Standard - Ergonomics of human-system interaction – Part 11: Usability: Defi- nitions and concepts. ISO/IEC/IEEE 9241-11:2018(E) (2018)

    2018

  5. [13]

    ACM Trans

    Jiang, J., Wang, F., Shen, J., Kim, S., Kim, S.: A survey on large language mod- els for code generation. ACM Trans. Softw. Eng. Methodol.35(2) (Jan 2026). https://doi.org/10.1145/3747588,https://doi.org/10.1145/3747588

    2026 doi

  6. [14]

    Symmetry14(2) (2022)

    Konstantinidis, M., Le, L.W., Gao, X.: An empirical comparative assessment of inter-rater agreement of binary outcomes and multiple raters. Symmetry14(2) (2022). https://doi.org/10.3390/sym14020262

    2022 doi

  7. [15]

    Kuric, E., Demcak, P., Krajcovic, M., Lang, J.: Systematic literature review of automation and artificial intelligence in usability issue detection (2025),https: //arxiv.org/abs/2504.01415

    2025

  8. [16]

    Leclercq, T., Abbasi, E.K., Dumas, B., Remiche, M.A., Heymans, P.: Essential expectations of users of web configurators: An empirical survey. Proc. ACM Hum.- Comput. Interact.6(EICS) (Jun 2022). https://doi.org/10.1145/3534519

    2022 doi

  9. [17]

    In: Joint Proceedings of the ACM IUI 2018 Workshops (2018), https://ceur-ws.org/Vol-2068/wii1.pdf

    Leclercq, T., Cordy, M., Dumas, B., Heymans, P.: On studying bad practices in configuration uis. In: Joint Proceedings of the ACM IUI 2018 Workshops (2018), https://ceur-ws.org/Vol-2068/wii1.pdf

    2018

  10. [18]

    In: Felfernig, A., Hotz, L., Bagley, C., Ti- ihonen, J

    Leitner, G., Felfernig, A., Blazek, P., Reinfrank, F., Ninaus, G.: Chapter 8 - user interfaces for configuration environments. In: Felfernig, A., Hotz, L., Bagley, C., Ti- ihonen, J. (eds.) Knowledge-Based Configuration, pp. 89–106. Morgan Kaufmann, Boston (2014). https://doi....

    2014 doi

  11. [19]

    Lubos, S., Felfernig, A., Garber, D., Le, V.M., Tran, T.N.T.: Towards llm-based usabilityanalysisforrecommenderuserinterfaces.In:Proceedingsofthe12thJoint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS 2025) (2025),https://ceur-ws.org/Vol-4027/...

    2025

  12. [20]

    Lubos, S., Felfernig, A., Garber, D., Leitner, G., Schwazer, J., Henrich, M.: Inves- tigating multimodal large language models to support usability evaluation (2026), https://arxiv.org/abs/2508.16165

    2026 arXiv

  13. [21]

    configurator-database.com/, accessed: 04 December 2025 20 S

    cyLEDGE Media: Configurator database (2022),https://www. configurator-database.com/, accessed: 04 December 2025 20 S. Lubos et al

    2022

  14. [22]

    In: Soares, M.M., Rosenzweig, E., Marcus, A

    Namoun, A., Alrehaili, A., Tufail, A.: A review of automated website usability evaluation tools: Research issues and challenges. In: Soares, M.M., Rosenzweig, E., Marcus, A. (eds.) Design, User Experience, and Usability: UX Research and Design. pp. 292–311. Springer Internatio...

    2021

  15. [23]

    In: Proceed- ings of the SIGCHI Conference on Human Factors in Computing Systems

    Nielsen, J.: Enhancing the explanatory power of usability heuristics. In: Proceed- ings of the SIGCHI Conference on Human Factors in Computing Systems. p. 152–158. CHI ’94, Association for Computing Machinery, New York, NY, USA (1994). https://doi.org/10.1145/191666.191729

    1994 doi

  16. [24]

    Perin, E., Trentin, A., Forza, C.: The effect of sales configurator capabilities on the valueperceivedbythecustomerthroughthecustomizationprocess.In:Proceedings of the 15th International Configuration Workshop (2013),https://ceur-ws.org/ Vol-1128/paper10.pdf

    2013

  17. [25]

    In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE).pp.675–675.IEEEComputerSociety,LosAlamitos,CA,USA(May2025)

    Pourasad, A.E., Maalej, W.: Does GenAI Make Usability Testing Obsolete? . In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE).pp.675–675.IEEEComputerSociety,LosAlamitos,CA,USA(May2025). https://doi.org/10.1109/ICSE55347.2025.00138

    2025 doi

  18. [26]

    In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering

    Rabiser, R., Grünbacher, P., Lehofer, M.: A qualitative study on user guid- ance capabilities in product configuration tools. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. p. 110–119. ASE ’12, Association for Computing Machine...

    2012 doi

  19. [27]

    In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

    Reynolds, L., McDonell, K.: Prompt programming for large language models: Be- yond the few-shot paradigm. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. CHI EA ’21, Association for Computing Machinery, New York, NY, USA (2021). https:/...

    2021 doi

  20. [28]

    In: Proceedings of the International Conference on Economic, Technical and Organisational aspects of Product Configuration Systems (PETO 2004), Lyngby, Denmark

    Rogoll, T., Piller, F.: Product configuration from the customer’s perspective: A comparison of configuration systems in the apparel industry. In: Proceedings of the International Conference on Economic, Technical and Organisational aspects of Product Configuration Systems (PET...

    2004

  21. [29]

    In: Proceedings of the 34th International Conference on Software Engineering

    Siegmund, N., Kolesnikov, S.S., Kästner, C., Apel, S., Batory, D., Rosenmüller, M., Saake, G.:Predicting performance via automated feature-interaction detection. In: Proceedings of the 34th International Conference on Software Engineering. p. 167–177. ICSE ’12, IEEE Press (2012)

    2012

  22. [30]

    ACM Comput

    Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv.47(1) (Jun 2014). https://doi.org/10.1145/2580950

    2014 doi

  23. [31]

    Computers in Industry64(4), 436–447 (2013)

    Trentin, A., Perin, E., Forza, C.: Sales configurator capabili- ties to avoid the product variety paradox: Construct develop- ment and validation. Computers in Industry64(4), 436–447 (2013). https://doi.org/https://doi.org/10.1016/j.compind.2013.02.006

    2013 doi

  24. [32]

    Family medicine37, 360–3 (06 2005)

    Viera, A., Garrett, J.: Understanding interobserver agreement: The kappa statistic. Family medicine37, 360–3 (06 2005)

    2005

  25. [33]

    National Science Review11(12) (Nov 2024)

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on mul- timodal large language models. National Science Review11(12) (Nov 2024). https://doi.org/10.1093/nsr/nwae403

    2024 doi

  26. [34]

    In: 2024 IEEE Conference on Software Testing, Verification and Validation (ICST)

    Yoon, J., Feldt, R., Yoo, S.: Intent-driven mobile gui testing with au- tonomous large language model agents. In: 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). pp. 129–139 (2024). https://doi.org/10.1109/ICST60714.2024.00020 Usability Analysis o...

    2024 doi

  27. [35]

    org/abs/2507.02306

    Zhong, R., McDonald, D.W., Hsieh, G.: Synthetic heuristic evaluation: A compar- ison between ai- and human-powered usability evaluation (2025),https://arxiv. org/abs/2507.02306

    2025

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.