Pith. sign in

REVIEW 3 major objections 5 minor 72 references

Card Sorting Simulator: Augmenting Design of Logical Information Architectures with Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-generated card sorting can reproduce the strongest grouping patterns of real participants, with mean normalized mutual information of 0.68 when the model is asked directly for an aggregate clustering, and performs markedly worse when…

desk verdict A solid, large-scale empirical benchmark of LLM card sorting, but the headline NMI numbers need calibration against chance and human consistency before we know how good 0.68 really is. read the letter →

arxiv 2505.09478 v1 pith:ZNZBN3AQ submitted 2025-05-14 cs.HC

Eduard Kuric , Peter Demcak , Matus Krajcovic This is my paper · ORCID
classification cs.HC
keywords cardsortinglargelanguagemodelssyntheticparticipantsinformationarchitectureuserresearchmentalclusteringgenerativeartificialintelligence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Card sorting asks users to group labeled items so designers can infer the mental model behind an information architecture. This paper argues that a large language model can stand in for that process well enough to give useful early feedback: across 28 real open card sorting studies with 1,399 participants, the best prompt design produced clusterings with a mean normalized mutual information of 0.68 against clusterings derived from human sorts. The strongest result is that the simulation should be prompted to produce one aggregate clustering rather than to imitate individual participants; the aggregate-level prompts (P2 and P3) both reach 0.68, while raw-data simulation (P1) reaches only 0.50. Agreement falls when studies have more cards or harder labels, and neither model choice nor added study context changes the outcome much. The paper positions the simulator as an augmenting tool for quick preliminary feedback, not a replacement for human participants.

What carries the argument

The load-bearing component is the Card Sorting Simulator's prompt template, with four output-format variants; the decisive variant is P3, Clustering Generation, which instructs the LLM to output a single best aggregate categorization as if it were the result of analyzed responses from a sample of participants. The comparison pipeline converts all real and simulated sorts into a common representation: a similarity matrix of pairwise co-grouping frequencies, reduced by multidimensional scaling, then clustered with k-means where the number of clusters $K$ is fixed by the knee/elbow point of the within-cluster sum-of-squares curve. Agreement between synthetic and human clusterings is measured with normalized mutual information as the primary metric, alongside adjusted Rand index, edit distance, and Mantel matrix correlation. The paper also includes qualitative inspection of similarity-matrix heatmaps to explain structural divergences such as phantom groupings and over-general category labels.

What would settle it

Re-run the evaluation with an independent reference: compare LLM outputs against the clusters participants themselves named or intended, or replace k-means with hierarchical best-merge clustering on the same similarity matrices. If the P2/P3 advantage over P1 shrinks or the 0.68 mean NMI drops substantially, the reported agreement depends on the chosen analysis pipeline rather than on genuine alignment of mental models.

Watch

Extended reading notes

Core claim

The central discovery is that LLM-generated card sorting agrees with human card sorting on the most salient groupings, while disagreeing about individual card placement, category abstraction level, and diversity. Using the study's own evaluation pipeline, Clustering Generation (P3) reaches a mean NMI of 0.68 (SD 0.12) and Similarity Matrix Generation (P2) also reaches 0.68 (SD 0.11), whereas Raw Data Simulation (P1) reaches only 0.50 (SD 0.15); ARI, edit distance, and matrix correlation follow the same ordering, with P3 and P2 close and P1 markedly worse. The authors interpret this as support for their hypothesis H1a, that the more abstract prompt format is the more accurate one, and H1b, that agreement is partial and not sufficient for substitution. Additional findings are that agreement is stable across repeated runs, that differences between Claude, Gemini, GPT-4o, and DeepSeek are small and mostly non-significant, and that adding contextual information (demographics, instructions, welcome messages) does not improve agreement. Study complexity modulates accuracy: mean NMI for P3 falls from 0.77 in low-card-count studies to 0.65 in high-card-count studies, and from 0.78 for simple labels to 0.64 for complex labels.

Load-bearing premise

The whole comparison depends on treating the statistically derived grouping of the real participants' sorts as ground truth; if that derived grouping is not what participants themselves meant, then every agreement score with the simulated sorts loses its meaning.

Editorial extensions

If this is right

  • If the central claim is right, UX researchers can obtain meaningful preliminary card-sorting feedback before recruiting participants, by prompting an LLM for an aggregate clustering rather than raw individual sorts.
  • The low-level P1 simulation is reliably worse, so tool builders should avoid spending tokens on generating fake individual participants and instead target aggregate outputs.
  • Because P3 and P2 perform alike but P2 needs a larger output, clustering generation is the practical default for an automated card-sorting assistant.
  • Since model choice and prompt context barely move the scores, implementation can prioritise cost, speed, and output reliability over model selection.
  • Agreement degrades with study complexity, so simulations from large or difficult card sets should be treated as more tentative than those from small, clear sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is whether the 0.68 agreement is stable under a different ground-truth pipeline: comparing LLM outputs directly with participants' own named categories, or using hierarchical clustering instead of MDS plus k-means, could show whether the reported ordering of P1, P2, and P3 is an artifact of the evaluation pipeline.
  • The failure of raw-data simulation suggests LLMs store typical associations rather than individual variation; injecting explicit diversity constraints, for example forcing several distinct sorting rationales per simulated participant, might close part of the gap and would be a cheap testable variant.
  • The complexity effect implies a practical calibration rule: for card sets above roughly 45 cards or with domain-heavy labels, the simulator's output should be labelled as a first-pass hypothesis and weighted less in architecture decisions.
  • If LLM simulation continues to improve with model generations, the same 28-study benchmark could be rerun as a longitudinal check on whether the measured ceiling is a property of current models rather than of the task itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the Card Sorting Simulator, an LLM-based prototype that generates synthetic card sorting results, and evaluates it against 28 real-world open card sorting studies (1,399 participants). Four prompt variants are compared: raw-data simulation (P1), similarity-matrix generation (P2), aggregate clustering generation (P3), and clustering without context (P4), across GPT-4o, Gemini, Claude, and DeepSeek. The central empirical claims are that aggregate clustering prompts (P3) achieve a mean NMI of 0.68 against human-derived clusterings, that raw simulation (P1) is markedly worse, that study complexity lowers agreement, and that model choice and prompt context have only minor effects. The authors conclude that LLM-generated card sorting can provide useful preliminary feedback but should not replace human participants.

Significance. The paper's main strength is its ecological validity: the evaluation uses diverse, practitioner-created card sorting studies rather than laboratory tasks, and the multi-prompt, multi-model comparison is well suited to inform tool design. The statistical analyses (nonparametric tests, repeated-measures comparisons) are generally appropriate, and the authors are careful to check output errors and variability. If the absolute agreement claim can be properly calibrated, the paper would be a valuable empirical foundation for AI-augmented information architecture research, and it would extend the sparse prior work on LLM-simulated card sorting. The distinction between abstract clustering generation and low-level participant simulation is a useful design insight, as is the finding that adding readily available context did not improve results.

major comments (3)
  1. [§5.1, Table 1] The headline claim that P2/P3 reach NMI ≈ 0.68 and therefore show 'good degree of agreement' is not calibrated against chance. NMI as computed by scikit-learn is not chance-adjusted, and with the study-level median of about 8 clusters and median 34 cards, two independent random partitions with fixed marginal cluster sizes can attain a non-negligible expected NMI (roughly 0.3 under a chi-square approximation). The paper reports no permutation null distribution or any chance baseline, so the absolute interpretation of 0.68 is underdetermined. Please add a permutation test that shuffles card labels within the ground-truth cluster sizes, report Adjusted Mutual Information, or otherwise quantify how far P3's NMI is above the chance floor. The relative comparisons (P1 vs. P2/P3, complexity effects) are less affected, but the abstract's and §5.1's absolute claim of 'good agreement' depends on this calibration.
  2. [§4.5 and Figure 4] The ground truth is not raw human sorts but a derived clustering obtained by converting participant sorts into a similarity matrix, applying MDS, and running k-means with K selected by a knee rule. Agreement scores compare LLM clusterings to this constructed aggregate. To interpret NMI = 0.68 as 'reasonable agreement,' the paper needs a human-level reference: the same pipeline should be applied to individual real participants, and the average NMI/ARI between each participant's sort and the aggregate should be reported. This quantity is computable from the already-available dataset. Without it, the same 0.68 could mean that the LLM is near the typical human-to-aggregate consistency or far below it; the current text cannot distinguish these.
  3. [§4.5, Eq. (1), §5.4] The knee/elbow method automatically selects K for every clustering, including the LLM-generated ones, and the reported NMI, ARI, and edit-distance values depend on these K choices. There is no sensitivity analysis showing that the main conclusions are stable with respect to K determination. Because the complexity results in §5.4 also rely on the same pipeline, a sensitivity analysis over a plausible range of K, or a comparison with an alternative clustering/K-selection method, would materially strengthen the quantitative claims.
minor comments (5)
  1. [§5.1] The test name is misspelled as 'Willcoxon' and should be 'Wilcoxon.' In addition, the sample sizes n = 54 for P1/P2 and n = 224 for P3 are not explained; please clarify how many studies and repeated trials contribute to each comparison, since Table 1 text implies four GPT-4o trials per study.
  2. [Table 3] The table header 'LLM output format' does not match the rows, which are model names. Rename the header to 'Model'.
  3. [Table 2] The table layout is ambiguous: each row appears to report the averaged within-study standard deviation for four metrics, but the column headers are visually split so that 'mean SD' appears as a single block. Please format the table so that each metric has a distinct mean and SD column, matching the text's description.
  4. [§5.1, Figure 5] The observation that P1 produced completely identical output for all simulated participants in 5 of 28 studies is an important failure mode that should be mentioned in the limitations or discussion, not only in the qualitative results.
  5. [§3.3] The prompts were refined through an iterative process, but the manuscript does not describe which card sorting studies were used in that development process or whether any of the 28 evaluation studies were also used to tune prompts. Please clarify this to rule out potential leakage from prompt engineering into the reported results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core claim is an empirical benchmark against external human card-sorting data, with no fitted parameter, self-definitional target, or load-bearing self-citation chain.

full rationale

The paper's central claim is an empirical benchmark, not a derivation. The Card Sorting Simulator's outputs are generated by prompting LLMs with card lists and study context, and agreement is measured against clusterings derived from 28 real card sorting studies with 1,399 participants. No numeric parameter of the LLM or of the evaluation model is fitted to the ground-truth clusterings, and no equation defining the human target is fed into the simulator. The equal treatment of real and synthetic data through the similarity-matrix/MDS/k-means pipeline is explicitly a standardization choice (Section 4.5, Figure 4: 'LLM-generated and real card sorting results are processed identically, with simulated result data entering the pipeline at a step that reflects their initial state'), not a construction that guarantees agreement: P1 and P3 both pass through the relevant processing and yield very different scores (NMI 0.50 vs 0.68), and P3, the headline variant, enters at the clustering stage and is compared directly to the human-derived clustering. The iterative prompt-engineering described in Section 3.3 ('Prompts were designed through an iterative process of continuous empirical evaluation') is a development choice aimed at format compliance and clarity; the paper does not report fitting any parameter to the specific human clusterings, so any leakage concern is a validity risk, not a demonstrated circular reduction. The self-citations (Kuric et al. 2024, 2025a, 2025b; Krajcovic et al. 2025) are contextual and are not load-bearing for the empirical comparison. The absence of a chance baseline and of a human-to-aggregate calibration affects how NMI 0.68 should be interpreted, but that is a statistical-correctness and framing issue, not circularity. Section 6.2's limitations (Western samples, limited screening context, manual preprocessing) are honest scope limits and do not indicate that the results are equivalent to their inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

None of the agreement scores are fitted to the human data; the main empirical quantities come from an external benchmark. The free parameters are analytic choices (cluster count K, tercile splits, label complexity coding) rather than fitted constants. Axioms concern representativeness of the proprietary dataset, validity of the clustering pipeline as ground truth, and the assumption that output corrections did not change semantics.

free parameters (3)
  • Number of clusters K = per-study via knee/elbow; median 8 for real card sorting
    K is selected for each study and each method by maximum curvature of the WCSS curve; this choice determines the clusterings used in every comparison and is not independent of the data.
  • Card count tercile boundaries = Low <= 26, Medium 27-45, High >= 46
    Hand-chosen division of studies for RQ4; affects the complexity analysis but not the main agreement estimate.
  • Label complexity ratings = Simple n=10, Moderate n=9, Complex n=9
    Assigned by collaborative coding by two researchers with a third resolving conflicts; subjective and informs RQ4.
assumptions (4)
  • domain assumption The 28 UXtweak studies, after consent and quality review, are representative of real-world open card sorting practice.
    External validity of all conclusions rests on this; dataset is English-only, Western-skewed, and from a single online research tool (Section 4.3).
  • domain assumption Similarity matrix, MDS, and k-means with knee-point K faithfully reconstruct the shared mental model of card sorters.
    This pipeline defines the ground truth for all agreement scores and is not independently validated against raw participant sort patterns (Section 4.5, Figure 4).
  • domain assumption LLM output validation and manual correction do not materially change the semantic content of generated results.
    17.14% of prompts were regenerated and some outputs required manual correction; the paper asserts corrections address only format, but this is not empirically demonstrated.
  • standard math Nonparametric statistical tests (Kruskal-Wallis, Wilcoxon, Friedman) are appropriate for this non-normal, repeated-measures data.
    All significance claims depend on these tests; no violations or alternative modeling are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Card Sorting Simulator: Augmenting Design of Logical Information Architectures with Large Language Models." pith.science (2026). https://pith.science/paper/ZNZBN3AQ

@misc{pith2026250509478,
  author       = {Pith},
  title        = {Pith review of: Card Sorting Simulator: Augmenting Design of Logical Information Architectures with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNZBN3AQ}},
  note         = {Machine review of arXiv:2505.09478}
}
read the original abstract

Card sorting is a common ideation technique that elicits information on users' mental organization of content and functionality by having them sort items into categories. For more robust card sorting research, digital card sorting tools could benefit from providing quick automated feedback. Our objective of this research is to advance toward an instrument that applies artificial intelligence (AI) to augment card sorting. For this purpose, we develop the Card Sorting Simulator, a prototype tool that leverages Large Language Models (LLMs) to generate informative categorizations of cards. To illuminate how aligned the simulation is with card sorting by actual participants, and to inform the instrument's design decisions, we conducted a generalizability-focused comparative study. We obtained 28 pre-existing card sorting studies from real practitioners, comprising 1,399 participants, along with diverse contents and origins. With this dataset, we conducted a comprehensive and nuanced analysis of the agreement between actual card sorting results (clusterings of cards) and synthetic clusterings across a multitude of LLMs and prompt designs. Mutual information scores indicate a good degree of agreement to real result clustering, although similarity matrices also demonstrate inconsistencies from mental models, which can be attributed to their top-down nature. Furthermore, the number of cards or complexity of their labels impact the accuracy of its simulation. These findings bolster the case for AI augmentation in card sorting research as a source of meaningful preliminary feedback and highlight the need for further study for the development and validation of intelligent user research tools.

Figures

Figures reproduced from arXiv: 2505.09478 by the authors.

Figure 1
Figure 1. Overview of AI-augumented Information Architecture (IA) ideation and validation workflow. Based on card sorting and tree testing information, as well as user (UX researcher) analytical demands, the Card Sorting Simulator generates outputs that support interactive analysis and rapid adaptation to findings. 3.2 General design decisions The prerequisite for implementing the system for AI-augmented research of informati… view at source ↗
Figure 2
Figure 2. General structure of the prompt template used for generating card sorting results. Sections are clearly delimited. Input information (left) originates from the configuration of the card sorting study (contextual details, card list) or is defined as part of the approach for generating results (experimental design variant). For specific prompt variant templates, refer to Appendix A Prompt templates. 4 Experimental met… view at source ↗
Figure 3
Figure 3. Domain distribution of card sorting studies in the analyzed dataset. For analysis of the effects of study parameters (RQ4), studies were organized into terciles based on the number of cards, being Low (≤ 26), Medium (between 27 and 45) and High (≥ 46). The complexity of card labels was determined through collaborative coding (see 4.2 Experiment design and procedure), resulting in a division of studies into three gro… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Simulated card sorting (CS) evaluation pipeline. LLM-generated and real card sorting results are processed identically, with simulated result data entering the pipeline at a step that reflects their initial state (raw data, similarity matrix, clusters), as determined b…
Figure 5
Figure 5. Figure 5: Example of similarity matrices calculated from real card sorting data and experimental generative methods of the Card Sorting Simulator, demonstrating structural divergence. GPT-4o was used for result generation [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Normalized Mutual Information scores across four LLMs. Visually, boxplots show only negligible differences between models. The number of cards as an indicator of the complexity of a card sorting study has a significant link to the accuracy of simulated ca…
Figure 7
Figure 7. Figure 7: Comparison of Normalized Mutual Information scores across three card count levels (a) and three card label complexity levels (b). Higher numbers of cards increase the scale of deviations between the simulations and real card sorting results, while simple card labels re…
Figure 8
Figure 8. Figure 8: Comparison of card clusterings between examples of studies with different complexity (a) 10 cards, simple labels, NMI = 1 (b) 27 cards, simple labels, NMI = 0.8, (c) 34 cards, moderate labels, NMI = 0.6, (d) 55 card, complex labels, NMI = 0.42. Clusterings from real pa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 31 canonical work pages

  1. [1]

    Card sorting: Uncover users' mental models for better information architecture

    Samhita Tankala and Katie Sherwin. Card sorting: Uncover users' mental models for better information architecture. https://www.nngroup.com/articles/card-sorting-definition/, 2024

    work page 2024

  2. [2]

    Understanding Your Users: A Practical Guide to User Research Methods

    Kathy Baxter, Catherine Courage, and Kelly Caine. Understanding Your Users: A Practical Guide to User Research Methods. Morgan Kaufmann, 2015. URL https://doi.org/10.1016/C2013-0-13611-2

    work page doi:10.1016/c2013-0-13611-2 2015

  3. [3]

    Lewis and Jeff Sauro

    James R. Lewis and Jeff Sauro. Usability and user experience: Design and evaluation, chapter 38, pages 972--1015. John Wiley & Sons, Ltd, 2021. ISBN 9781119636113. URL https://doi.org/10.1002/9781119636113.ch38

    work page doi:10.1002/9781119636113.ch38 2021

  4. [4]

    Munim, Iyolita Islam, Md

    Kazi Md. Munim, Iyolita Islam, Md. Musfiqur Rahman Milton, Laila Arzuman Ara, Faiz Al Faisal, and Muhammad Nazrul Islam. Exploring the impact of design technique on usability: A case study on designing the ehealth websites using card sorting and interactive dialogue model. Engineering Reports, 6 0 (3): 0 e12738, 2024. URL https://doi.org/10.1002/eng2.12738

    work page doi:10.1002/eng2.12738 2024

  5. [6]

    Task-based open card sorting: Towards a new method to produce usable information architectures

    Christos Katsanos, Vasileios Christoforidis, and Christina Demertzi. Task-based open card sorting: Towards a new method to produce usable information architectures. In Hirohiko Mori and Yumi Asahi, editors, Human Interface and the Management of Information, pages 68--80, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-35132-7. URL https://doi.org/...

    work page doi:10.1007/978-3-031-35132-7_5 2023

  6. [7]

    Effect of self-efficacy on open card sorts for websites

    Christos Katsanos, Georgia Zafeiriou, and Alexandros Liapis. Effect of self-efficacy on open card sorts for websites. In Sakae Yamamoto and Hirohiko Mori, editors, Human Interface and the Management of Information: Visual and Information Design, pages 75--87, Cham, 2022. Springer International Publishing. ISBN 978-3-031-06424-1. URL https://doi.org/10.100...

    work page doi:10.1007/978-3-031-06424-1_7 2022

  7. [8]

    Participatory theme elicitation: open card sorting for user led qualitative data analysis

    Paul Best, Jennifer Badham, Tracey McConnell, and Ruth F Hunter and. Participatory theme elicitation: open card sorting for user led qualitative data analysis. International Journal of Social Research Methodology, 25 0 (2): 0 213--231, 2022. URL https://doi.org/10.1080/13645579.2021.1876616

    work page arXiv 2022

  8. [9]

    Evaluating chinese government wechat official accounts in public service delivery: A user-centered approach

    Tingting Jiang, Ying Wang, Tianqianjin Lin, and Lina Shangguan. Evaluating chinese government wechat official accounts in public service delivery: A user-centered approach. Government Information Quarterly, 38 0 (1): 0 101548, 2021 a . ISSN 0740-624X. URL https://doi.org/10.1016/j.giq.2020.101548

    arXiv 2021

Show all 72 references
  1. [10]

    Mac \'i as and Alma L

    Jos \'e A. Mac \'i as and Alma L. Cul \'e n. Enhancing decision-making in user-centered web development: a methodology for card-sorting analysis. World Wide Web, 24 0 (6): 0 2099--2137, Nov 2021. ISSN 1573-1413. URL https://doi.org/10.1007/s11280-021-00950-y

    2021 doi

  2. [11]

    Anisbed Naranjo-Rojas, Luis Ángel Perula-de Torres, and Guillermo Molina-Recio. Patients, caregivers, and healthcare professionals' needs when designing the content of a mobile application for the clinical monitoring of patients with chronic obstructive pulmonary disease and h...

    2022

  3. [12]

    Taxonomy for building permit system - organizing knowledge for building permit digitalization

    Judith Fauth, Tanya Bloch, Francesca Noardo, Nicholas Nisbet, Stefanie-Brigitte Kaiser, Peter Nørkjær Gade , and Jernej Tekavec. Taxonomy for building permit system - organizing knowledge for building permit digitalization. Advanced Engineering Informatics, 59: 0 102312, 2024....

    2024

  4. [13]

    A taxonomy of social cues for conversational agents

    Jasper Feine, Ulrich Gnewuch, Stefan Morana, and Alexander Maedche. A taxonomy of social cues for conversational agents. International Journal of Human-Computer Studies, 132: 0 138--161, 2019. ISSN 1071-5819. URL https://doi.org/10.1016/j.ijhcs.2019.07.009

    2019 doi

  5. [14]

    Can ai serve as a substitute for human subjects in software engineering research? Automated Software Engineering, 31 0 (1): 0 13, Jan 2024

    Marco Gerosa, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. Can ai serve as a substitute for human subjects in software engineering research? Automated Software Engineering, 31 0 (1): 0 13, Jan 2024. ISSN 1573-7535. URL https://doi.org/10.1007/s10515-023-00409-6

    2024 doi

  6. [15]

    Is usability testing valid with prototypes where clickable hotspots are highlighted upon misclick? Journal of Systems and Software, 226: 0 112446, 2025

    Matus Krajcovic, Peter Demcak, and Eduard Kuric. Is usability testing valid with prototypes where clickable hotspots are highlighted upon misclick? Journal of Systems and Software, 226: 0 112446, 2025. ISSN 0164-1212. URL https://doi.org/10.1016/j.jss.2025.112446

    2025

  7. [16]

    Heuristic and think aloud method to evaluate the low fidelity prototype of game-based language learning application

    Kashif Ishaq, Fadhilah Rosdi, Nor Azan Mat Zin, and Adnan Abid. Heuristic and think aloud method to evaluate the low fidelity prototype of game-based language learning application. In 2021 International Conference on Innovative Computing (ICIC), pages 1--10, 2021. URL https://...

    2021

  8. [17]

    Rosenfeld, P

    L. Rosenfeld, P. Morville, and J. Arango. Information Architecture: For the Web and Beyond. O'Reilly Media, 2015. ISBN 9781491913543

    2015

  9. [18]

    6 - information architecture and web navigation

    Andrew Schall. 6 - information architecture and web navigation. In Jennifer Romano Bergstrom and Andrew Jonathan Schall, editors, Eye Tracking in User Experience Design, pages 139--162. Morgan Kaufmann, Boston, 2014. ISBN 978-0-12-408138-3. URL https://doi.org/10.1016/B978-0-1...

    2014 doi

  10. [19]

    Validity of the open card sorting method for producing website information structures

    Maria Ntouvaleti and Christos Katsanos. Validity of the open card sorting method for producing website information structures. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA '22, pages 1--7, New York, NY, USA, 2022. Association f...

    2022

  11. [20]

    Thomas and Ian Johnson

    Robert L. Thomas and Ian Johnson. Merging methodologies: Combining individual and group card sorting. In Aaron Marcus, editor, Design, User Experience, and Usability. Design Philosophy, Methods, and Tools, pages 417--426, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. I...

    2013 doi

  12. [21]

    Lewis and Peter Hepburn

    Krystal M. Lewis and Peter Hepburn. Open card sorting and factor analysis: a usability case study. The Electronic Library, 28 0 (3): 0 401--416, Jan 2010. ISSN 0264-0473. URL 10.1108/02640471011051981

    2010 doi

  13. [22]

    Redesign based on card sorting: How universally applicable are card sort results? In Vincent G

    Jobke Wentzel, Nienke Beerlage de Jong, and Thea van der Geest. Redesign based on card sorting: How universally applicable are card sort results? In Vincent G. Duffy, editor, Digital Human Modeling: Applications in Health, Safety, Ergonomics and Risk Management, pages 381--388...

    2016 doi

  14. [23]

    Linking card sorting to browsing performance--are congruent municipal websites more efficient to use? Behaviour & information technology, 35 0 (6): 0 452--470, 2016

    Martin Schmettow and Jan Sommer. Linking card sorting to browsing performance--are congruent municipal websites more efficient to use? Behaviour & information technology, 35 0 (6): 0 452--470, 2016. URL https://doi.org/10.1080/0144929X.2016.1157207

    2016

  15. [24]

    Validation of information architecture: Cross-methodological comparison of tree testing variants and prototype user testing

    Eduard Kuric, Peter Demcak, and Matus Krajcovic. Validation of information architecture: Cross-methodological comparison of tree testing variants and prototype user testing. Information and Software Technology, 183: 0 107740, 2025 a . ISSN 0950-5849. URL https://doi.org/10.101...

    2025

  16. [25]

    Adapting card sorting for blind people: Evaluation of the interaction design in talkback

    Teresita de Jesús Álvarez Robles, Francisco Javier Álvarez Rodríguez, Edgard Benítez-Guerrero, and Cristian Rusu. Adapting card sorting for blind people: Evaluation of the interaction design in talkback. Computer Standards & Interfaces, 66: 0 103356, 2019. ISSN 0920-5489. URL ...

    2019

  17. [26]

    Hybrid digital card sorting: New research technique or mere variant? In Malte Smits, editor, Information for a Better World: Shaping the Global Future, pages 50--67, Cham, 2022

    Elke Greifeneder and Paulina Bressel. Hybrid digital card sorting: New research technique or mere variant? In Malte Smits, editor, Information for a Better World: Shaping the Global Future, pages 50--67, Cham, 2022. Springer International Publishing. ISBN 978-3-030-96960-8. UR...

    2022 doi

  18. [27]

    A systematic review of the comparison of different types of card sorting

    Elinda Tchivi, Bibhya Sharma, and Sione Paea. A systematic review of the comparison of different types of card sorting. IEEE Access, 13: 0 52334--52352, 2025. URL https://doi.org/10.1109/ACCESS.2025.3552949

    2025

  19. [28]

    Keeley, Michael C

    Ethan Lantz, Jared W. Keeley, Michael C. Roberts, Maria Elena Medina-Mora, Pratap Sharan, and Geoffrey M. Reed. Card sorting data collection methodology: How many participants is most efficient? Journal of Classification, 36 0 (3): 0 649--658, Oct 2019. ISSN 1432-1343. URL htt...

    2019 doi

  20. [29]

    Card sorting, category validity, and contextual navigation

    Stefano Bussolon. Card sorting, category validity, and contextual navigation. Journal of Information Architecture, 1 0 (2), 2009

    2009

  21. [30]

    A supporting tool for enhancing user’s mental model elicitation and decision-making in user experience research

    Marina Mart \' n and Jos \'e A Mac \' as. A supporting tool for enhancing user’s mental model elicitation and decision-making in user experience research. International Journal of Human--Computer Interaction, 39 0 (1): 0 183--202, 2023. URL https://doi.org/10.1080/10447318.202...

    2023

  22. [31]

    Webb, Robert Zwitser, and Pablo Cesar

    Abdallah El Ali, Liam Ashby, Andrew M. Webb, Robert Zwitser, and Pablo Cesar. Uncovering perceived identification accuracy of in-vehicle biometric sensing. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications:...

    2019

  23. [32]

    Card sort analysis best practices

    Carol Righi, Janice James, Michael Beasley, Donald L Day, Jean E Fox, Jennifer Gieber, Chris Howe, and Laconya Ruby. Card sort analysis best practices. Journal of Usability Studies, 8 0 (3): 0 69--89, 2013

    2013

  24. [33]

    Householders’ mental models of domestic energy consumption: Using a sort-and-cluster method to identify shared concepts of appliance similarity

    Elizabeth Gabe-Thomas, Ian Walker, Bas Verplanken, and Gavin Shaddick. Householders’ mental models of domestic energy consumption: Using a sort-and-cluster method to identify shared concepts of appliance similarity. PLOS ONE, 11 0 (7): 0 1--15, 07 2016. URL https://doi.org/10....

    2016 doi

  25. [34]

    Oswald, and Claudia Ziegler Acemyan

    Ian Robertson, Philip Kortum, Frederick L. Oswald, and Claudia Ziegler Acemyan. Novices perform like experts on a closed card sort but not an open card sort. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64 0 (1): 0 1249--1253, 2020. URL https://doi.o...

    2020 doi

  26. [35]

    Ai assistance for ux: A literature review through human-centered ai, 2024 a

    Yuwen Lu, Yuewen Yang, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. Ai assistance for ux: A literature review through human-centered ai, 2024 a . URL https://arxiv.org/abs/2402.06089

    2024 arXiv

  27. [36]

    From explainable to interactive ai: A literature review on current trends in human-ai interaction

    Muhammad Raees, Inge Meijerink, Ioanna Lykourentzou, Vassilis-Javed Khan, and Konstantinos Papangelis. From explainable to interactive ai: A literature review on current trends in human-ai interaction. International Journal of Human-Computer Studies, 189: 0 103301, 2024. ISSN ...

    2024

  28. [37]

    What is human-centered about human-centered ai? a map of the research landscape

    Tara Capel and Margot Brereton. What is human-centered about human-centered ai? a map of the research landscape. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, pages 1--23, New York, NY, USA, 2023. Association for Computing Machinery....

    2023

  29. [38]

    Data processing and analysing techniques in ux research

    Aigerim Aitim and Muslima Abdulla. Data processing and analysing techniques in ux research. Procedia Computer Science, 251: 0 591--596, 2024. ISSN 1877-0509. URL https://doi.org/10.1016/j.procs.2024.11.154. 15th International Conference on Emerging Ubiquitous Systems and Perva...

    2024 doi

  30. [39]

    Comparing apples and oranges: Human and computer clustered affinity diagrams under the microscope

    Parzival Borlinghaus and Stephan Huber. Comparing apples and oranges: Human and computer clustered affinity diagrams under the microscope. In Proceedings of the 26th International Conference on Intelligent User Interfaces, IUI '21, page 413–422, New York, NY, USA, 2021. Associ...

    2021

  31. [40]

    Brubaker

    Jialun Aaron Jiang, Kandrea Wade, Casey Fiesler, and Jed R. Brubaker. Supporting serendipity: Opportunities and challenges for human-ai collaboration in qualitative analysis. Proc. ACM Hum.-Comput. Interact., 5 0 (CSCW1), April 2021 b . URL https://doi.org/10.1145/3449168

    2021 doi

  32. [41]

    How do users like this feature? a fine grained sentiment analysis of app reviews

    Emitza Guzman and Walid Maalej. How do users like this feature? a fine grained sentiment analysis of app reviews. In 2014 IEEE 22nd International Requirements Engineering Conference (RE), pages 153--162, 2014. URL https://doi.org/10.1109/RE.2014.6912257

    2014

  33. [42]

    Satisfied or not: user experience of mobile augmented reality in using natural language processing techniques on review comments

    Yeonju Jang and Eunil Park. Satisfied or not: user experience of mobile augmented reality in using natural language processing techniques on review comments. Virtual Reality, 26 0 (3): 0 839--848, Sep 2022. ISSN 1434-9957. URL https://doi.org/10.1007/s10055-021-00599-y

    2022 doi

  34. [43]

    Generating personas using llms and assessing their viability

    Andreas Schuller, Doris Janssen, Julian Blumenr\" o ther, Theresa Maria Probst, Michael Schmidt, and Chandan Kumar. Generating personas using llms and assessing their viability. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA '24, page...

    2024

  35. [44]

    Iona Gessinger, Katie Seaborn, Madeleine Steeds, and Benjamin R. Cowan. Chatgpt and me: First-time and experienced users’ perceptions of chatgpt’s communicative ability as a dialogue partner. International Journal of Human-Computer Studies, 194: 0 103400, 2025. ISSN 1071-5819....

    2025

  36. [45]

    Ziang Xiao, Michelle X. Zhou, Q. Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. Tell me about yourself: Using an ai-powered chatbot to conduct conversational surveys with open-ended questions. ACM Trans. Comput.-Hum. Interact., 27 0 (3), June 2020. ISSN 107...

    2020 doi

  37. [46]

    Unmoderated usability studies evolved: Can gpt ask useful follow-up questions? International Journal of Human–Computer Interaction, 0 0 (0): 0 1--18, 2024

    Eduard Kuric, Peter Demcak, and Matus Krajcovic. Unmoderated usability studies evolved: Can gpt ask useful follow-up questions? International Journal of Human–Computer Interaction, 0 0 (0): 0 1--18, 2024. URL https://doi.org/10.1080/10447318.2024.2427978

    2024

  38. [47]

    Opportunities and risks of llms in survey research

    David M Rothschild, James Brand, Hope Schroeder, and Jenny Wang. Opportunities and risks of llms in survey research. Available at SSRN, 2024

    2024

  39. [48]

    Zavod, Donald E

    Merrill J. Zavod, Donald E. Rickert, Steven H. Brown, and State Farm Mutual. The automated card-sort as an interface design tool: A comparison of products. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46 0 (5): 0 646--650, 2002. URL https://doi.org/1...

    2002 doi

  40. [49]

    Comparing chatgpt to card sorting results, 2024

    Jeff Sauro, Will Schiavone, and Jim Lewis. Comparing chatgpt to card sorting results, 2024. URL https://measuringu.com/comparing-chatgpt-to-card-sorting-results/

    2024

  41. [50]

    a m\" a l\

    Perttu H\" a m\" a l\" a inen, Mikke Tavast, and Anton Kunnari. Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, pages 1--19, New York, NY, USA, 2...

    2023

  42. [51]

    a m\" a l\

    Perttu H\" a m\" a l\" a inen, Mikke Tavast, and Anton Kunnari. Neural language models as what if? -engines for hci research. In Companion Proceedings of the 27th International Conference on Intelligent User Interfaces, IUI '22 Companion, page 77–80, New York, NY, USA, 2022. A...

    2022

  43. [52]

    Synthetic users: insights from designers’ interactions with persona-based chatbots

    (Eric) Heng Gu, Senthil Chandrasegaran, and Peter Lloyd. Synthetic users: insights from designers’ interactions with persona-based chatbots. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 39: 0 e2, 2025. URL https://doi.org/10.1017/S0890060424000283

    2025 doi

  44. [53]

    Systematic literature review of automation and artificial intelligence in usability issue detection, 2025 b

    Eduard Kuric, Peter Demcak, Matus Krajcovic, and Jan Lang. Systematic literature review of automation and artificial intelligence in usability issue detection, 2025 b . URL https://arxiv.org/abs/2504.01415

    2025 arXiv

  45. [54]

    Simuser: Generating usability feedback by simulating various users interacting with mobile applications

    Wei Xiang, Hanfei Zhu, Suqi Lou, Xinli Chen, Zhenghua Pan, Yuping Jin, Shi Chen, and Lingyun Sun. Simuser: Generating usability feedback by simulating various users interacting with mobile applications. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Sy...

    2024

  46. [55]

    Ai-augmented surveys: Leveraging large language models and surveys for opinion prediction, 2024

    Junsol Kim and Byungkyu Lee. Ai-augmented surveys: Leveraging large language models and surveys for opinion prediction, 2024. URL https://arxiv.org/abs/2305.09620

    2024 arXiv

  47. [56]

    Sanders, Alex Ulinich, and Bruce Schneier

    Nathan E. Sanders, Alex Ulinich, and Bruce Schneier. Demonstrations of the potential of ai-based political issue polling, 2023. URL https://arxiv.org/abs/2307.04781

    2023 arXiv

  48. [57]

    From system 1 to system 2: A survey of reasoning large language models, 2025

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhiwei Li, Bao-Long Bi, Ling-Rui Mei, Junfeng Fang, Zhijiang Guo, Le Song, and Cheng-Lin Liu. From sys...

    2025 arXiv

  49. [58]

    The fine line between automation and augmentation in website usability evaluation

    Andrea Esposito, Giuseppe Desolda, and Rosa Lanzilotti. The fine line between automation and augmentation in website usability evaluation. Scientific Reports, 14 0 (1): 0 10129, May 2024. ISSN 2045-2322. URL https://doi.org/10.1038/s41598-024-59616-0

    2024 doi

  50. [59]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276

    2024 arXiv

  51. [60]

    Gemini: A family of highly capable multimodal models, 2024

    Google. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv.org/abs/2312.11805

    2024 arXiv

  52. [61]

    Llms and generative agent-based models for complex systems research

    Yikang Lu, Alberto Aleta, Chunpeng Du, Lei Shi, and Yamir Moreno. Llms and generative agent-based models for complex systems research. Physics of Life Reviews, 51: 0 283--293, 2024 b . ISSN 1571-0645. URL https://doi.org/10.1016/j.plrev.2024.10.013

    2024 doi

  53. [62]

    Can large language models help predict results from a complex behavioural science study? Royal Society Open Science, 11 0 (9): 0 240682, 2024

    Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis Uhlmann, null null, and Thomas Pfeiffer. Can large language models help predict results from a complex behavioural science study? Royal Society Open Science, 11 0 (9): 0 240682, 2024....

    2024 doi

  54. [63]

    Emotional intelligence of large language models

    Xuena Wang, Xueting Li, Zi Yin, Yue Wu, and Jia Liu. Emotional intelligence of large language models. Journal of Pacific Rim Psychology, 17: 0 18344909231213958, 2023. URL https://doi.org/10.1177/18344909231213958

    2023 doi

  55. [64]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15 ...

    2024 doi

  56. [65]

    Prompt engineering in large language models

    Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabende. Prompt engineering in large language models. In I. Jeena Jacob, Selwyn Piramuthu, and Przemyslaw Falkowski-Gilski, editors, Data Intelligence and Cognitive Informatics, pages 387--402, Singapore, 20...

    2024

  57. [66]

    A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023. URL https://doi.org/10.48550/arXiv.2302.11382

    2023 doi arXiv

  58. [67]

    A survey of large language models, 2025

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2025 arXiv

  59. [68]

    Text generation and prompting, 2025

    OpenAI. Text generation and prompting, 2025. URL https://platform.openai.com/docs/guides/text. Accessed: 2025-04-22

    2025

  60. [69]

    Finding a "kneedle" in a haystack: Detecting knee points in system behavior

    Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a "kneedle" in a haystack: Detecting knee points in system behavior. In 2011 31st International Conference on Distributed Computing Systems Workshops, pages 166--171, 2011. URL https://doi.org/10.1109/I...

    2011 doi

  61. [70]

    Unsupervised feature selection by self-paced learning regularization

    Wei Zheng, Xiaofeng Zhu, Guoqiu Wen, Yonghua Zhu, Hao Yu, and Jiangzhang Gan. Unsupervised feature selection by self-paced learning regularization. Pattern Recognition Letters, 132: 0 4--11, 2020. ISSN 0167-8655. URL https://doi.org/10.1016/j.patrec.2018.06.029. Multiple-Task ...

    2020 doi

  62. [71]

    The data science handbook

    Field Cady. The data science handbook. John Wiley & Sons, 2024

    2024

  63. [72]

    A comparison of card-sorting analysis methods

    Ather Nawaz. A comparison of card-sorting analysis methods. In APCHI'12. Proceedings of the 10th Asia Pacific Conference on Computer-Human Interaction, pages 583--592. Association for Computing Machinery, 2012

    2012

  64. [73]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Inform...

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.