Pith. sign in

REVIEW 4 major objections 3 minor 46 references

LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis

T0 review · 4 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An open-source tool with two alignment metrics shows that LLM-assisted qualitative coding replicates a designer's surface patterns from few examples but not the deeper interpretive lens behind them.

desk verdict An honest, well-scoped HCI paper with a useful open-source tool and a real empirical observation, but the abstract's headline claim overreaches the evidence. read the letter →

arxiv 2504.16671 v1 pith:CUYUSCDZ submitted 2025-04-23 cs.HC

Joel Oksanen , Andrés Lucero , Perttu Hämäläinen This is my paper · ORCID
classification cs.HC
keywords qualitativecodinglargelanguagemodelsresearcher-AIalignmentin-contextlearningIntersectionoverUnionModifiedHausdorffDistanceresearchfordesignhuman-AIcollaboration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LLMCode, an open-source tool that scores how closely an LLM's qualitative coding matches a designer's own coding, using two metrics: character-level Intersection over Union for what text gets highlighted, and Modified Hausdorff Distance between code-label embeddings for how the highlighted text is labelled. Across two studies with 26 designers, the paper argues that a state-of-the-art LLM can learn to reproduce the surface patterns of a designer's coding from a few examples, but its ability to extrapolate the designer's broader interpretive lens to new, unfamiliar material is limited. The result matters because research for design relies on each designer's situated, reflexive perspective, and tools that only imitate surface patterns could flatten the diversity of insights while appearing trustworthy. The paper also documents a reciprocal dynamic: designers refine the model's outputs, but also revise their own codes in response to the model's suggestions. The authors position the two metrics as a practical way to make that alignment visible and to guide iterative example selection.

What carries the argument

The carrying mechanism is the pair of alignment metrics inside LLMCode. IoU (Intersection over Union, character-level) measures overlap between the text segments a human highlights and the segments the model highlights, capturing agreement about what content is salient. Modified Hausdorff Distance (MHD) maps the human's and the model's code labels into a shared embedding space and measures the average distance between the two point clouds, capturing whether the labels are semantically related even when wording differs. Together the two metrics make the 'lens' partially visible: IoU tracks focus, MHD tracks conceptual association, and their joint trend across example-set size is the evidence base for the paper's claim about surface versus deep alignment.

What would settle it

Give designers a text that contains a genuinely new theme, ask them to code it, have the LLM code it after few-shot training on a different theme, and check whether the model's codes, when shown to an independent panel, are judged to reflect the designer's reasoning rather than merely reuse surface vocabulary. If the model's codes are repeatedly judged as sharing the designer's interpretive stance even when IoU and MHD are low, the paper's central limitation claim would be refuted. Alternatively, measure whether designers' own codes change systematically toward the model's embeddings when no explicit alignment instruction is given; if they do not, the reciprocal-influence claim weakens.

Watch

Extended reading notes

Core claim

The central claim is that LLM-assisted qualitative coding can be evaluated and steered by comparing highlights and code labels against a single designer's annotations, and that under this comparison the model's apparent learning is mostly surface-level. In Study 1, increasing the number of few-shot examples raised IoU and lowered MHD, but the improvement tracked how representative the examples were of the test texts; when test texts' codes were dissimilar from the example set, the model's output diverged, with a Pearson correlation of 0.53 between that dissimilarity and the model's error. The paper reads this as evidence that the model reuses codes it has seen rather than inductively adopting the researcher's evolving viewpoint. In Study 2, designers who iteratively refined their examples improved IoU beyond a random baseline but the MHD trend was less conclusive, and think-aloud data showed participants adopting some of the model's codes as their own. The paper's conclusion is that in-context learning supports deductive-style application of established codes, but genuine alignment with a designer's emergent interpretation remains unmet and requires human oversight.

Load-bearing premise

The load-bearing premise is that character-level highlight overlap (IoU) and embedding distance between code labels (MHD) adequately capture what it means for an AI to share a designer's interpretive lens, so that the conclusion about 'surface-level' emulation rests on these two measures rather than on an independent test of interpretive depth.

Editorial extensions

If this is right

  • Researchers can use IoU and MHD to locate texts where the model's interpretation diverges, sort by disagreement, and decide where to intervene.
  • Scaling manual coding to larger corpora is most reliable when the codebook is already established and new texts stay thematically close to the annotated examples.
  • When novel themes appear in new data, the model should not be expected to keep pace without human recoding or new examples.
  • Designers will change their own codes in response to the model's output, so the baseline for alignment shifts during iteration and alignment scores must be read with that in mind.
  • Separating validation and test sets during example iteration is necessary for trustworthy alignment claims, and the paper spells out this recommendation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The two metrics could be borrowed by other qualitative tools as a cheap proxy for 'who is driving the interpretation,' but character overlap and embedding distance will miss structural reinterpretations where a designer re-segments text or renames codes in ways embeddings do not capture.
  • A testable extension is to run the same few-shot scaling analysis with other LLMs and embedding models to see whether the apparent plateau in inductive alignment is model-specific or a general property of in-context learning.
  • The reciprocal-influence finding implies that reported alignment gains may be inflated because the human half of the pair is drifting toward the model; using a pre-registered, separately-coded test set would quantify that drift directly.
  • A concrete design principle follows from the paper's observations: future tools should let users show corrections ('you misunderstood this text') rather than only positive examples, aligning the interaction with the chat mental models users already have.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces LLMCode, an open-source tool for LLM-assisted qualitative coding that integrates two alignment metrics, character-level Intersection over Union (IoU) and embedding-based Modified Hausdorff Distance (MHD), to compare human and LLM annotations. The central research questions concern how to measure researcher-AI alignment, whether an LLM can emulate an individual designer's interpretive perspective through in-context learning, and how LLM assistance shapes design research insights. Two studies are reported: Study 1 with 19 design students who manually coded forum data and then used LLMCode in a notebook environment, and Study 2 with 7 designers who used a graphical interface to iteratively refine few-shot examples and prompts. The paper reports that IoU and MHD improve with more few-shot examples, that the model's performance degrades when texts are dissimilar to the examples, and that iterative human refinement yields improvements over a random baseline, especially for IoU. Qualitative analysis suggests participants both adapted to and were influenced by the model's outputs. The authors conclude that while LLMs can capture surface patterns and perform well in deductive coding, they are limited in emulating the designer's deeper interpretive lens, and they discuss overfitting and the need for held-out test sets.

Significance. The paper addresses a timely and important problem: how to evaluate and support appropriate reliance on LLM-assisted qualitative analysis in research-for-design contexts, where interpretive depth is central. The main contributions are the design and open-source release of a tool with two concrete, computable alignment metrics, and two empirical studies that provide qualitative insights into how designers interact with such a tool, including reciprocal influence between human and AI coding. The qualitative findings, particularly participants' mental models of few-shot example selection and their willingness to adopt AI-suggested codes, are valuable for the HCI and CSCW communities. If the central claims were fully supported, the work would provide a practical framework for calibrating trust in LLM-driven qualitative coding. However, the quantitative evidence is thin and, as the paper itself acknowledges, Study 2 has no held-out test set; the 'limited deeper interpretive lens' conclusion is inferred from surface and semantic similarity metrics without a human-human baseline or an independent measure of interpretive depth.

major comments (4)
  1. [Section 5.2 and Section 6.3] The claim that human oversight improves alignment is not fully established because the reported improvements are computed on the validation set used to iteratively select examples and refine prompts. The paper explicitly acknowledges in Section 6.3 that no separate test set was employed and that participants' own annotations may have shifted during iteration, yet Section 5.2 presents the improvement over the random baseline as a finding. This is a validation-loop circularity: the examples were chosen using the same IoU and MHD scores that are then reported as the outcome, so part of the observed gain may reflect optimization toward the evaluation metric rather than genuine generalization. To support the claim, the authors should either re-analyze the data on a held-out test set (even a small one, coded after the iteration phase) or explicitly reframe Section 5.2 as demonstrating validation-set alignment, with the generalization question left open.
  2. [Section 3.1, Section 5.1, Section 6.1] The central negative claim—that the LLM cannot emulate a designer's deeper interpretive lens—is derived from IoU and MHD, which measure character-level highlight overlap and embedding distance between code labels. These are surface and semantic-similarity measures, not direct measures of interpretive depth. The paper itself concedes in Section 5.1.1 that improvement in MHD may reflect codebook convergence rather than deeper emulation of the researcher's perspective. Moreover, no human-human baseline is reported, so the observed divergence is uncalibrated: in a reflexive, constructivist coding task, disagreement between two human coders could plausibly be of a similar magnitude. The authors should either add a human-human comparison (e.g., have two human coders code the same texts and compute the same metrics) or restrict the conclusion to 'the model's outputs are not highly similar to this specific designer's annotations,' removing the unsupported inference about 'deeper interpretive lens.'
  3. [Abstract, Section 7, Section 5.1.1] The abstract and conclusion state that 'the model performs well with deductive coding,' but no direct deductive coding experiment is reported. No study condition uses a fixed, predefined codebook applied to new texts; the claim is instead inferred from the MHD behavior when codes already appear in the few-shot examples (Section 5.1.1). This is an overgeneralization from an indirect observation. The authors should either add a direct evaluation of deductive coding (e.g., supplying a fixed codebook and measuring assignment accuracy) or soften the claim to something like 'the model can apply seen codes to related texts,' which is what the current data support.
  4. [Section 5.1.2, Figure 5] The correlation evidence in Figure 5 is reported as a Pearson coefficient of 0.53 with no confidence interval, no significance test, and no accounting for the non-independence of multiple texts from the same participant. With N=8 participants, this is thin quantitative support for the statement that 'the model performs better on texts that are more similar to its example set.' Additionally, the number of K-Means clusters (five) is presented without justification, and the analysis depends on this free parameter. A mixed-effects model with participant as a random effect, or at minimum bootstrap confidence intervals for the correlation, would materially strengthen the claim.
minor comments (3)
  1. [Section 5.3] There is a duplicated word in the sentence 'allowing themes to to be determined organically'; it should read 'allowing themes to be determined organically.'
  2. [Table 1 caption] The caption uses 'Hausdorff distance' while the paper's metric is the Modified Hausdorff Distance (MHD). Please use the full term consistently, and clarify in the caption that MHD is an average of embedding-based cosine distances, not the original point-cloud distance, since the interpretation of the values depends on this.
  3. [Section 3.2, Section 4.1] The description of example-set selection differs between the interface description (manual selection) and the Study 1 analysis (automatic chronological selection for the learning-curve experiment). Consider adding a sentence in Section 5.1.1 clarifying that the automatic selection was for the controlled comparison and that manual selection is the intended use, to avoid confusion about how the results relate to the tool's workflow.

Circularity Check

1 steps flagged · score 4.0 of 10

Study 2's reported alignment gains are measured on the same validation set used to select examples via the same IoU/MHD metrics, with no held-out test set, so part of the measured improvement is optimization toward the evaluation metric.

  1. fitted input called prediction [Section 4.2 (procedure) and Section 6.3 (training/validation/test caveat)]
    "During the second phase, participants were instructed to complete two additional iterations of example selection and LLM-assisted coding using the tool’s features, with the objective of improving the code quality metrics. ... In Study 2, we did not employ a separate test set due to the time-intensive nature of manual coding and to avoid participant fatigue. As a result, while the score improvements reported in Section 5.2 signal that the tool helps researchers align their analysis within the validation set, these scores may not fully reflect the true human-AI alignment."

    The few-shot examples and prompt instructions are the tunable inputs, and participants are explicitly directed to improve IoU and MHD. The interface shows the metrics per text and lets users sort by them, so example selection is guided by the same scores that are then reported as evidence of improved alignment. Because no held-out test set is used in Study 2, the gains reported in Section 5.2 are at least partly optimization toward the evaluation metric rather than independent evidence of generalized researcher-AI alignment. The paper's own caveat in Section 6.3 concedes this, making the reported improvement partially circular.

full rationale

The paper's central Study 1 result is not circular: the learning-curve analysis uses a held-out evaluation partition, and the extrapolation analysis in Section 5.1.2 intentionally tests codes drawn from clusters absent from the example set, so those are genuine predictions. IoU and MHD are standard external measures with stated definitions, not constructed to entail the qualitative conclusion. The self-citations to prior work by the same authors are used for provenance and framing, not as load-bearing derivations. The main partial circularity is confined to Study 2's alignment-improvement claim: participants are instructed to improve IoU/MHD, the interface lets them select examples and prompts by those same metrics, and no separate test set is used, so the reported gains partly reflect optimization on the evaluation metric. Because the abstract's stronger negative claim about the limited 'deeper interpretive lens' rests primarily on Study 1's held-out analyses and the paper explicitly acknowledges the Study 2 limitation, the overall circularity score is moderate rather than high.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of IoU and MHD as alignment measures, on treating human coding as the reference standard, and on embedding similarity as a proxy for semantic relatedness. No new physical or theoretical entities are introduced; LLMCode is a software tool, not an invented entity.

free parameters (3)
  • Number of K-Means clusters = 5
    Section 5.1.2 clusters each participant's final codes into five groups to build example and test sets; changing this number changes which codes are considered unfamiliar and can alter the correlation in Figure 5.
  • Inclusion thresholds for Study 1 analyses = ncoded >= 38, nuncoded >= 38, ncodes >= 6
    Section 5.1.1 restricts the learning-curve analysis to 11 of 19 participants; these thresholds are not justified and reduce the sample further.
  • Random baseline repetitions = 5
    Section 5.2 averages the random baseline over five generated example sets; more repetitions would change variance estimates and could alter the comparison in Figure 6.
assumptions (4)
  • domain assumption IoU and MHD are valid proxies for qualitative coding alignment.
    Section 3.1 assumes character-level overlap and embedding cosine distances reflect how closely the model matches a designer's interpretive lens; no validation against an external measure of interpretive quality is provided.
  • domain assumption Human annotations are the reference baseline for alignment.
    Sections 2.2 and 3 treat the designer's coding as ground truth, consistent with the reflexive RfD paradigm, but this is a value choice rather than an objective standard.
  • domain assumption Embedding distances capture semantic relatedness of codes.
    MHD relies on text-embedding-3-large; Section 3.1.2 acknowledges model-specific bias but still uses the resulting distances as a core measure.
  • domain assumption In-context learning can transfer an individual's coding style from few-shot examples.
    Section 3 and Appendix A assume that a small set of human-coded examples teaches the model to reproduce a researcher's coding behavior; this is the standard paradigm the paper sets out to test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis." pith.science (2026). https://pith.science/paper/CUYUSCDZ

@misc{pith2026250416671,
  author       = {Pith},
  title        = {Pith review of: LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUYUSCDZ}},
  note         = {Machine review of arXiv:2504.16671}
}
read the original abstract

The use of large language models (LLMs) in qualitative analysis offers enhanced efficiency but raises questions about their alignment with the contextual nature of research for design (RfD). This research examines the trustworthiness of LLM-driven design insights, using qualitative coding as a case study to explore the interpretive processes central to RfD. We introduce LLMCode, an open-source tool integrating two metrics, namely Intersection over Union (IoU) and Modified Hausdorff Distance, to assess the alignment between human and LLM-generated insights. Across two studies involving 26 designers, we find that while the model performs well with deductive coding, its ability to emulate a designer's deeper interpretive lens over the data is limited, emphasising the importance of human-AI collaboration. Our results highlight a reciprocal dynamic where users refine LLM outputs and adapt their own perspectives based on the model's suggestions. These findings underscore the importance of fostering appropriate reliance on LLMs by designing tools that preserve interpretive depth while facilitating intuitive collaboration between designers and AI.

Figures

Figures reproduced from arXiv: 2504.16671 by the authors.

Figure 1
Figure 1. LLMCode interface for researcher-aligned LLM-assisted qualitative coding. The interface enables users to compare the model’s [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Mean IoU plotted against the number of few-shot examples included in the coding prompt (Study 1, N=11). The metric [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Mean MHD plotted against the number of few-shot examples included in the coding prompt (Study 1, N=11). The metric [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: New codes as a fraction of all annotated codes in the given time frame (Study 1, N=15). The rate at which new codes are [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Scatter plot showing the correlation (Pearson coefficient = 0.53) between the similarity of human-annotated codes to the [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The development of (a) IoU and (b) Modified Hausdorff Distance as participants in Study 2 manually iterated on their few-shot [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 21 canonical work pages

  1. [1]

    Open AI. 2022. Introducing ChatGPT. https://openai.com/index/chatgpt/. (30 Nov 2022). Accessed: 2025-01-08

    work page 2022

  2. [2]

    Julian Ashwin, Aditya Chhabra, and Vijayendra Rao. 2023. Using Large Language Models for Qualitative Analysis can Introduce Serious Bias. (2023). https://doi.org/10.48550/ARXIV.2309.17147

    arXiv doi:10.48550/arxiv.2309.17147 2023

  3. [3]

    Jeffrey Bardzell and Shaowen Bardzell. 2016. Humanistic HCI. interactions 23, 2 (Feb. 2016), 20–29. https://doi.org/10.1145/2888576

    doi:10.1145/2888576 2016

  4. [5]

    Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589–597. https://doi.org/10.1080/2159676X.2019.1628806

    arXiv 2019

  5. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  6. [7]

    Nigel Cross. 2011. Design Thinking: Understanding How Designers Think and Work . Berg, Oxford. https://doi.org/10.5040/9781474293884

    doi:10.5040/9781474293884 2011

  7. [8]

    Shih-Chieh Dai, Aiping Xiong, and Lun-Wei Ku. 2023. LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9993–10001. https://doi.org/10.18653/v1/2023.findings-...

    doi:10.18653/v1/2023.findings-emnlp.669 2023

  8. [9]

    Tom Dietterich. 1995. Overfitting and undercomputing in machine learning. ACM computing surveys (CSUR) 27, 3 (1995), 326–327

    work page 1995

Show all 46 references
  1. [10]

    Dovetail. [n. d.]. Customer Insights Hub. https://dovetail.com/. Accessed: 2024-12-31

    2024

  2. [11]

    M-P Dubuisson and Anil K Jain. 1994. A modified Hausdorff distance for object matching. In Proceedings of 12th international conference on pattern recognition, Vol. 1. IEEE, 566–568

    1994

  3. [12]

    Zackary Okun Dunivin. 2024. Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks. (2024). https://doi.org/10.48550/ARXIV.2401.15170

    2024 doi arXiv

  4. [13]

    Anthony Dunne and Fiona Raby. 2013. Speculative Everything: Design, Fiction, and Social Dreaming . The MIT Press. Manuscript submitted to ACM LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis 21

    2013

  5. [14]

    Feuston and Jed R

    Jessica L. Feuston and Jed R. Brubaker. 2021. Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 469 (2021), 25 pages. https://doi.org/10.1145/3479856

    2021 doi

  6. [15]

    Glassman, and Toby Jia-Jun Li

    Simret Araya Gebreegziabher, Zheng Zhang, Xiaohang Tang, Yihao Meng, Elena L. Glassman, and Toby Jia-Jun Li. 2023. PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Comput...

    2023

  7. [16]

    Carole Gray and Julian Malins. 2004. Visualizing Research: A Guide to the Research Process in Art and Design . Routledge, London

    2004

  8. [17]

    Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating Large Language Models in Generating Synthetic HCI Research Data: A Case Study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Com...

    2023

  9. [18]

    Leah Hamilton, Desha Elliott, Aaron Quick, Simone Smith, and Victoria Choplin. 2023. Exploring the Use of AI in Qualitative Analysis: A Comparative Study of Guaranteed Income Data. International Journal of Qualitative Methods 22 (Oct. 2023), 16094069231201504. https://doi.org/...

    2023

  10. [19]

    Marsh, Jessica L

    Matt-Heun Hong, Lauren A. Marsh, Jessica L. Feuston, Janet Ruppert, Jed R. Brubaker, and Danielle Albers Szafir. 2022. Scholastic: Graphical Human-AI Collaboration for Inductive and Interpretive Text Analysis. InProceedings of the 35th Annual ACM Symposium on User Interface So...

    2022

  11. [20]

    Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) ...

    2021

  12. [21]

    Brubaker

    Jialun Aaron Jiang, Kandrea Wade, Casey Fiesler, and Jed R. Brubaker. 2021. Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 94 (2021), 23 pages. https://doi.org/10.1145/3449168

    2021 doi

  13. [22]

    Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter development team. 2016. Jupyter Noteboo...

    2016

  14. [23]

    Klaus Krippendorff. 2005. The Semantic Turn: A New Foundation for Design (1. ed.). CRC Press, Boca Raton, FL, USA. https://doi.org/10.4324/ 9780203299951

    2005

  15. [24]

    Lee and Katrina A

    John D. Lee and Katrina A. See. 2004. Trust in Automation: Designing for Appropriate Reliance. Human Factors 46, 1 (2004), 50–80. https: //doi.org/10.1518/hfes.46.1.50_30392

    2004 doi

  16. [25]

    Saríah Lopez-Fierro and Ha Nguyen. 2024. Making Human-AI Contributions Transparent in Qualitative Coding. In Proceedings of the 17th International Conference on Computer-Supported Collaborative Learning-CSCL 2024, pp. 3-10 . International Society of the Learning Sciences

    2024

  17. [26]

    Andrés Lucero. 2012. Framing, aligning, paradoxing, abstracting, and directing: how design mood boards work. In Proceedings of the Designing Interactive Systems Conference (Newcastle Upon Tyne, United Kingdom) (DIS ’12). Association for Computing Machinery, New York, NY, USA, ...

    2012

  18. [27]

    Andrés Lucero. 2015. Using Affinity Diagrams to Evaluate Interactive Prototypes. In Human-Computer Interaction – INTERACT 2015 , Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing, Cham, 231–248

    2015

  19. [28]

    Megh Marathe and Kentaro Toyama. 2018. Semi-Automated Coding for Qualitative Research: A User-Centered Inquiry and Initial Prototypes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Mac...

    2018

  20. [29]

    Mayer, James H

    Roger C. Mayer, James H. Davis, and F. David Schoorman. 1995. An Integrative Model of Organizational Trust. The Academy of Management Review 20, 3 (1995), 709–734. https://doi.org/10.2307/258792

    1995 doi

  21. [30]

    Pietro Micheli, Sarah J. S. Wilner, Sabeen Hussain Bhatti, Matteo Mura, and Michael B. Beverland. 2019. Doing Design Thinking: Conceptual Review, Synthesis, and Research Agenda. Journal of Product Innovation Management 36, 2 (2019), 124–148. https://doi.org/10.1111/jpim.12466

    2019 doi

  22. [31]

    Michael Huberman, and Johnny Saldaña

    Matthew Miles, A. Michael Huberman, and Johnny Saldaña. 2014. Qualitative Data Analysis: A Methods Sourcebook (3. ed.). SAGE Publications, Inc

    2014

  23. [32]

    Gjoko Muratovski, Ken Friedman, Don Norman, and Steven Heller. 2022. Research for Designers: A Guide to Methods and Practice (2. ed.). SAGE Publications, London

    2022

  24. [33]

    Joel Oksanen. 2024. Bridging the Integrity Gap: Towards AI-assisted Design Research. Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (May 2024), 1–5. https://doi.org/10.1145/3613905.3647962

    2024

  25. [34]

    OpenAI. 2024. New embedding models and API updates. https://openai.com/index/new-embedding-models-and-api-updates/. Accessed: 2025-01-10

    2024

  26. [35]

    OpenAI. 2024. What are tokens and how to count them? https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them. Accessed: 2025-01-10

    2024

  27. [36]

    OpenAI. n.d.. Models - OpenAI API. https://platform.openai.com/docs/models/. Accessed: 2025-01-08

    2025

  28. [37]

    Raimundo Real and Juan M Vargas. 1996. The probabilistic basis of Jaccard’s index of similarity. Systematic biology 45, 3 (1996), 380–385

    1996

  29. [38]

    Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 658–...

    2019

  30. [39]

    Sanders and Pieter Jan Stappers

    Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2008. Co-creation and the new landscapes of design. CoDesign 4, 1 (March 2008), 5–18. https: //doi.org/10.1080/15710880701875068

    2008 doi

  31. [40]

    Donald A. Schön. 1983. The Reflective Practitioner: How Professionals Think in Action . Basic Books, New York, NY

    1983

  32. [41]

    Ravi Sinha, Idris Solola, Ha Nguyen, Hillary Swanson, and LuEttaMae Lawrence. 2024. The Role of Generative AI in Qualitative Research: GPT-4’s Contributions to a Grounded Theory Analysis (LDT ’24). Association for Computing Machinery, New York, NY, USA, 17–25. https: //doi.org...

    2024

  33. [42]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  34. [43]

    Sukriti Verma and Vagisha Nidhi. 2017. Extractive summarization using deep learning. arXiv preprint arXiv:1708.04439 (2017)

    2017 arXiv

  35. [44]

    Leonardo Silva Vianna, Alexandre Leopoldo Gonçalves, and João Artur Souza. 2024. Analysis of learning curves in predictive modeling using exponential curve fitting with an asymptotic approach. PLOS ONE 19, 4 (04 2024), 1–23. https://doi.org/10.1371/journal.pone.0299811

    2024 doi

  36. [45]

    Katja Tschimmel Violeta Clemente and Fátima Pombo. 2017. A Future Scenario for a Methodological Approach applied to PhD Design Research. Development of an Analytical Canvas. The Design Journal 20, sup1 (2017), S792–S802. https://doi.org/10.1080/14606925.2017.1353025

    2017

  37. [46]

    Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer

    Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer. 2023. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In Companion Proceedings of the 28th International Conference on Intelligent ...

    2023

  38. [47]

    {code}: {description}

    Fengxiang Zhao, Fan Yu, and Yi Shang. 2024. A New Method Supporting Qualitative Data Analysis Through Prompt Generation for Inductive Coding. In 2024 IEEE International Conference on Information Reuse and Integration for Data Science (IRI) . 164–169. https://doi.org/10.1109/IR...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.