pith. sign in

arxiv: 2604.25312 · v2 · submitted 2026-04-28 · 💻 cs.HC

Author response to commentaries on H is for Human and How (Not) to Evaluate Qualitative Research in HCI

Andy Crabtree This is my paper

Pith reviewed 2026-05-07 15:36 UTC · model grok-4.3

classification 💻 cs.HC
keywords qualitative researchHCI evaluationmethodological debatehuman-computer interactionresearch standardscommentary response
0
0 comments X

The pith

The author addresses commentaries to clarify and uphold the original case for evaluating qualitative HCI research on its own human-centered terms.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is the author's direct reply to seven commentaries on an earlier article titled H is for Human and How (Not) to Evaluate Qualitative Research in HCI. The author takes up the specific concerns raised by each commentator in turn. A sympathetic reader would care because the choice of evaluation standards shapes what counts as good qualitative work and influences what kinds of studies get published and funded in HCI. If the response is effective it keeps the focus on methods that fit the character of qualitative inquiry instead of borrowing mismatched criteria from other traditions.

Core claim

The author maintains that the commentaries do not overturn the central position that qualitative research in HCI must be judged by its success in revealing human experience, context, and practice rather than by quantitative-style criteria of generalizability or statistical rigor, and supplies point-by-point clarifications to move the discussion forward.

What carries the argument

Point-by-point engagement with each of the seven commentaries, using the original framing that evaluation criteria should be attuned to the human character of the research.

If this is right

  • Readers gain explicit guidance on applying the proposed evaluation approach in their own reviewing and writing.
  • The exchange makes visible the specific points of tension between qualitative and other methodological traditions in HCI.
  • Future papers can cite the response when defending their choice of evaluation standards against similar objections.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The public commentary-and-response format may encourage similar exchanges on other methodological questions in the field.
  • If the clarified position takes hold, journal and conference guidelines could shift to include human-experience criteria alongside traditional ones.
  • Empirical testing could involve asking reviewers to apply the defended criteria to sample papers and measure consistency.

Load-bearing premise

That addressing these particular commentaries is enough to settle or advance the broader disagreement over how qualitative work should be judged.

What would settle it

A later survey of HCI reviewers or editors showing that the clarified criteria have not altered how qualitative papers are accepted or rejected would indicate the response did not achieve its aim.

read the original abstract

This is the authors response to commentaries on the original article H is for Human and How (Not) to Evaluate Qualitative Research in HCI, https://doi.org/10.1080/07370024.2025.2475743 Commentaries were provided by: Jeffrey Bardzell, https://doi.org/10.1080/07370024.2025.2612474 Alan Blackwell, https://doi.org/10.1080/07370024.2025.2591878 Paul Dourish, https://doi.org/10.1080/07370024.2025.2594529 Bonnie Nardi, https://doi.org/10.1080/07370024.2025.2596752 Peter Pirolli, https://doi.org/10.1080/07370024.2025.2596745 Jennifer Rode, https://doi.org/10.1080/07370024.2025.2598800 Peter Tolmie, https://doi.org/10.1080/07370024.2025.2591872 Please feel free to copy, redistribute, adapt, and build on any part of this article in accordance with the CC BY 4.0 license: https://creativecommons.org/licenses/by/4.0/

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 1 minor

Summary. This manuscript is the authors' response to peer commentaries on their original article 'H is for Human and How (Not) to Evaluate Qualitative Research in HCI'. It directly engages with points raised by Jeffrey Bardzell, Alan Blackwell, Paul Dourish, Bonnie Nardi, Peter Pirolli, Jennifer Rode, and Peter Tolmie, offering clarifications, defenses, and expansions on the evaluation of qualitative research in HCI.

Significance. As a response piece, the manuscript contributes to methodological discourse in HCI by enabling authors to address expert feedback on qualitative evaluation standards. The explicit engagement with each commentary is a strength, supporting transparent scholarly exchange; no new empirical claims, derivations, or data are introduced, so significance rests on interpretive clarification rather than novel findings.

minor comments (1)
  1. Abstract: The list of commentators and DOIs is clear and helpful for traceability, but adding one sentence on the primary themes addressed in the responses would improve immediate reader orientation without altering length substantially.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive evaluation of our response to the peer commentaries and for recommending acceptance. The manuscript provides clarifications and expansions on the evaluation of qualitative research in HCI through direct engagement with the commentators.

Circularity Check

0 steps flagged

No significant circularity

full rationale

This is an author response to external peer commentaries on a prior article. The text contains no derivations, equations, fitted parameters, models, or new empirical claims. It engages directly with listed external commentaries without any load-bearing step that reduces a prediction or result to its own inputs by construction, self-citation chains, or ansatz smuggling. The document is self-contained as clarification and engagement rather than a derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a discussion response rather than a research derivation. No free parameters, axioms, or invented entities are introduced or relied upon.

pith-pipeline@v0.9.0 · 5535 in / 908 out tokens · 26428 ms · 2026-05-07T15:36:45.252701+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [2]

    self-evident

    to a leading HCI journal which was rejected on the basis that the validity of its “self-evident” findings was “compromised” by a small set of interviews from a small sample; the trigger, as reported, for H is for Human. I was tired of making the same old arguments to reviewers and so here we are having a much grander debate, a “necessary conversation abou...

    work page 2026

  2. [3]

    sociological enterprise

    but its use in sociology to understand people in our own societies and culture outstrips anthropology by several orders of magnitude, if for no other very practical reason than few people are actually prepared to dedicate years of their life living with the natives in faraway places, in contrast to spending considerably shorter and more comfortable period...

    work page 1992

  3. [4]

    I take Sudnow and Lynch’s methodological reflections seriously: however you find it will do. I would argue that we need to stop regulating how to find the world and understand other people’s social realities, stop trying to impose inappropriate methodological procedures based on mythologies of science: e.g., that there is a scientific method, that science...

    work page 1993

  4. [5]

    offers no guarantees

    – they would not be relevant to Human Science as I explained in H is for Human. Reflexivity “offers no guarantees” either as Lynch (2000) makes clear. What chance would a paper like Faltering from Ethnography to Design have in today’s methodological minefield? We should dispense with methodological stipulations. Stop making people jump through a random ar...

    work page 2000

  5. [6]

    qualitative and interpretive methods are essential

    Blackwell goes on to argue that “qualitative and interpretive methods are essential” to HCI; not only “methods of practice-led and artistic inquiry”, but also “the rigorous social sciences of anthropology, sociology, education, public policy, economics and others.” The Human Sciences, in other words. The Human Sciences are not all of apiece. They do not a...

    work page 2025

  6. [7]

    H is for human and how (not) to evaluate qualitative research in HCI

    A commentary on Crabtree’s “H is for human and how (not) to evaluate qualitative research in HCI”. Human–Computer Interaction, 1-3. https://doi.org/10.1080/07370024.2025.2612474 Bittner, E

    work page doi:10.1080/07370024.2025.2612474 2025

  7. [8]

    Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘15), Seoul, Republic of Korea, pp

    HCI as an inter-discipline. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘15), Seoul, Republic of Korea, pp. 503–516. ACM Press. https://doi.org/10.1145/2702613.2732505 Blackwell, A.F

    work page doi:10.1145/2702613.2732505

  8. [9]

    H is for human and how (not) to evaluate qualitative research in HCI

    Commentary: There is also a C in HCI. Human–Computer Interaction, 1-3. https://doi.org/10.1080/07370024.2025.2591878 Button, G., Crabtree, A., Rouncefield, M. and Tolmie, P

    work page doi:10.1080/07370024.2025.2591878 2025

  9. [10]

    Being really confidently wrong

    Deconstructing Ethnography: Towards a Social Methodology for the Design of Ubiquitous and Interactive Computing Systems. Springer. Clarke, V., Braun, V., Adams, J., Callaghan, J. E. M., LaMarre, A., & Semlyen, J. (2025). “Being really confidently wrong” - qualitative researchers’ experiences of methodologically incongruent peer review feedback. Qualitativ...

    work page doi:10.1037/qup0000322 2025

  10. [11]

    How many bloody examples do you want?

    “How many bloody examples do you want?” – fieldwork and generalisation. Proceedings of the 13th European Conference on Computer Supported Cooperative Work (ECSCW ‘13), Paphos, Cyprus , pp. 1-20. Springer. https://doi.org/10.1007/978-1-4471-5346-7_1 Crabtree, A., McGarry, G. and Urquhart, L

    work page doi:10.1007/978-1-4471-5346-7_1

  11. [12]

    AI & Society, vol

    AI and the iterable epistopics of risk. AI & Society, vol. 40, pp. 1425–1438.https://doi.org/10.1007/s00146-024-02021-y Crabtree, A

    work page doi:10.1007/s00146-024-02021-y

  12. [13]

    H is for human and how (not) to evaluate qualitative research in HCI

    H is for human and how (not) to evaluate qualitative research in HCI. Human-Computer Interaction, pp. 1-24, https://doi.org/10.1080/07370024.2025.2475743 7 Deegan, M.J

    work page doi:10.1080/07370024.2025.2475743 2025

  13. [14]

    H is for human and how (not) to evaluate qualitative research in HCI

    The Chicago School of ethnography. Handbook of Ethnography (ed. Atkinson, P., Coffey, A., Delamont, S., Lofland, J. and Lofland, L.), pp. 11-25. Sage. Dourish, P. 2026 Commentary: philosophy of science, seen from the inside, Human–Computer Interaction, 1-3. https://doi.org/10.1080/07370024.2025.2594529 Feyerabend, P.1975. Against Method. New Left Books. G...

    work page doi:10.1080/07370024.2025.2594529 2026

  14. [15]

    Social Psychology Quarterly, vol

    Ethnomethodology’s program. Social Psychology Quarterly, vol. 59 (1), pp. 5-31. https://doi.org/10.2307/2787116 Garfinkel, H

    work page doi:10.2307/2787116

  15. [16]

    Retrieved 18 February, 2026, from https://martynhammersley.wordpress.com/wp-content/uploads/2017/12/ethnography-what-is-it-and-why-do-it.pdf Hughes, J

    Ethnography: What Is It, and Why Do It? Keynote, Qual-World Interactive Virtual Conference 2017, International Institute for Qualitative Methodology, University of Alberta. Retrieved 18 February, 2026, from https://martynhammersley.wordpress.com/wp-content/uploads/2017/12/ethnography-what-is-it-and-why-do-it.pdf Hughes, J. A., Randal, D. and Shapiro, D

    work page 2017

  16. [17]

    Proceedings of the 1992 ACM Conference on Computer Supported Cooperative Work (CSCW ‘92), Toronto, Canada, pp

    Faltering from ethnography to design. Proceedings of the 1992 ACM Conference on Computer Supported Cooperative Work (CSCW ‘92), Toronto, Canada, pp. 115–122. ACM Press. https://doi.org/10.1145/143457.143469 Husserl, E

    work page doi:10.1145/143457.143469 1992

  17. [18]

    Human Studies, vol

    Silence in context: ethnomethodology and social theory. Human Studies, vol. 22, pp. 211-233. https://doi.org/10.1023/A:1005440501638 Lynch, M

    work page doi:10.1023/a:1005440501638

  18. [19]

    Theory, Culture & Society, vol

    Against reflexivity as an academic virtue and source of privileged knowledge. Theory, Culture & Society, vol. 17 (3), pp. 26–54. https://doi.org/10.1177/02632760022051202 Nardi, B

    work page doi:10.1177/02632760022051202

  19. [20]

    H is for human and how (not) to evaluate qualitative research in HCI

    Q is for qualitative and how (not) to draw too small a boundary around qualitative research in HCI. Human–Computer Interaction, 1-4. https://doi.org/10.1080/07370024.2025.2596752 Newstead, A

    work page doi:10.1080/07370024.2025.2596752 2025

  20. [21]

    Saying, showing, and knowing: epistemological dimensions of Wittgenstein’s response to scepticism in On Certainty. Retrieved 13 February, 2026, from https://www.academia.edu/59098913/Saying_Showing_and_Knowing_Epistemological_Dimensions_of_Wittgensteins_Response_to_Scepticism_in_On_Certainty Pirolli, P

    work page arXiv 2026

  21. [22]

    H is for human and how (not) to evaluate qualitative research in HCI

    Qualitative research as science: commentary on A. Crabtree (2025) “H is for human and how (not) to evaluate qualitative research in HCI”. Human–Computer Interaction, pp.1-5. https://doi.org/10.1080/07370024.2025.2596745 Platt, J

    work page doi:10.1080/07370024.2025.2596745 2025

  22. [23]

    British Journal of Sociology, vol

    Weber’s verstehen and the history of qualitative research: the missing link. British Journal of Sociology, vol. 36 (3), pp. 448–466. https://doi.org/10.2307/590460 Pollner, M

    work page doi:10.2307/590460

  23. [24]

    H is for human and how (not) to evaluate qualitative research in HCI

    Commentary: emphasizing the role of positionality, theory, and ethnographic writing conventions in reflexive texts. Human–Computer Interaction, 1-4. https://doi.org/10.1080/07370024.2025.2598800 Soden, R., Toombs, A. and Thomas, M

    work page doi:10.1080/07370024.2025.2598800 2025

  24. [25]

    Interactions, vol

    Evaluating interpretive research in HCI. Interactions, vol. 31 (1), pp. 38–42. https://doi.org/10.1145/3633200 Spencer, L., Ritchie, J., Lewis, J. and Dillon, L

    work page doi:10.1145/3633200

  25. [26]

    Cabinet Office

    Quality in qualitative evaluation: a framework for assessing research evidence. Cabinet Office. Retrieved 18 February, 2026, from https://www.gov.uk/government/publications/government-social-research-framework-for-assessing-research-evidence Tolmie, P

    work page 2026

  26. [27]

    H is for human and how (not) to evaluate qualitative research in HCI

    Commentary: The common-sense grounds of positivist reviewing. Human–Computer Interaction, 1-3. https://doi.org/10.1080/07370024.2025.2591872 Wittgenstein, L

    work page doi:10.1080/07370024.2025.2591872 2025