Qualitative Research Methods in Software Engineering: Past, Present, and Future

Carolyn Seaman; Rashina Hoda; Robert Feldt

Qualitative Research Methods in Software Engineering: Past, Present, and Future

T0 review · 0 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 1999 methods paper seeded software engineering's qualitative turn, this retrospective argues.

desk verdict An invited retrospective that is honestly framed and useful as a piece of disciplinary history; judge it as a reflection piece, not a research claim. read the letter →

arxiv 2502.07220 v1 pith:VET6AXTK submitted 2025-02-11 cs.SE

Carolyn Seaman , Rashina Hoda , Robert Feldt This is my paper · ORCID

classification cs.SE

keywords qualitativeresearchsoftwareengineeringmethodsempiricalgroundedtheorycasestudyethnographymixed

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This retrospective argues that qualitative methods have become a necessary and productive part of software engineering research, enabling lines of inquiry into human factors, agile practices, and distributed teams that quantitative methods alone could not support. The 1999 journal article that started this thread served not as an immediate trigger but as a seed: adoption was slow for a decade, then accelerated through new communities and venues that welcomed human-focused studies. The authors ground their account in reflective interviews among themselves, which they did not transcribe or code, so the history is presented as shared recollection rather than systematic analysis. The paper closes with a forward-looking agenda: more methodological variety, tailored guidelines, reflexivity, ethics, and careful use of large language models.

What carries the argument

The carrying mechanism is the retrospective itself: a set of reflective interviews among the three coauthors, used to reconstruct the history of qualitative methods, combined with a structured table of method guide contributions and a twelve-item set of recommendations for the future. The analysis is thematic but informal, since the interviews were not transcribed or coded.

What would settle it

A systematic content analysis of empirical software engineering papers from 1999 through 2025 that finds no measurable increase in qualitative methods after the mid-2000s, or attributes the growth entirely to external factors such as the agile movement, would undercut the claimed causal role of the 1999 paper and the community efforts described.

Watch

Extended reading notes

Core claim

The central claim is stated plainly: the greater and better use of qualitative methods has enabled many lines of knowledge creation that would not otherwise have been possible. The paper traces this from the 1999 methods article through the founding and growth of communities devoted to human aspects of software engineering, the rise of agile studies, and the research on distributed development that became newly relevant during the pandemic. It documents how software-engineering-specific method guides, from case study research to ethnography to socio-technical grounded theory, emerged as the field matured, and it argues that future progress depends on continued methodological diversity and innovation.

Load-bearing premise

The paper's narrative and predictions rest on the authors' recollections of their own untranscribed, uncoded interviews, so if those memories are selective or biased the historical account loses its evidence base.

Editorial extensions

If this is right

The software industry's relatively smooth pivot to remote work during the pandemic owed in part to earlier qualitative studies of distributed teams.
The growth of agile research and of a community focused on cooperative and human aspects created a self-reinforcing cycle between studying human factors and using qualitative methods.
Researchers now have a body of software-engineering-specific method guides for case studies, ethnography, and grounded theory, so qualitative work no longer depends on ad hoc adaptations.
Future advances depend on explicit attention to reflexivity, emergence, ethics, longitudinal designs, and responsible use of large language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The retrospective's own method—untranscribed, uncoded interviews among the authors—illustrates that even senior qualitative researchers sometimes trade systematic analysis for narrative, so the field's quality bar is still uneven.
A bibliometric study of leading software engineering venues could test whether the claimed adoption timeline matches actual citation and method-use patterns.
The paper's 'methodology camps' diagnosis implies that editorial guidance and reviewer training may matter more than writing additional method guides.
LLM-assisted qualitative analysis could itself become a research subject: empirically comparing themes extracted with and without LLM help would calibrate how much trust to place in such tools.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is not a research paper, and it would be wrong to referee it as one. It is an invited retrospective on Seaman's 1999 TSE paper, written by Seaman with two colleagues, and it reads that way: part memoir, part disciplinary history, part wish list. Judged on those terms, it succeeds. The historical narrative is coherent and well supported by citations, the account of slow adoption followed by a CHASE/agile-era inflection point matches what I know of the field, and Table 2's twelve recommendations are a reasonable consolidation of ongoing conversations. The paper's real value is as a time capsule and a discussion piece for the SE methods community, not as a source of new evidence.

What is new is almost nothing empirically—no data, no systematic review, no framework—but that is not a flaw given the genre. The authors are transparent about their method: they interviewed each other, did not transcribe or code the interviews, and explicitly say they are sharing insights rather than offering a systematic analysis. That disclosure is honest and appropriate. The central claim in Section 4—that qualitative methods enabled lines of knowledge creation that would not otherwise have been possible—is an interpretive historical claim, and the paper supports it with venue histories, award examples, and citations to published work. It holds up.

Soft spots: the self-referential evidence base is real but not damaging, because the narrative does not rest on the interviews; it rests on published literature the authors participated in. The pandemic-preparedness aside in Section 4 is speculative, but the authors hedge it with "It can be argued," so I would call it a minor overreach, not an error. The citation pattern is heavy on the authors' own work, which is normal in a retrospective and does not strike me as inappropriate here. There is also a fair amount of editorializing about reviewer resistance and methodology camps, but that is clearly opinion and labeled as such.

Who is this for? Anyone interested in how qualitative methods became mainstream in SE, and early-career researchers who want a compact overview of the key method guides (Table 1). It deserves a serious referee if only because it will be widely read as an official TSE retrospective; the referee should check the historical claims and the recommendations for internal consistency. I would be comfortable accepting it after minor revisions. Bring it to a reading group only if you want a debate about qualitative methods and the role of retrospectives; otherwise it is a quick, pleasant read.

Referee Report

0 major / 6 minor

Summary. The paper is a retrospective on Carolyn Seaman's 1999 TSE article "Qualitative Methods in Empirical Studies of Software Engineering," co-authored by Seaman, Rashina Hoda, and Robert Feldt. It recounts the history of qualitative methods in software engineering from 1999 to 2025, argues in Section 4 that these methods enabled knowledge creation that would not otherwise have been possible, and offers a consolidated set of recommendations for the future in Table 2. The authors disclose in Section 1 that they conducted interviews among themselves that were not transcribed or coded, and that the retrospective draws on those discussions plus selected literature.

Significance. If accepted as an interpretive historical account, the paper is a valuable contribution to the field's understanding of its methodological evolution. It documents key milestones such as the growth of CHASE, the role of agile and distributed-development research, and Storey's 2019 ICSE keynote, while also identifying the importance of methodological allies in editorial and review positions. The paper is transparent about its non-systematic interview process, and its central claim is supported by directional evidence from the cited literature rather than by quantitative analysis. As an invited retrospective, it appropriately fills a different niche from a primary research article.

minor comments (6)

[Section 1] The phrase "using (of course) qualitative methods" reads as a joke that may confuse readers about the methodological status of the interviews; please clarify the intended meaning or rephrase it for directness.
[Abstract] In the abstract, "the impact it's had" should be "the impact its had" or more conventionally "the impact it has had"; the current contraction is grammatically incorrect.
[Table 2] Recommendation 12 contains a missing space in "Community:Create"; it should read "Community: Create".
[Section 3] The assertion that qualitative use "did not increase significantly until about 10 years later" could be strengthened with a bibliometric count or citation analysis, or explicitly framed as personal recollection from the authors' experience.
[Section 4] The pandemic-preparedness statement "It can be argued that the software industry was better prepared..." is speculative and would benefit from a supporting citation or a softer formulation, since no evidence is cited that links earlier qualitative distributed-work research to industry readiness during COVID-19.
[Section 4] When listing highlighted resources, the paper says it will not be exhaustive but does not explain the selection criteria; adding one sentence on why these particular works were chosen would help readers contextualize the list.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the retrospective's interpretive claims are supported by external citations and disclosed history, not by a derivation that reduces to its inputs.

full rationale

This is an invited retrospective and methodological commentary, not a formal derivation or empirical prediction. The central claim in Section 4 that greater use of qualitative methods enabled new knowledge creation is an interpretive historical statement supported by documented venue developments (CHASE, XP, distributed development), third-party citations (Easterbrook et al., Runeson et al., Sharp et al., Lenberg et al.), and the authors' own participation, which is appropriate for a retrospective. The self-conducted interviews are explicitly disclosed as not transcribed or coded and are used only 'to share some of the insights' (Section 1); they are not presented as the evidential basis for the paper's claims. No equation, fitted parameter, or uniqueness theorem is invoked, so no claim reduces by construction to its inputs. Self-citations to Hoda (2022; 2024) and Lenberg et al. (2024) function as examples in a literature review and as references for future-direction statements, not as load-bearing premises that force the paper's conclusions. At most, the paper is self-referential in genre, which is not circularity; accordingly the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's claims rest on the authors' unsystematic reflections and a selected set of references rather than on formal derivations or systematic data. No free parameters or invented entities apply; the central support comes from domain assumptions about qualitative methodology and representative memory.

assumptions (3)

ad hoc to paper Unsystematic self-interviews among coauthors yield valid and representative insights.
Section 1 states the interviews were not transcribed or coded and no systematic method was applied, yet the paper presents those insights as the basis of the retrospective.
domain assumption The selected cited works are representative of the evolution of qualitative methods in software engineering.
Table 1 lists selected guides without specifying inclusion criteria, and the narrative depends on this selection being fair and comprehensive.
domain assumption Qualitative methods can produce valid scientific knowledge when properly applied.
The entire advocacy for qualitative methods presupposes this philosophical stance, which is argued for elsewhere in the literature but not established in this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Qualitative Research Methods in Software Engineering: Past, Present, and Future." pith.science (2026). https://pith.science/paper/VET6AXTK

@misc{pith2026250207220,
  author       = {Pith},
  title        = {Pith review of: Qualitative Research Methods in Software Engineering: Past, Present, and Future},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VET6AXTK}},
  note         = {Machine review of arXiv:2502.07220}
}

read the original abstract

The paper entitled "Qualitative Methods in Empirical Studies of Software Engineering" by Carolyn Seaman was published in TSE in 1999. It has been chosen as one of the most influential papers from the third decade of TSE's 50 years history. In this retrospective, the authors discuss the evolution of the use of qualitative methods in software engineering research, the impact it's had on research and practice, and reflections on what is coming and deserves attention.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 24 canonical work pages

[1]

Large language models for qualitative research in software engineering: exploring opportunities and challenges,

M. Bano, R. Hoda, D. Zowghi, and C. Treude, “Large language models for qualitative research in software engineering: exploring opportunities and challenges,” Automated Software Engineering , vol. 31, no. 1, p. 8, 2024

work page 2024
[2]

Editorial for the special issue on qualitative software engineering research,

Y. Dittrich, M. John, J. Singer, and B. Tessem, “Editorial for the special issue on qualitative software engineering research,” Infor- mation and software technology, vol. 49, no. 6, pp. 531–539, 2007

work page 2007
[3]

Qualitative research in software engineering,

T. Dyb ˚a, R. Prikladnicki, K. R ¨onkk¨o, C. Seaman, and J. Sillito, “Qualitative research in software engineering,” Empirical Software Engineering, vol. 16, pp. 425–429, 2011

work page 2011
[4]

Selecting empirical methods for software engineering research,

S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian, “Selecting empirical methods for software engineering research,” Guide to advanced empirical software engineering, pp. 285–311, 2008

work page 2008
[5]

Building theories from case study research,

K. M. Eisenhardt, “Building theories from case study research,” Academy of management review, vol. 14, no. 4, pp. 532–550, 1989

work page 1989
[6]

Socio-technical grounded theory for software engineer- ing,

R. Hoda, “Socio-technical grounded theory for software engineer- ing,” IEEE Transactions on Software Engineering, vol. 48, no. 10, pp. 3808–3832, 2022

work page 2022
[7]

Springer, 2024

——, Qualitative Research with Socio-Technical Grounded Theory . Springer, 2024

work page 2024
[8]

Becoming agile: a grounded theory of agile transitions in practice,

R. Hoda and J. Noble, “Becoming agile: a grounded theory of agile transitions in practice,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 141–151

work page 2017

Show all 26 references

[9]

The rise and evolution of agile software development,

R. Hoda, N. Salleh, and J. Grundy, “The rise and evolution of agile software development,” IEEE software , vol. 35, no. 5, pp. 58–63, 2018

2018
[10]

Challenges and im- provements in distributed software development: A systematic review,

M. Jim ´enez, M. Piattini, and A. Vizca ´ıno, “Challenges and im- provements in distributed software development: A systematic review,” Advances in Software Engineering , vol. 2009, no. 1, p. 710971, 2009

2009
[11]

Qualitative software engineering research: Re- flections and guidelines,

P . Lenberg, R. Feldt, L. Gren, L. G. Wallgren Tengberg, I. Tidefors, and D. Graziotin, “Qualitative software engineering research: Re- flections and guidelines,” Journal of Software: Evolution and Process, vol. 36, no. 6, p. e2607, 2024

2024
[12]

Behavioral software engineering: A definition and systematic literature review,

P . Lenberg, R. Feldt, and L. G. Wallgren, “Behavioral software engineering: A definition and systematic literature review,”Journal of Systems and software, vol. 107, pp. 15–37, 2015

2015
[13]

Qualitative data analysis: An expanded sourcebook,

M. B. Miles, “Qualitative data analysis: An expanded sourcebook,” Thousand Oaks, 1994

1994
[14]

Standards for reporting qualitative research: a synthesis of recommendations,

B. C. O’Brien, I. B. Harris, T. J. Beckman, D. A. Reed, and D. A. Cook, “Standards for reporting qualitative research: a synthesis of recommendations,” Academic medicine, vol. 89, no. 9, pp. 1245– 1251, 2014

2014
[15]

Pandemic programming: How covid-19 affects software developers and how their organizations can help,

P . Ralph, S. Baltes, G. Adisaputri, R. Torkar, V . Kovalenko, M. Kali- nowski, N. Novielli, S. Yoo, X. Devroey, X. Tan et al., “Pandemic programming: How covid-19 affects software developers and how their organizations can help,” Empirical software engineering , vol. 25, pp. ...

2020
[16]

Guidelines for conducting and reporting case study research in software engineering,

P . Runeson and M. H¨ost, “Guidelines for conducting and reporting case study research in software engineering,” Empirical software engineering, vol. 14, pp. 131–164, 2009

2009
[17]

Runeson, M

P . Runeson, M. Host, A. Rainer, and B. Regnell, Case study research in software engineering: Guidelines and examples. John Wiley & Sons, 2012

2012
[18]

Qualitative methods in empirical studies of software engineering,

C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Transactions on software engineering , vol. 25, no. 4, pp. 557–572, 1999

1999
[19]

An empirical study of communi- cation in code inspections,

C. B. Seaman and V . R. Basili, “An empirical study of communi- cation in code inspections,” in Proceedings of the 19th international conference on Software engineering, 1997, pp. 96–106

1997
[20]

Communication and organization: An empirical study of discussion in inspection meetings,

——, “Communication and organization: An empirical study of discussion in inspection meetings,” IEEE Transactions on Software Engineering, vol. 24, no. 7, pp. 559–572, 1998

1998
[21]

The role of ethno- graphic studies in empirical software engineering,

H. Sharp, Y. Dittrich, and C. R. De Souza, “The role of ethno- graphic studies in empirical software engineering,” IEEE Transac- tions on Software Engineering, vol. 42, no. 8, pp. 786–804, 2016. ACCEPTED TO IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, FEB 2025, DOI=HTTPS://DOI.O...

2016
[22]

Publish or perish: Questioning the impact of our research on the software developer,

M. A. Storey, “Publish or perish: Questioning the impact of our research on the software developer,” in 2019 IEEE/ACM 41st Inter- national Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2019, pp. 2–2

2019
[23]

Guiding principles for mixed methods research in software engi- neering,

M. A. Storey, R. Hoda, A. M. P . Milani, and M. T. Baldassarre, “Guiding principles for mixed methods research in software engi- neering,” arXiv preprint arXiv:2404.06011, 2025

2025 arXiv
[24]

Ethical interviews in software engineering,

P . E. Strandberg, “Ethical interviews in software engineering,” in 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 2019, pp. 1–11

2019
[25]

Consolidated criteria for reporting qualitative research (coreq): a 32-item checklist for inter- views and focus groups,

A. Tong, P . Sainsbury, and J. Craig, “Consolidated criteria for reporting qualitative research (coreq): a 32-item checklist for inter- views and focus groups,” International journal for quality in health care, vol. 19, no. 6, pp. 349–357, 2007

2007
[26]

Some guid- ance on conducting and reporting qualitative studies,

P . Twining, R. S. Heller, M. Nussbaum, and C.-C. Tsai, “Some guid- ance on conducting and reporting qualitative studies,” Computers & education, vol. 106, pp. A1–A9, 2017. AUTHOR BIOGRAPHIES Carolyn Seaman is a Professor of Information Systems at the University of Maryland Ba...

2017

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Large language models for qualitative research in software engineering: exploring opportunities and challenges,

M. Bano, R. Hoda, D. Zowghi, and C. Treude, “Large language models for qualitative research in software engineering: exploring opportunities and challenges,” Automated Software Engineering , vol. 31, no. 1, p. 8, 2024

work page 2024

[2] [2]

Editorial for the special issue on qualitative software engineering research,

Y. Dittrich, M. John, J. Singer, and B. Tessem, “Editorial for the special issue on qualitative software engineering research,” Infor- mation and software technology, vol. 49, no. 6, pp. 531–539, 2007

work page 2007

[3] [3]

Qualitative research in software engineering,

T. Dyb ˚a, R. Prikladnicki, K. R ¨onkk¨o, C. Seaman, and J. Sillito, “Qualitative research in software engineering,” Empirical Software Engineering, vol. 16, pp. 425–429, 2011

work page 2011

[4] [4]

Selecting empirical methods for software engineering research,

S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian, “Selecting empirical methods for software engineering research,” Guide to advanced empirical software engineering, pp. 285–311, 2008

work page 2008

[5] [5]

Building theories from case study research,

K. M. Eisenhardt, “Building theories from case study research,” Academy of management review, vol. 14, no. 4, pp. 532–550, 1989

work page 1989

[6] [6]

Socio-technical grounded theory for software engineer- ing,

R. Hoda, “Socio-technical grounded theory for software engineer- ing,” IEEE Transactions on Software Engineering, vol. 48, no. 10, pp. 3808–3832, 2022

work page 2022

[7] [7]

Springer, 2024

——, Qualitative Research with Socio-Technical Grounded Theory . Springer, 2024

work page 2024

[8] [8]

Becoming agile: a grounded theory of agile transitions in practice,

R. Hoda and J. Noble, “Becoming agile: a grounded theory of agile transitions in practice,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 141–151

work page 2017