Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

From Inductive to Deductive: LLMs-Based Qualitative Data Analysis in Requirements Engineering

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4, prompted with a few labeled examples and detailed system context, reproduces human qualitative labels on software requirements at substantial agreement (Cohen's kappa up to 0.738), while zero-shot labeling lags.

desk verdict Useful empirical measurements, but the consensus-only ground truth likely inflates the headline Kappa values, and the paper overclaims by saying LLMs can match or exceed human performance. read the letter →

arxiv 2504.19384 v1 pith:KOYGBMPN submitted 2025-04-27 cs.SE cs.AI

Syed Tauhid Ullah Shah , Mohamad Hussein , Ann Barcomb , Mohammad Moshirpour This is my paper · ORCID
classification cs.SEcs.AI
keywords RequirementsEngineeringQualitativeDataAnalysisLargeLanguageModelsGPT-4Few-shotLearningZero-shotCohen'sKappaAnnotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that a general-purpose LLM can carry out the labeling step of qualitative data analysis for requirements engineering at a level close to human judgment, provided it is given a few labeled examples and enough system context. Its headline evidence is GPT-4 reaching Cohen's $\kappa$ of 0.738 and 0.734 in the few-shot setting on two test systems, which falls in the substantial-agreement band, while zero-shot agreement stays near 0.54. The paper also claims that this result is driven by prompt design: longer prompts and fuller system descriptions improve agreement, and GPT-4's labels are highly reproducible across runs. The reason to care is that this labeling step currently consumes dozens of analyst-hours on a single project, and the resulting labels feed directly into traceable domain models.

What carries the argument

The carrying mechanism is the prompt template, varied along three independent dimensions: shot type (zero examples for inductive, one or several for deductive), prompt length (short, medium, long), and context level (none, some, full system description). These prompts turn qualitative coding into a constrained multi-class labeling task in which the model must output a single one-word category per requirement. The evaluation protocol then compares model labels against the consensus labels of two analysts using Cohen's $\kappa$, accuracy, precision, recall, and F1, and checks repeated-run reliability with standard deviation and intraclass correlation; all models were run at temperature 0.0.

What would settle it

Take the same few-shot prompts to a third requirements document from a different domain, score the model against a panel of five analysts who label every requirement, including the ambiguous ones, and check whether Cohen's $\kappa$ stays above roughly 0.70; if it falls substantially, the claimed agreement depends on the consensus-trimmed ground truth or on the two chosen systems.

Watch

Extended reading notes

Core claim

The paper's central claim is that deductive annotation, implemented as one-shot and few-shot prompting, lets the tested LLMs approach human performance on qualitative coding of software requirements, whereas inductive zero-shot annotation does not; GPT-4 is the strongest of the three models. On the Library Management System, GPT-4 achieves Cohen's $\kappa=0.738$ in the few-shot setting, and on the Smart Home System $\kappa=0.734$, against a ground truth formed by the labels on which two human analysts agreed, whose own agreement was 0.80 and 0.78 respectively. The paper states that these scores indicate substantial agreement and that accuracy, precision, and recall are correspondingly high, for example accuracy 0.86 and F1 0.79 in the Library few-shot setting. It further claims that adding full contextual background and longer instructions is the main lever that moves scores up from the roughly 0.54 zero-shot baseline, and that GPT-4 shows higher run-to-run consistency (ICC 0.93 and 0.92) than the open models tested.

Load-bearing premise

The whole evaluation rests on treating the labels on which two analysts agreed, for just two software systems, as the correct labels for requirements annotation in general; if that consensus subset skips the hardest or most ambiguous requirements, the reported agreement overstates how the method would perform in real use.

Editorial extensions

If this is right

  • A few-shot LLM could serve as a first-pass annotator for deductive coding, cutting the roughly 74 analyst-hours the paper reports for manually labeling two systems.
  • Structured labels produced automatically can be mapped to classes in a domain model, shortening the path from stakeholder input to a design model.
  • Teams wanting reliable annotations would need to invest in long, context-rich prompts with a few worked examples, since shorter or context-free prompts produce materially lower agreement.
  • Zero-shot, fully inductive annotation is not ready for autonomous use; the near-0.54 kappa scores put it below the threshold for dependable coding.
  • Run-to-run consistency above ICC 0.85 makes the labels reproducible enough for workflows that need auditing or repeatable outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but the consensus-only ground truth makes the reported kappas an upper bound; a deployment that labels every requirement, including disputed ones, would likely show lower agreement.
  • The one-word-label setup is effectively multiclass classification, so calibration, confidence thresholds, or abstention options could raise the currently weak zero-shot performance.
  • The same few-shot prompt recipe could plausibly be reused for other coding tasks in requirements analysis, such as tagging ambiguity or stakeholder intent, since it only needs a few examples and a system description.
  • A further consequence the paper leaves implicit: once deductive annotation is cheap, the bottleneck of qualitative analysis shifts from labeling to constructing and validating the codebook, because the few-shot examples encode the categories.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates the use of large language models (LLMs) for qualitative data analysis (QDA) in requirements engineering, specifically for labeling/coding requirements statements. The authors map inductive QDA to zero-shot prompting and deductive QDA to one-shot and few-shot prompting, testing GPT-4, Mistral, and LLaMA-2 on two test cases (Library Management System and Smart Home System). Human analysts labeled the data independently and then reconciled their labels; Cohen's Kappa between them was 0.80 and 0.78. The LLMs were evaluated against the human consensus labels using Kappa, accuracy, precision, recall, F1, and consistency metrics (SD, ICC). The main finding is that GPT-4 in the few-shot setting achieves Kappa 0.738 and 0.734 for the two test cases, indicating substantial agreement with human analysts, while zero-shot performance is around 0.54. The paper also reports that longer, context-rich prompts improve performance, and that GPT-4 is more consistent across runs than the other models.

Significance. If the central result is valid, the paper provides useful evidence that a general-purpose LLM can serve as a supporting tool for deductive qualitative coding in RE, potentially reducing manual effort. The inclusion of a human-human baseline (Kappa 0.80/0.78) is a strength, as is the evaluation of multiple models and prompt configurations, and the public availability of the code. However, the evaluation's validity is undermined by the use of consensus-only ground truth, which likely excludes the most difficult items, and by an overstatement in Section 5.3 that LLMs "can match or even exceed human performance" despite lower Kappa than the human-human baseline. These issues need to be addressed before the significance claim can be fully accepted.

major comments (4)
  1. [5.1] Section 5.1 states that "we used only the labels on which both analysts reached consensus as the ground truth for evaluating LLM performance." Because the human analysts themselves achieved only substantial agreement (Kappa 0.80 for the Library Management System and 0.78 for the Smart Home System, Section 3), a nontrivial fraction of requirement items were initially disputed. Excluding those items likely removes the hardest, most ambiguous cases, so the reported Kappa values (e.g., 0.738 and 0.734 for GPT-4 few-shot in Table 4) measure agreement only on an easy subset rather than on the full set of stakeholder requirements. The paper does not report the number or proportion of excluded items, and it does not separately report performance on the full 'resolved' set of labels described in Section 3. The central claim that GPT-4 achieves substantial agreement with human analysts in deductive annotation is therefore not established for the full dataset. The authors should either evaluate on the full resolved ground truth or quantify the exclusion and show that the result is robust to it.
  2. [5.3] In Section 5.3 the paper claims that LLMs "can match or even exceed human performance" in deductive settings. This is contradicted by the paper's own numbers: the human-human Kappa is 0.80 and 0.78 (Section 3), while the best LLM-human Kappa is 0.738 and 0.734 (Table 4). A lower LLM-human Kappa than the human-human baseline does not support "match or exceed." This sentence should be corrected to reflect that LLMs approach but do not reach human-human agreement, unless additional evidence is provided.
  3. [6.1] Section 6.1 states that prompts were "refining them through an iterative process" and that different lengths and context levels were tested. If the prompt templates and few-shot examples were selected after evaluating on the same test sets used for the reported results, the Kappa values may be optimistically biased by prompt overfitting. The paper should report whether a held-out development set was used to choose the prompt configuration, and how the few-shot examples were selected (e.g., randomly or based on performance). Without this, the internal validity of the reported agreement scores is in question.
  4. [5.1 and 5.5] The paper reports Cohen's Kappa in Table 4 without dataset sizes, category counts, or confidence intervals, making it impossible to assess whether the small numerical differences between models (e.g., 0.738 vs. 0.734 for GPT-4) are meaningful. Additionally, Section 5.5 applies Intraclass Correlation Coefficient (ICC) and standard deviation to categorical label outputs in Table 6; ICC is designed for continuous measurements, and the standard deviation of categorical labels depends on an arbitrary numeric encoding. The paper should either justify the use of ICC for this categorical data or use an appropriate categorical consistency metric, and should report the number of items and uncertainty intervals for the Kappa estimates.
minor comments (5)
  1. [7] The conclusion refers to 'LLM' as a model name ("specifically LLM, Mistral, and LLaMA-2"); this should be 'GPT-4'.
  2. [Table 4] The caption of Table 4 and the surrounding text should specify the exact prompt configuration (length and context level) used for the reported Kappa scores, since Table 5 shows variation by prompt length and Table 7 shows variation by context level.
  3. [Abstract] The abstract says "Cohen's Kappa scores exceeding 0.7", but the introduction says "up to 0.738"; clarify whether the claim is about the point estimates exceeding 0.7 in both test cases.
  4. [5.3] The phrase "approaching human performance" is used before the overclaim; consider using consistent language throughout the results section.
  5. [6.1] The internal validity section does not discuss the consensus-only ground truth issue; this should be acknowledged as a threat to internal validity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GPT-4 agreement is benchmarked against external human consensus labels.

full rationale

The paper's central quantitative claim is that GPT-4 achieves Cohen's Kappa scores of 0.738 and 0.734 against human consensus labels in the few-shot setting (Section 5.3, Table 4). These scores are computed by comparing LLM outputs to human-produced ground truth labels (Section 5.1), so the result is not defined in terms of the model's own outputs. No model parameter or codebook is fitted to the target Kappa values; the prompt templates and few-shot examples are reported as fixed experimental conditions, and temperature is set to 0.0 to reduce randomness (Section 5.2). The paper's self-citations to QDAcity-RE and related work by overlapping authors (references [4], [17], [20], [21]) motivate the labeling-to-domain-model connection, but they are not load-bearing for the agreement results, which stand on the reported experiments. The consensus-only ground truth choice (Section 5.1) and the phrase 'match or even exceed human performance' (Section 5.3) are potential threats to validity or overstatement, not circularity: the evaluation still measures agreement against an external reference rather than reducing to the model's own predictions or to a fitted parameter. Accordingly, no step in the derivation chain is equivalent to its inputs by construction, and no self-citation chain forces the central result.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

This is an empirical benchmark study, not a derivation, so there are no fitted constants. The central claim rests on assumptions about the validity of human consensus labels, the suitability of Cohen's Kappa and ICC for this categorical, small-sample setting, and the representativeness of two test systems. No new entities are introduced.

assumptions (5)
  • domain assumption Human analyst consensus labels are a valid ground truth for evaluating LLM annotation.
    Section 3 and Section 5.1 treat labels agreed by both analysts as ground truth; if those labels are biased or unrepresentative, all agreement metrics lose their reference point.
  • standard math Cohen's Kappa thresholds for 'substantial agreement' apply to this multi-class, imbalanced annotation setting.
    Section 5.1 interprets Kappa without reporting class distribution or prevalence, both of which can strongly affect Kappa values.
  • domain assumption The two test cases, Library Management and Smart Home, and the selected documents are representative enough to support general claims about QDA in RE.
    Section 3 selects two systems from the PURE dataset plus internet supplements; Section 6.2 admits this may not capture the diversity of real-world projects.
  • domain assumption LLM-generated label strings map directly onto human analyst labels with no normalization step.
    Section 5.1 assumes the model's one-word labels align with the analysts' label set; no synonym handling or string matching is described.
  • domain assumption Intraclass correlation is an appropriate reliability measure for categorical labels.
    Section 5.1 and Table 6 use ICC and SD for label consistency across runs without explaining how categorical labels were converted to continuous scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Inductive to Deductive: LLMs-Based Qualitative Data Analysis in Requirements Engineering." pith.science (2026). https://pith.science/paper/KOYGBMPN

@misc{pith2026250419384,
  author       = {Pith},
  title        = {Pith review of: From Inductive to Deductive: LLMs-Based Qualitative Data Analysis in Requirements Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOYGBMPN}},
  note         = {Machine review of arXiv:2504.19384}
}
read the original abstract

Requirements Engineering (RE) is essential for developing complex and regulated software projects. Given the challenges in transforming stakeholder inputs into consistent software designs, Qualitative Data Analysis (QDA) provides a systematic approach to handling free-form data. However, traditional QDA methods are time-consuming and heavily reliant on manual effort. In this paper, we explore the use of Large Language Models (LLMs), including GPT-4, Mistral, and LLaMA-2, to improve QDA tasks in RE. Our study evaluates LLMs' performance in inductive (zero-shot) and deductive (one-shot, few-shot) annotation tasks, revealing that GPT-4 achieves substantial agreement with human analysts in deductive settings, with Cohen's Kappa scores exceeding 0.7, while zero-shot performance remains limited. Detailed, context-rich prompts significantly improve annotation accuracy and consistency, particularly in deductive scenarios, and GPT-4 demonstrates high reliability across repeated runs. These findings highlight the potential of LLMs to support QDA in RE by reducing manual effort while maintaining annotation quality. The structured labels automatically provide traceability of requirements and can be directly utilized as classes in domain models, facilitating systematic software design.

Figures

Figures reproduced from arXiv: 2504.19384 by the authors.

Figure 1
Figure 1. outlines our approach to integrating LLMs into QDA for RE. We begin by taking requirement statements (Section. 3) as input. The requirements are subsequently formatted into structured prompts optimized for inductive or deductive annotations (Section. 4.2). Inductive prompts, used in zero-shot learning, allow LLMs to identify patterns without predefined categories, while deductive prompts, supporting one-shot and few… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GenAI Is No Silver Bullet for Qualitative Research in Software Engineering

    cs.SE 2026-03 conditional novelty 4.0 of 10

    GenAI tools can usefully assist narrow deductive coding and transcription in qualitative SE research, but current evidence does not support autonomous or interpretive use.

Reference graph

Works this paper leans on

33 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    B. H. Cheng, J. M. Atlee, Research directions in requirements engineering, Future of software engineering (FOSE’07) (2007) 285–303

    work page 2007

  2. [2]

    Carrizo, O

    D. Carrizo, O. Dieste, N. Juristo, Systematizing requirements elicitation technique selection, Information and Software Technology 56 (2014) 644–669

    work page 2014

  3. [3]

    Mucha, The QDAcity-RE-RS Method for Creating Complete, Consistent, and Traceable Require- ments Specifications, Friedrich-Alexander-Universitaet Erlangen-Nuernberg (Germany), 2023

    J. Mucha, The QDAcity-RE-RS Method for Creating Complete, Consistent, and Traceable Require- ments Specifications, Friedrich-Alexander-Universitaet Erlangen-Nuernberg (Germany), 2023

    work page 2023

  4. [4]

    Kaufmann, J

    A. Kaufmann, J. Krause, N. Harutyunyan, A. Barcomb, D. Riehle, A validation of QDAcity–RE for domain modeling using qualitative data analysis, Requirements Engineering (2021). URL: https://link.springer.com/article/10.1007/s00766-021-00360-6. doi: https://doi.org/10.1007/ s00766-021-00360-6

    work page doi:10.1007/s00766-021-00360-6 2021

  5. [5]

    Saldaña, The coding manual for qualitative researchers (2021)

    J. Saldaña, The coding manual for qualitative researchers (2021)

    work page 2021

  6. [6]

    Qualitative Data Analysis in Software Engineering: Techniques and Teaching Insights

    C. Treude, Qualitative data analysis in software engineering: Techniques and teaching insights, arXiv preprint arXiv:2406.08228 (2024)

    work page Pith review arXiv 2024

  7. [7]

    S. Tsang, An experiment exploring the theoretical and methodological challenges in developing a semi-automated approach to analysis of small-n qualitative data, arXiv preprint arXiv:2002.04513 (2020)

    work page Pith review arXiv 2020

  8. [8]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

Show all 33 references
  1. [9]

    G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al., Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  2. [10]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhar- gava, S. Bhosale, et al., Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  3. [11]

    X. Sun, X. Li, J. Li, F. Wu, S. Guo, T. Zhang, G. Wang, Text classification via large language models, in: The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  4. [12]

    Zhang, F

    T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, T. B. Hashimoto, Benchmarking large language models for news summarization, Transactions of the Association for Computational Linguistics 12 (2024) 39–57

    2024

  5. [13]

    Zhang, B

    B. Zhang, B. Haddow, A. Birch, Prompting large language model for machine translation: A case study, in: International Conference on Machine Learning, PMLR, 2023, pp. 41092–41110

    2023

  6. [14]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901

    2020

  7. [15]

    Krishna, B

    M. Krishna, B. Gaur, A. Verma, P. Jalote, Using llms in software requirements specifications: An empirical evaluation, arXiv preprint arXiv:2404.17842 (2024)

    2024 arXiv

  8. [16]

    Nuseibeh, S

    B. Nuseibeh, S. Easterbrook, Requirements engineering: a roadmap, in: Proceedings of the Conference on the Future of Software Engineering, 2000, pp. 35–46

    2000

  9. [17]

    Kaufmann, J

    A. Kaufmann, J. Krause, N. Harutyunyan, A. Barcomb, D. Riehle, A validation of qdacity-re for domain modeling using qualitative data analysis, Requirements Engineering 27 (2022) 31–51

    2022

  10. [18]

    N.-C. Chen, R. Kocielnik, M. Drouhard, V. Peña-Araya, J. Suh, K. Cen, X. Zheng, C. R. Aragon, Challenges of applying machine learning to qualitative coding, in: ACM SIGCHI Workshop on Human-Centered Machine Learning, 2016

    2016

  11. [19]

    Glaser, A

    B. Glaser, A. Strauss, Discovery of grounded theory: Strategies for qualitative research, Routledge, 2017

    2017

  12. [20]

    Kaufmann, D

    A. Kaufmann, D. Riehle, The QDAcity-RE method for structural domain modeling using qualitative data analysis, Requirements Engineering 24 (2019) 85–102

    2019

  13. [21]

    Kaufmann, A

    A. Kaufmann, A. Barcomb, D. Riehle, Supporting interview analysis with autocoding, in: 53rd Hawaii International Conference on System Sciences, HICSS 2020, Maui, Hawaii, USA, January 7-10, 2020, ScholarSpace, 2020, pp. 1–10

    2020

  14. [22]

    Vogelsang, J

    A. Vogelsang, J. Fischbach, Using large language models for natural language processing tasks in requirements engineering: A systematic guideline, arXiv e-prints (2024) arXiv–2402

    2024

  15. [23]

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, J. M. Zhang, Large language models for software engineering: Survey and open problems, arXiv preprint arXiv:2310.03533 (2023)

    2023 arXiv

  16. [24]

    M. Bano, R. Hoda, D. Zowghi, C. Treude, Large language models for qualitative research in software engineering: exploring opportunities and challenges, Automated Software Engineering 31 (2024) 8

    2024

  17. [25]

    Alhoshan, A

    W. Alhoshan, A. Ferrari, L. Zhao, Zero-shot learning for requirements classification: An exploratory study, Information and Software Technology 159 (2023) 107202

    2023

  18. [26]

    D. Kici, G. Malik, M. Cevik, D. Parikh, A. Basar, A bert-based transfer learning approach to text classification on software requirements specifications., in: Canadian AI, 2021

    2021

  19. [27]

    Ferrari, G

    A. Ferrari, G. O. Spagnolo, S. Gnesi, Pure: a dataset of public requirements documents, Unspecified Journal Unspecified Volume (2023) Unspecified Pages

    2023

  20. [28]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  21. [29]

    K. E. Wiegers, J. Beatty, Software requirements, Pearson Education, 2013

    2013

  22. [30]

    M. L. Coleman, M. Ragan, T. Dari, Intercoder reliability for use in qualitative research and evaluation, Measurement and Evaluation in Counseling and Development 57 (2024) 136–146

    2024

  23. [31]

    R. Chew, J. Bollenbacher, M. Wenger, J. Speer, A. Kim, Llm-assisted content analysis: Using large language models to support deductive coding, arXiv preprint arXiv:2306.14924 (2023)

    2023 arXiv

  24. [32]

    Turpin, J

    M. Turpin, J. Michael, E. Perez, S. Bowman, Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting, Advances in Neural Information Processing Systems 36 (2024)

    2024

  25. [33]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837

    2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.