Pith. sign in

REVIEW 3 major objections 7 minor 24 references

Analysis of LLMs vs Human Experts in Requirements Engineering

T0 review · 3 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In a controlled comparison, GPT-4-generated software requirements were rated more aligned with stakeholders' ideas than those written by human experts, even though participants believed the more aligned documents were human-authored.

desk verdict Transparent exploratory study whose headline alignment claim is confounded by unequal output minimums; the secondary authorship-bias findings are the real contribution. read the letter →

arxiv 2501.19297 v2 pith:YZ7OIX2S submitted 2025-01-31 cs.SE cs.AI

Cory Hymel , Hiroe Johnson This is my paper · ORCID
classification cs.SEcs.AI
keywords largelanguagemodelsrequirementsengineeringelicitationGPT-4userstoriesalignmentcompletenesshuman-AIcomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a controlled comparison between GPT-4 and human requirements experts on the first step of software development: turning a stakeholder's one-paragraph idea into epics and user stories. The authors set out to test whether an LLM can elicit requirements that stakeholders judge as aligned with their original intentions, and whether people can tell who wrote them. They found that LLM-generated documents scored higher on perceived alignment (a mean difference of +1.12 on a 10-point scale, p=0.002) and showed a non-significant trend toward being rated more complete, while costing about 0.06% as much and taking a fraction of the time. At the same time, participants tended to assume that the more aligned document had been written by a human. If the result holds, it supports using LLMs as first-draft requirement generators, with human experts reviewing and refining the output.

What carries the argument

The study's mechanism is a pairwise, source-anonymized comparison: each participant's project description is fed once to GPT-4 through a single-shot prompt that demands at least 40 user stories in a category/feature table, and once to a freelance project manager given one hour and a minimum of 20 features and 5 epics. The two outputs become Document 01 and Document 02, presented blind to the participant, who scores alignment, completeness, guessed source, and self-reported AI familiarity. The comparison is what carries the argument: any observable difference in scores is attributed to the generator, while the design's acknowledged asymmetries (output-length instructions, time budget, single-shot vs interactive) are the key threats to that attribution.

What would settle it

Run the same task with identical minimum-output instructions for both generators (e.g., both instructed to produce at least 40 user stories) and with both allowed the same opportunity for follow-up questions; if the +1.12 alignment advantage shrinks to statistical non-significance, the paper's central claim that LLM output is perceived as more aligned would be refuted. A cheaper check is to test whether alignment scores correlate with document length across all documents in the current dataset.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that in a single-shot, time-boxed comparison, an LLM (GPT-4) given only a stakeholder's short text description generated epics and user stories that stakeholders rated as more aligned with their original idea than a document produced by a freelance project manager in one hour. The average alignment difference was +1.12 on a 10-point scale (8.45 vs 7.35, t(48)=3.179, p=0.002), and the LLM document had a non-significant trend toward being seen as more complete (+10.2% in the 'fairly complete' category). Participants identified the AI document correctly 60.5% of the time but the human document only 41.9% of the time, and they gave higher scores to documents they believed were human-written even when they were actually LLM-generated. The authors take this to show that LLMs can act as first-draft requirement generators, and that the practical barrier to adoption may be an authorship bias rather than a quality deficit.

Load-bearing premise

The result assumes that a single-shot GPT-4 prompt and one hour of a human expert's time are comparable conditions, and in particular that telling the LLM to produce at least 40 user stories while telling the human to produce at least 20 does not itself drive the higher alignment and completeness scores.

Editorial extensions

If this is right

  • Requirements elicitation could shift from a weeks-long, expert-driven process to an iterative loop in which an LLM generates a first draft in seconds and a human expert validates, prunes, and enriches it.
  • At 720x speed and roughly 0.06% of the cost per document, even modest quality parity would make LLM-assisted elicitation economically compelling for early-stage product discovery and rapid prototyping.
  • Stakeholders' tendency to attribute higher-alignment documents to humans implies that disclosing AI authorship may lower perceived quality; organizations adopting LLM drafts should plan for trust-building and transparency norms.
  • Because self-reported AI expertise did not reliably improve detection, LLM-generated requirements are likely to pass as human-written in practice, which raises governance questions about accountability for requirements quality.
  • The wider variance in human scores suggests that human experts bring interpretation and creativity that LLMs currently lack, so the likely near-term outcome is a hybrid workflow rather than full replacement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the alignment advantage stems mainly from the larger number of user stories the LLM was instructed to produce, then matching the minimum counts would likely erase the gap; this is directly testable and would separate a genuine quality edge from a quantity artifact.
  • The finding that participants rate perceived-human documents higher even when they are actually LLM-written suggests a general 'human-authority bias' that may apply to any AI-generated work product, not just requirements; this could be probed with the same design but with authorship labels swapped.
  • The economic figures (about $0.06 vs $100 per document) imply that even a small quality penalty would not overcome the cost advantage for many use cases, so the practical adoption argument may rest more on speed and cost than on the alignment result.
  • A natural next experiment is conversational elicitation, where both the LLM and the human can ask clarifying questions; the paper's authors note this is missing, and it would clarify whether the single-shot constraint artificially deflates human performance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper reports a comparative study between GPT-4 and human expert requirements engineers in generating initial epics and features (user stories) for software systems described by stakeholders. Fifty stakeholders rated the anonymized outputs for alignment, completeness, and perceived authorship. The authors report that LLM-generated requirements received significantly higher alignment scores (mean difference +1.12, t(48)=3.179, p=0.002), a non-significant trend toward higher completeness (+10.2% in the 'Fairly Complete' category, p=0.877), and that participants over-attributed documents to AI. They conclude that LLMs will play an increasingly important role in requirements elicitation. A central design feature is that the LLM was given a single-shot prompt requesting at least 40 user stories, while human experts were instructed to produce a minimum of 20 features and 5 epics within one hour.

Significance. If the alignment advantage were causal, the finding would be practically significant for requirements engineering, suggesting that LLMs can produce stakeholder-aligned requirement documents at a fraction of the time and cost of human experts. The paper also reports an interesting secondary result on authorship attribution bias that is not affected by the main confound. However, the headline claim about alignment is not supported by the current design because the two conditions differ in required output volume and in the time/effort assumption. The paper is a useful pilot with transparent appendices, but its central conclusion requires a matched experimental design to be credible.

major comments (3)
  1. [IV-B and Appendix Figs. A-I/A-II] The two conditions differ in required output volume: the LLM prompt demands 'at a minimum, 40 user stories' (Appendix Fig. A-I), while the human expert instructions require 'Minimum of 20 features, Minimum of 5 epics' (Appendix Fig. A-II). The authors also report in Section VI that AI-generated documents were on average five user stories longer. Because perceived alignment and completeness can be influenced by document length and level of detail, the observed +1.12 alignment difference cannot be uniquely attributed to the generator (GPT-4 vs. human expert). The acknowledgment in IV-B that the instruction differs does not control for this; no analysis matching output length or including length as a covariate is provided. This is a load-bearing confound for Q1 and the abstract's central claim.
  2. [Abstract and Section V-C] The abstract characterizes the completeness result as a 'trend of being more complete (+10.2%)', but the underlying test is not statistically significant (t(94)=0.155, p=0.877). The +10.2% refers to a difference in the proportion of 'Fairly Complete' ratings, not an overall completeness score. Reporting a non-significant difference as a 'trend' in the abstract overstates the evidence and is misleading.
  3. [Section IV-F (Limitation 5) and Section VI-A] The design equates a single-shot LLM prompt with one hour of human expert work, an assumption the authors acknowledge as a limitation. However, the Discussion (Section VI-A) then concludes that LLMs 'demonstrated a significant ability to create aligned and comprehensive requirements quickly,' which overstates what the design can support. The single-shot vs. one-hour asymmetry, combined with the unequal output-minimum instructions, means the observed differences could reflect effort or output volume rather than the intrinsic capability of the generator. The conclusion should be tempered accordingly.
minor comments (7)
  1. [Section III] The paper describes a 'mixed-methods approach,' but no qualitative analysis appears anywhere in the results; the study is entirely survey-based quantitative. Either include qualitative data or remove the 'mixed-methods' label.
  2. [Table I] The note 'These tables are not correlated and presented in this way for formatting purposes only' is confusing; clarify whether the two columns come from the same respondents or are separate univariate distributions.
  3. [Figures] There are two figures labeled 'Fig. 1' (the study process and the alignment scores). Renumber the figures sequentially throughout the paper.
  4. [Appendix Table A-7] The formula for LLM cost reads '(TI * TIP) + (TR + TRP)', but the units suggest it should be '(TI * TIP) + (TR * TRP)'. Please check and correct.
  5. [Section V-A] The phrase 'statistically significant (t(48) = 3.179, p = 0.002) understanding of alignment score' appears to contain a typo ('understanding' likely should be 'difference').
  6. [Section V-C] The opening sentence says completeness ratings were 'roughly the same' but then reports category-level differences of +10.2% and +14.3%; these statements are in tension and should be rephrased for clarity.
  7. [Appendix] The text references Appendix Fig. A-I and Fig. A-II, but the manuscript as provided does not include the actual images; ensure the final version contains the figures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an external participant-judgment comparison, not a derivation in which outputs are defined by inputs.

full rationale

This paper reports an empirical comparison of GPT-4-generated requirements against human-expert output using blinded participant ratings of alignment, completeness, and source attribution. There is no fitted parameter that is later renamed as a prediction, no quantity is defined in terms of another quantity being predicted, and no load-bearing self-citation: all 23 references are to external prior work, and none of them supplies a uniqueness theorem or an ansatz that the present study needs. The central numeric claims (+1.12 alignment, +10.2% completeness trend) are observed mean differences from participant survey responses, not outputs of a model whose inputs include those same responses. The paper explicitly discloses the main design asymmetry in Section IV-B: the LLM prompt required at least 40 user stories while human experts were instructed to provide a minimum of 20 features and 5 epics, and Section IV-A and Section IV-F acknowledge the 'single shot = 1 hour' assumption as a pragmatic limitation. These are genuine internal-validity concerns about whether the comparison is fair, but they do not make the argument circular: the alignment and completeness ratings are external judgments made by stakeholders who did not know the document source, and the conclusion would not be true by construction even under the unequal instructions. A confound is not a circularity. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The study's conclusions rest on several unverified domain assumptions about the validity of subjective ratings, the representativeness of crowdsourced experts, and the absence of bias from data removal. The most consequential free parameters are the output minimums, which were chosen differently for the two conditions.

free parameters (3)
  • LLM minimum user stories = 40
    The single-shot prompt explicitly requires at least 40 user stories (Appendix Fig. A-I). This was chosen after early testing, and it differs from the human expert minimum of 20 features, directly confounding output volume with quality.
  • Human expert minimum features = 20
    The human expert prompt requires a minimum of 20 features and 5 epics (Appendix Fig. A-II). This lower minimum than the LLM means humans may produce shorter documents, biasing completeness and alignment ratings.
  • Time-box parity = 1 hour vs single-shot
    The study equates a single LLM call with one hour of human expert work (Section IV-A). This arbitrary equivalence affects cost and speed comparisons and is acknowledged as a limitation by the authors.
assumptions (4)
  • domain assumption Participant self-reported alignment and completeness are valid measures of requirements quality.
    The study uses 1-10 self-ratings by the idea submitter as the ground truth for alignment and categorical ratings for completeness. No external or objective requirements-quality metric is used.
  • domain assumption Upwork freelancers meeting the listed criteria represent human requirements-engineering experts.
    Human experts were crowdsourced from Upwork with conversational English and skills in requirements specification and project management. No verification of expertise beyond self-attestation and Upwork profile is reported (Section IV-B).
  • domain assumption The survey 'bug' that allowed both AI and Human answers affected a random subset and did not bias results.
    Section IV-E states data from those subjects were removed, but the number and characteristics are not reported, so selection bias from removal cannot be excluded.
  • standard math Standard t-test assumptions hold for the Likert and categorical ratings.
    The paper applies paired and independent t-tests to 1-10 scale and categorical data without reporting normality or variance checks; the human-document scores show variance 5.23 vs 1.84 for AI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analysis of LLMs vs Human Experts in Requirements Engineering." pith.science (2026). https://pith.science/paper/YZ7OIX2S

@misc{pith2026250119297,
  author       = {Pith},
  title        = {Pith review of: Analysis of LLMs vs Human Experts in Requirements Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZ7OIX2S}},
  note         = {Machine review of arXiv:2501.19297}
}
read the original abstract

The majority of research around Large Language Models (LLM) application to software development has been on the subject of code generation. There is little literature on LLMs' impact on requirements engineering (RE), which deals with the process of developing and verifying the system requirements. Within RE, there is a subdiscipline of requirements elicitation, which is the practice of discovering and documenting requirements for a system from users, customers, and other stakeholders. In this analysis, we compare LLM's ability to elicit requirements of a software system, as compared to that of a human expert in a time-boxed and prompt-boxed study. We found LLM-generated requirements were evaluated as more aligned (+1.12) than human-generated requirements with a trend of being more complete (+10.2%). Conversely, we found users tended to believe that solutions they perceived as more aligned had been generated by human experts. Furthermore, while LLM-generated documents scored higher and performed at 720x the speed, their cost was, on average, only 0.06% that of a human expert. Overall, these findings indicate that LLMs will play an increasingly important role in requirements engineering by improving requirements definitions, enabling more efficient resource allocation, and reducing overall project timelines.

Figures

Figures reproduced from arXiv: 2501.19297 by the authors.

Figure 1
Figure 1. Alignment scores for both AI and human generated documents. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Heuristics of perceived alignment comparing scores given to solutions identified as AI-generated and human-generated regardless of whether these identifications were correct. Users tended to believe solutions they perceived as more aligned were generated by human experts. Solutions perceived to be AI-generated received lower scores, with seven ratings of 3 or lower, while solutions perceived to be human-generated ha… view at source ↗
Figure 3
Figure 3. Distribution of those attributing sources to each document. Full breakdown in Appendix Table A-2 Notably, 60.5% of Participants correctly identified Document 01 as AI-generated, while only 41.9% accurately recognized Document 02 as human-generated. Participants chose AI as the primary document creator both times, suggesting a tendency to over-attribute content to AI sources. C. COMPLETENESS RATINGS Completeness rati… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

  1. [1]

    Hou et al., Large Language Models for Software Engineering: A Systematic Literature Review

    X. Hou et al., Large Language Models for Software Engineering: A Systematic Literature Review. 2024. [Online]. Available: https://arxiv.org/abs/2308.10620

    arXiv 2024

  2. [2]

    The Role of Requirements in the Success or Failure of Software Projects,

    A. Hussain, E. Mkpojiogu, and F. Kamal, “The Role of Requirements in the Success or Failure of Software Projects,” EJ Econjournals, vol. 6, pp. 6–7, Oct. 2016

    work page 2016

  3. [3]

    Artificial Intelligence in Software Requirements Engineering: State-of-the-Art,

    K. Liu, S. Reddivari, and K. Reddivari, “Artificial Intelligence in Software Requirements Engineering: State-of-the-Art,” in 2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI), 2022, pp. 106 –111. doi: 10.1109/IRI54793.2022.00034

    work page arXiv 2022

  4. [4]

    Requirements Elicitation Problems: A Literature Analysis,

    B. Davey and K. R. Parker, “Requirements Elicitation Problems: A Literature Analysis,” Issues in Informing Science and Information Technology, vol. 12, pp. 71 –82, 2015, [Online]. Available: https://api.semanticscholar.org/CorpusID:61495831

    work page 2015

  5. [5]

    Requirements engineering: a roadmap,

    B. Nuseibeh and S. Easterbrook, “Requirements engineering: a roadmap,” in Proceedings of the Conference on The Future of Software Engineering, 2000, pp. 35–46. doi: 10.1145/336512.336523

    work page arXiv

  6. [6]

    Zowghi, D., Coulin , C. (2005). Requirements Elicitation: A Survey of Techniques, Approaches, and Tools. In: Aurum, A., Wohlin, C. (eds) Engineering and Managing Software Requirements. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-28244-0_2

    doi:10.1007/3-540-28244-0_2 2005

  7. [7]

    From Ideas to Expressed Needs: an Empirical Study on the Evolution of Requirements during Elicitation,

    S. Debnath, P. Spoletini, and A. Ferrari, “From Ideas to Expressed Needs: an Empirical Study on the Evolution of Requirements during Elicitation,” Sep. 2021, pp. 233– 244. doi: 10.1109/RE51729.2021.00028

    work page arXiv 2021

  8. [8]

    Systematizing requirements elicitation technique selection,

    D. C. Moreno, Óscar Dieste Tubío, and N. J. Juzgado, “Systematizing requirements elicitation technique selection,” Inf. Softw. Technol., vol. 56, pp. 644– 669, 2014, [Online]. Available: https://api.semanticscholar.org/CorpusID:195712559

    work page 2014

Show all 24 references
  1. [9]

    Requirements Elicitation and Modelling of Artificial Intelligence Systems: An Empirical Study,

    K. Ahmad, M. Abdelrazek, C. Arora, J. C. Grundy, and M. Bano, “Requirements Elicitation and Modelling of Artificial Intelligence Systems: An Empirical Study,” 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:256827598

    2023

  2. [10]

    Requirements elicitation: Towards the unknown unknowns,

    A. Sutcliffe and P. Sawyer, “Requirements elicitation: Towards the unknown unknowns,” in 2013 21st IEEE International Requirements Engineering Conference (RE), 2013, pp. 92 –104. doi: 10.1109/RE.2013.6636709

    2013

  3. [11]

    Gervasi, V. et al. (2013). Unpacking Tacit Knowledge for Requirements Engineering. In: Maalej, W., Thurimella, A. (eds) Managing Requirements Knowledge. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34419-0_2

    2013 doi

  4. [12]

    A new approach for software requirements elicitation,

    P. Rajagopal, R. Y. Lee, T. Ahlswede, C.- C. Chiang, and D. Karolak, “A new approach for software requirements elicitation,” Sixth International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing and First ACIS Internatio...

    2005

  5. [13]

    Padmakumar, V., & He, H. (2024). Does Writing with Language Models Reduce Content Diversity? https://arxiv.org/abs/2309.05196

    2024 arXiv

  6. [14]

    R., Shah, J

    Anderson, B. R., Shah, J. H., & Kreminski, M. (2024). Homogenization Effects of Large Language Models on Human Creative Ideation. Proceedings of the 16th Conference on Creativity & Cognition, 413 –

    2024

  7. [15]

    Lermann Henestrosa, A., & Kimmerle, J. (2024). The Effects of Assumed AI vs. Human Authorship on the Perception of a GPT - Generated Text. Journalism and Media, 5(3), 1085 –1097. https://doi.org/10.3390/journalmedia5030069

    2024 doi

  8. [16]

    Revealing the source: How awareness alters perceptions of AI and human-generated mental health responses

    Jain G, Pareek S, Carlbring P. Revealing the source: How awareness alters perceptions of AI and human-generated mental health responses. Internet Interv. 2024 Apr 27;36:100745. doi: 10.1016/j.invent.2024.100745. PMID: 38746873; PMCID: PMC11090870

    2024

  9. [17]

    (2023) Human favoritism, not AI aversion: People’s perceptions (and bias) toward generative AI, human experts, and human –GAI collaboration in persuasive content generation

    Zhang Y, Gosline R. (2023) Human favoritism, not AI aversion: People’s perceptions (and bias) toward generative AI, human experts, and human –GAI collaboration in persuasive content generation. Judgement and Desicion Making. Vol.18:e41 1 -16 doi:10.1017/jdm.2023.37

    2023 doi

  10. [18]

    (2020) AI -generated vs

    Ragot M, Martin N, Cojean S. (2020) AI -generated vs. Human Artworks. A Perception Bias Towards Artificial Intelligence?. HAL open science. CHI ’20: CHI Conference on Human Factors in Computing Systems, Apr 2020, Honolulu, United States. doi:10.1145/3334480.3382892

    2020

  11. [19]

    (2014) Enter the Robot Journalist: Users' perceptions of automated content

    Clerwall, C. (2014) Enter the Robot Journalist: Users' perceptions of automated content. Journalism Practice. http://dx.doi.org/10.1080/17512786.2014.883116

    2014

  12. [20]

    (2003) How do users evaluate the credibility of web sites? a study with over 2,500 Participants

    Fogg B, Soohoo C, Danielson D, Marable L, Stanford J, Tauber E. (2003) How do users evaluate the credibility of web sites? a study with over 2,500 Participants. In Proceedings of the 2003 Conference on Designing for User Experiences, DUX '03, page 1-15, New York, NY, USA, 2003...

    2003

  13. [21]

    (2023) Value attributed to text -based archives generated by artificial intelligence

    Darda K, Carre M, Cross E. (2023) Value attributed to text -based archives generated by artificial intelligence. https://doi.org/10.1098/rsos.220915

    2023 doi

  14. [22]

    (2024) The Effects of Perceived AI Use On Content Perceptions

    Rae I. (2024) The Effects of Perceived AI Use On Content Perceptions. CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems Article No.: 978, Pages 1 - 14. https://dl.acm.org/doi/full/10.1145/3613904.3642076

    2024

  15. [23]

    Fully Complete

    Gao C, Howard F, Markov N, Dyer E, Ramesh S, Luo Y, Pearson A. (2023) Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. npj Digit. Med. 6, 75 (2023). npj Digit. Med. 6, 75 (2023). https://doi.org/10.1038/s41746-02...

    2023 doi

  16. [425]

    https://doi.org/10.1145/3635636.3656204

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.