{"id":"d1c5c30a-622b-4a9d-8655-b82157b3a0e3","arxiv_id":"2504.19384","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4 labels software requirements with substantial agreement to human analysts (Cohen's Kappa up to 0.738) when given detailed few-shot prompts, while zero-shot performance is only moderate.","lead":"This paper tested whether three large language models (GPT-4, Mistral, and LLaMA-2) can label software requirements the way human analysts do, using prompts with or without examples. It found that GPT-4 agreed substantially with human labels when given detailed examples, suggesting AI could reduce the manual effort in requirements analysis.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Consensus-only ground truth may inflate GPT-4 agreement; Kappa on full resolved set needed.","rationale":"The reader's weakest_assumption correctly identifies the consensus-only ground truth as the key threat. My analysis of the manuscript confirms this: Section 5.1 explicitly restricts evaluation to consensus labels, and Section 3 reports human-human Kappa of only 0.80/0.78, meaning a substantial portion of items (roughly 20% if percent agreement mirrored Kappa, though Kappa is not a proportion) were disagreements and thus excluded. This directly affects the central claim that GPT-4 reaches substantial agreement with human analysts: the measured agreement is with a filtered gold standard, not with human judgment on the full corpus. The paper's additional claim that LLMs 'can match or even exceed human performance' is contradicted by its own data, since human-human Kappa (0.80/0.78) exceeds the best LLM-human Kappa (0.738/0.734). This is not a rejection of the paper, because the Kappa values may still be plausible and the authors provide a public code link, but the central claim needs to be re-evaluated on the full resolved dataset. The reader's CONDITIONAL verdict is appropriate; my concern aligns with the reader's weakest_assumption, so no change to the verdict is required. I would, however, emphasize in the revision that the authors must report performance on all items (including initially disagreed-upon ones) and remove or correct the 'match or exceed human performance' sentence, as it is unsupported by the reported numbers.","tokens_in":10204,"tokens_out":3218,"duration_ms":31497,"concrete_test":"Recompute GPT-4's Cohen's Kappa (and accuracy) against the full resolved ground truth that includes all items, not just the consensus subset, using the unified labels produced after the analysts' discussion (Section 3). Also compute the same metrics separately on the subset of items where the two analysts initially disagreed. If the full-data Kappa remains above 0.70, the central claim survives; if it drops materially, the claim must be qualified as applying only to unambiguous requirements. Report the number and percentage of excluded items to assess the magnitude of the truncation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 states: 'we used only the labels on which both analysts reached consensus as the ground truth for evaluating LLM performance.' The two human analysts themselves achieved only 'substantial' agreement (Cohen's Kappa 0.80 for the Library Management System and 0.78 for the Smart Home System, Section 3). This means a nontrivial fraction of requirement items were initially disagreed upon; those items are likely the most ambiguous or hardest to label. By excluding them from the evaluation, the reported Kappa values (0.738 and 0.734 for GPT-4 few-shot, Table 4) measure agreement only on the easy subset of requirements, not on the full set of stakeholder inputs that a practical QDA tool would need to handle. The central claim—that GPT-4 achieves 'substantial agreement with human analysts' in deductive annotation—is therefore not established for the full dataset. Moreover, Section 5.3 claims LLMs 'can match or even exceed human performance,' but the human-human Kappa (0.80/0.78) is higher than the best LLM-human Kappa (0.738/0.734); this specific sentence is unsupported by the paper's own numbers. The authors flag external validity in Section 6.2, but they do not acknowledge this internal truncation of ground truth, which is a more direct threat to the quantitative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the use of large language models (LLMs) for qualitative data analysis (QDA) in requirements engineering, specifically for labeling/coding requirements statements. The authors map inductive QDA to zero-shot prompting and deductive QDA to one-shot and few-shot prompting, testing GPT-4, Mistral, and LLaMA-2 on two test cases (Library Management System and Smart Home System). Human analysts labeled the data independently and then reconciled their labels; Cohen's Kappa between them was 0.80 and 0.78. The LLMs were evaluated against the human consensus labels using Kappa, accuracy, precision, recall, F1, and consistency metrics (SD, ICC). The main finding is that GPT-4 in the few-shot setting achieves Kappa 0.738 and 0.734 for the two test cases, indicating substantial agreement with human analysts, while zero-shot performance is around 0.54. The paper also reports that longer, context-rich prompts improve performance, and that GPT-4 is more consistent across runs than the other models.","tokens_in":10474,"tokens_out":6964,"duration_ms":59439,"significance":"If the central result is valid, the paper provides useful evidence that a general-purpose LLM can serve as a supporting tool for deductive qualitative coding in RE, potentially reducing manual effort. The inclusion of a human-human baseline (Kappa 0.80/0.78) is a strength, as is the evaluation of multiple models and prompt configurations, and the public availability of the code. However, the evaluation's validity is undermined by the use of consensus-only ground truth, which likely excludes the most difficult items, and by an overstatement in Section 5.3 that LLMs \"can match or even exceed human performance\" despite lower Kappa than the human-human baseline. These issues need to be addressed before the significance claim can be fully accepted.","major_comments":[{"comment":"Section 5.1 states that \"we used only the labels on which both analysts reached consensus as the ground truth for evaluating LLM performance.\" Because the human analysts themselves achieved only substantial agreement (Kappa 0.80 for the Library Management System and 0.78 for the Smart Home System, Section 3), a nontrivial fraction of requirement items were initially disputed. Excluding those items likely removes the hardest, most ambiguous cases, so the reported Kappa values (e.g., 0.738 and 0.734 for GPT-4 few-shot in Table 4) measure agreement only on an easy subset rather than on the full set of stakeholder requirements. The paper does not report the number or proportion of excluded items, and it does not separately report performance on the full 'resolved' set of labels described in Section 3. The central claim that GPT-4 achieves substantial agreement with human analysts in deductive annotation is therefore not established for the full dataset. The authors should either evaluate on the full resolved ground truth or quantify the exclusion and show that the result is robust to it.","section":"5.1"},{"comment":"In Section 5.3 the paper claims that LLMs \"can match or even exceed human performance\" in deductive settings. This is contradicted by the paper's own numbers: the human-human Kappa is 0.80 and 0.78 (Section 3), while the best LLM-human Kappa is 0.738 and 0.734 (Table 4). A lower LLM-human Kappa than the human-human baseline does not support \"match or exceed.\" This sentence should be corrected to reflect that LLMs approach but do not reach human-human agreement, unless additional evidence is provided.","section":"5.3"},{"comment":"Section 6.1 states that prompts were \"refining them through an iterative process\" and that different lengths and context levels were tested. If the prompt templates and few-shot examples were selected after evaluating on the same test sets used for the reported results, the Kappa values may be optimistically biased by prompt overfitting. The paper should report whether a held-out development set was used to choose the prompt configuration, and how the few-shot examples were selected (e.g., randomly or based on performance). Without this, the internal validity of the reported agreement scores is in question.","section":"6.1"},{"comment":"The paper reports Cohen's Kappa in Table 4 without dataset sizes, category counts, or confidence intervals, making it impossible to assess whether the small numerical differences between models (e.g., 0.738 vs. 0.734 for GPT-4) are meaningful. Additionally, Section 5.5 applies Intraclass Correlation Coefficient (ICC) and standard deviation to categorical label outputs in Table 6; ICC is designed for continuous measurements, and the standard deviation of categorical labels depends on an arbitrary numeric encoding. The paper should either justify the use of ICC for this categorical data or use an appropriate categorical consistency metric, and should report the number of items and uncertainty intervals for the Kappa estimates.","section":"5.1 and 5.5"}],"minor_comments":[{"comment":"The conclusion refers to 'LLM' as a model name (\"specifically LLM, Mistral, and LLaMA-2\"); this should be 'GPT-4'.","section":"7"},{"comment":"The caption of Table 4 and the surrounding text should specify the exact prompt configuration (length and context level) used for the reported Kappa scores, since Table 5 shows variation by prompt length and Table 7 shows variation by context level.","section":"Table 4"},{"comment":"The abstract says \"Cohen's Kappa scores exceeding 0.7\", but the introduction says \"up to 0.738\"; clarify whether the claim is about the point estimates exceeding 0.7 in both test cases.","section":"Abstract"},{"comment":"The phrase \"approaching human performance\" is used before the overclaim; consider using consistent language throughout the results section.","section":"5.3"},{"comment":"The internal validity section does not discuss the consensus-only ground truth issue; this should be acknowledged as a threat to internal validity.","section":"6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the overall idea is promising, but the main quantitative claim rests on a potentially biased ground truth. The explicit statement in Section 5.1 makes this easy to verify. Please press the authors to provide a full-set analysis or a quantitative justification for the exclusion, and to correct the \"match or exceed\" overclaim. If these are addressed, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read: this is a generally sound empirical study that reports new measurements—Cohen's Kappa, precision, recall, F1 for GPT-4, Mistral, and LLaMA-2 on two RE datasets across zero/one/few-shot prompting. That alone is worth something. The human baseline is provided, the setup is described clearly, and there's a public code link. Credit where due: the few-shot design with context-rich prompts is sensible, and the result that GPT-4 lands around 0.73-0.74 Kappa is in the right ballpark for deductive coding with off-the-shelf LLMs.\n\nThe soft spots are real, though. The stress-test concern lands: Section 5.1 says they evaluate only on labels where both analysts reached consensus. Given the analysts themselves only hit 0.80/0.78 Kappa, a nontrivial chunk of items were disputed—likely the hard ones. So the reported LLM-human Kappas measure agreement on the easy subset, not the full requirement set. That directly weakens the central claim. The authors flag external validity but never mention this internal truncation, which is more damaging.\n\nAlso, Section 5.3 says LLMs 'can match or even exceed human performance.' The paper's own numbers contradict that: human-human Kappa is 0.80/0.78, while best LLM-human is 0.738/0.734. That's an unsupported overclaim. Minor but annoying: ICC on categorical labels is used without justification, and no dataset sizes or confidence intervals are given anywhere.\n\nThese flaws are fixable. The right path is not rejection but major revision: report per-item agreement and re-run evaluation on the full resolved set, add error bars, and correct the overclaim. The core phenomenon—LLMs can support deductive annotation in RE—is probably robust; the exact numbers just aren't trustworthy as presented.\n\nWho gets value? RE researchers and practitioners considering LLM-assisted coding. It deserves a serious referee, but the referee should push for the full-data analysis before publication.\n\nRecommendation: send to peer review with a request for major revisions. I wouldn't cite it in its current form, but I'd revisit after the authors address the ground-truth issue.","headline":"Useful empirical measurements, but the consensus-only ground truth likely inflates the headline Kappa values, and the paper overclaims by saying LLMs can match or exceed human performance.","tokens_in":10966,"tokens_out":1352,"would_cite":false,"duration_ms":16032,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4, prompted with a few labeled examples and detailed system context, reproduces human qualitative labels on software requirements at substantial agreement (Cohen's kappa up to 0.738), while zero-shot labeling lags.","keywords":["Requirements Engineering","Qualitative Data Analysis","Large Language Models","GPT-4","Few-shot Learning","Zero-shot Learning","Cohen's Kappa","Requirements Annotation"],"falsifier":"Take the same few-shot prompts to a third requirements document from a different domain, score the model against a panel of five analysts who label every requirement, including the ambiguous ones, and check whether Cohen's $\\kappa$ stays above roughly 0.70; if it falls substantially, the claimed agreement depends on the consensus-trimmed ground truth or on the two chosen systems.","tokens_in":9932,"feed_emoji":"🏷️","tokens_out":13094,"duration_ms":117299,"temperature":0.7,"pith_summary":"The paper aims to establish that a general-purpose LLM can carry out the labeling step of qualitative data analysis for requirements engineering at a level close to human judgment, provided it is given a few labeled examples and enough system context. Its headline evidence is GPT-4 reaching Cohen's $\\kappa$ of 0.738 and 0.734 in the few-shot setting on two test systems, which falls in the substantial-agreement band, while zero-shot agreement stays near 0.54. The paper also claims that this result is driven by prompt design: longer prompts and fuller system descriptions improve agreement, and GPT-4's labels are highly reproducible across runs. The reason to care is that this labeling step currently consumes dozens of analyst-hours on a single project, and the resulting labels feed directly into traceable domain models.","feed_headline":"GPT-4 labels requirements at near-human agreement with a few examples","feed_subtitle":"Few-shot prompts with full context push its Cohen's kappa above 0.73 on two software systems","key_machinery":"The carrying mechanism is the prompt template, varied along three independent dimensions: shot type (zero examples for inductive, one or several for deductive), prompt length (short, medium, long), and context level (none, some, full system description). These prompts turn qualitative coding into a constrained multi-class labeling task in which the model must output a single one-word category per requirement. The evaluation protocol then compares model labels against the consensus labels of two analysts using Cohen's $\\kappa$, accuracy, precision, recall, and F1, and checks repeated-run reliability with standard deviation and intraclass correlation; all models were run at temperature 0.0.","core_discovery":"The paper's central claim is that deductive annotation, implemented as one-shot and few-shot prompting, lets the tested LLMs approach human performance on qualitative coding of software requirements, whereas inductive zero-shot annotation does not; GPT-4 is the strongest of the three models. On the Library Management System, GPT-4 achieves Cohen's $\\kappa=0.738$ in the few-shot setting, and on the Smart Home System $\\kappa=0.734$, against a ground truth formed by the labels on which two human analysts agreed, whose own agreement was 0.80 and 0.78 respectively. The paper states that these scores indicate substantial agreement and that accuracy, precision, and recall are correspondingly high, for example accuracy 0.86 and F1 0.79 in the Library few-shot setting. It further claims that adding full contextual background and longer instructions is the main lever that moves scores up from the roughly 0.54 zero-shot baseline, and that GPT-4 shows higher run-to-run consistency (ICC 0.93 and 0.92) than the open models tested.","pith_inferences":["The paper does not test this, but the consensus-only ground truth makes the reported kappas an upper bound; a deployment that labels every requirement, including disputed ones, would likely show lower agreement.","The one-word-label setup is effectively multiclass classification, so calibration, confidence thresholds, or abstention options could raise the currently weak zero-shot performance.","The same few-shot prompt recipe could plausibly be reused for other coding tasks in requirements analysis, such as tagging ambiguity or stakeholder intent, since it only needs a few examples and a system description.","A further consequence the paper leaves implicit: once deductive annotation is cheap, the bottleneck of qualitative analysis shifts from labeling to constructing and validating the codebook, because the few-shot examples encode the categories."],"forward_implications":["A few-shot LLM could serve as a first-pass annotator for deductive coding, cutting the roughly 74 analyst-hours the paper reports for manually labeling two systems.","Structured labels produced automatically can be mapped to classes in a domain model, shortening the path from stakeholder input to a design model.","Teams wanting reliable annotations would need to invest in long, context-rich prompts with a few worked examples, since shorter or context-free prompts produce materially lower agreement.","Zero-shot, fully inductive annotation is not ready for autonomous use; the near-0.54 kappa scores put it below the threshold for dependable coding.","Run-to-run consistency above ICC 0.85 makes the labels reproducible enough for workflows that need auditing or repeatable outputs."],"supporting_citations":[{"why":"Supplies the GPT-4 model whose few-shot outputs form the main result.","marker":"[8]"},{"why":"Supplies the LLaMA-2 model used as a comparison system.","marker":"[10]"},{"why":"Establishes the few-shot prompting paradigm the deductive annotation setup borrows.","marker":"[14]"},{"why":"Provides the domain-modeling method that motivates mapping labels to classes.","marker":"[20]"},{"why":"Provides the public requirement documents used as the two test cases.","marker":"[27]"},{"why":"Supplies the Mistral model used as another comparison system.","marker":"[28]"},{"why":"Provides the Cohen's kappa convention used to judge rater agreement.","marker":"[30]"},{"why":"Demonstrates prior use of LLMs for deductive coding that this study extends to requirements engineering.","marker":"[31]"}],"fun_headline_variants":["Few-shot GPT-4 matches human coding of requirements with kappa above 0.7","LLMs in RE: GPT-4 excels with few examples, zero-shot falls short","Context-rich few-shot prompts lift GPT-4's requirement coding to near-human","GPT-4's few-shot annotation hits substantial agreement in requirements analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on treating the labels on which two analysts agreed, for just two software systems, as the correct labels for requirements annotation in general; if that consensus subset skips the hardest or most ambiguous requirements, the reported agreement overstates how the method would perform in real use.","fun_headline_variants_meta":{"raw":{"variants":["Few-shot GPT-4 matches human coding of requirements with kappa above 0.7","LLMs in RE: GPT-4 excels with few examples, zero-shot falls short","Context-rich few-shot prompts lift GPT-4's requirement coding to near-human","GPT-4's few-shot annotation hits substantial agreement in requirements analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000794,"raw_usage":{"total_tokens":3513,"prompt_tokens":976,"completion_tokens":2537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2451}},"tokens_in":592,"tokens_out":2537,"duration_ms":17308,"temperature":1.0,"reasoning_tokens":2451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:53:46.917223+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same few-shot prompts to a third requirements document from a different domain, score the model against a panel of five analysts who label every requirement, including the ambiguous ones, and check whether Cohen's $\\kappa$ stays above roughly 0.70; if it falls substantially, the claimed agreement depends on the consensus-trimmed ground truth or on the two chosen systems.","supporting_citations":[{"cited_title":"Kaufmann, D","cited_arxiv_id":null,"evidence_quote":"Provides the domain-modeling method that motivates mapping labels to classes."},{"cited_title":"Ferrari, G","cited_arxiv_id":null,"evidence_quote":"Provides the public requirement documents used as the two test cases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Cohen's kappa convention used to judge rater agreement."}],"review_version":1}