Pith. sign in

REVIEW 3 major objections 4 minor 77 references

Demystifying Feature Requests: Leveraging LLMs to Refine Feature Requests in Open-Source Software

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPT-4o, guided by a small corpus of human-annotated feature requests and in-context-learning prompts, can detect ambiguity and incompleteness in GitHub feature requests and generate clarification questions that largely match human-written…

desk verdict A useful, honest empirical study with a new annotated corpus and a transparent prompt pipeline; the headline incompleteness numbers are not settled because the ground truth is revised after seeing model output. read the letter →

arxiv 2507.13555 v1 pith:5TL62VVT submitted 2025-07-17 cs.SE

Pragyan KC , Rambod Ghandiparsi , Thomas Herron , John Heaps , Mitra Bokaei Hosseini This is my paper
classification cs.SE
keywords featurerequestsnaturallanguagedefectsambiguitydetectionincompletenessclarificationquestionslargemodelsin-contextlearningopen-sourcesoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a general-purpose large language model, used without fine-tuning, can serve as a practical first-pass editor for natural-language feature requests on open-source issue trackers. It claims GPT-4o can detect where a request is ambiguous or incomplete, classify the defect into an established ambiguity subclass or incompleteness, and generate clarification questions that address the defect. Using 100 annotated requests from Mastodon and Signal, it reports strong detection for lexical and syntactic ambiguity and high recall for incompleteness, with moderate precision caused by over-predicting missing information. Interviews with seven Signal developers confirm that ambiguity and incompleteness waste time and that asking clarifying questions is developers' first strategy, aligning the automation with real practice.

What carries the argument

The load-bearing machinery is a small, carefully annotated corpus: 100 feature requests from Mastodon and Signal, labeled by two annotators using a grounded-theory-derived heuristics document for ambiguity subclasses (lexical, syntactic, semantic, pragmatic, vagueness) and incompleteness, with reconciled interpretations and gold clarification questions. Around this corpus, the paper builds in-context-learning prompts for GPT-4o in three settings—zero-shot, few-shot with positive and negative paired demonstrations, and few-shot with reasoning—and evaluates defect detection with exact, coreferential, partial, and ROUGE matches, and clarification-question generation with cosine similarity plus manual qualitative review.

What would settle it

A reader could take the released prompts and corpus, recruit two new annotators to relabel the same 100 feature requests using only the published heuristics document, and compare the resulting ground truth to the published labels; if the defect labels shift substantially, the reported F1 values cannot be reproduced as stable measurements of LLM detection skill.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that LLMs can detect and classify defects in feature requests to a considerable extent and that LLM-generated clarification questions are highly effective at refining those defects. Concretely, GPT-4o with in-context learning identifies lexical and syntactic ambiguities well, performs less reliably on semantic and pragmatic ambiguity and on vagueness, and detects incompleteness with high recall but moderate precision, reaching 0.588 precision, 0.981 recall, and 0.735 F1 at six shots. In the manual evaluation of generated clarification questions, the model produced correct questions for 69 of 70 lexical, 17 of 17 syntactic, 6 of 7 semantic, 22 of 23 pragmatic, 10 of 10 vagueness, and 40 of 40 incompleteness test instances. The paper also finds that zero-shot or few-shot prompting without reasoning usually beats prompting that asks the model to reason, and that some LLM-flagged false positives for incompleteness look reasonable, suggesting the human annotators may have missed details worth asking about.

Load-bearing premise

The evaluation treats the reconciled human annotations as the correct ground truth for what counts as ambiguous or incomplete, even though the two annotators only agreed moderately, with Cohen's Kappa ranging from 0.40 to 0.87 across batches.

Editorial extensions

If this is right

  • OSS maintainers could use GPT-4o as a pre-triage filter that flags ambiguous or incomplete requests and attaches clarifying questions before a human reads them.
  • Few-shot examples do not reliably help ambiguity detection, and zero-shot prompts are often best, so deployment could be cheap and require only prompt engineering rather than model training.
  • For incompleteness, the model's tendency to over-predict missing details may be useful: it surfaces non-functional and privacy concerns that non-expert requesters omit, giving maintainers more than the human annotations caught.
  • A hybrid workflow follows directly from the paper's findings: the LLM pinpoints localized defects and structured questions, while human developers supply goal-level why reasoning, and combining the two would improve requirements elicitation.
  • Prompt design is a first-order factor in performance, since reasoning-style prompts initially hallucinate and improve only with more examples, so the prompt regime should be chosen per defect class.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The moderate inter-annotator agreement (Cohen's Kappa 0.40–0.87) implies the ground truth itself is one plausible reading; a second annotation team could yield different labels, so the reported F1 values should be read as performance against this specific corpus rather than as an absolute ceiling.
  • Because Mastodon and Signal are both messaging and social apps with UI-heavy requests, transfer to non-UI domains such as libraries, infrastructure, or scientific software is untested, and a replication on those repositories would likely change the defect mix.
  • The developers' preference for why questions suggests a cheap extension: prompt the LLM to ask goal-oriented questions before terminological clarification, which may align automation more closely with actual maintainer workflow.
  • The finding that the LLM surfaced missing information that annotators overlooked raises the possibility of using LLM detections to bootstrap improved annotation heuristics, making the human ground truth iterative rather than fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an LLM-based pipeline for detecting ambiguity and incompleteness in GitHub feature requests and for generating clarifying questions to refine those defects. The authors build a corpus of 100 annotated feature requests from Mastodon and Signal, evaluate GPT-4o under zero-shot, few-shot, and few-shot-with-reasoning settings, and compare model outputs against human annotations. They also interview seven Signal developers about how they perceive and handle such defects. The central claims are that LLMs can detect and classify defects in feature requests to a considerable extent and that LLM-generated clarification questions are highly effective.

Significance. If substantiated, the results would support a practical LLM-assisted front-end for OSS issue triage, which is a genuinely useful direction for requirements engineering in decentralized settings. The paper is transparent in publishing its repository, annotation tooling, prompts, and data, and it augments the quantitative evaluation with developer interviews, which is a strength. However, the current evidence for the central claims is weakened by dependence on an unstable ground truth, test-set-based selection of shots and seeds, and a manual evaluation of clarification questions without reported inter-annotator agreement. The contribution is ultimately defensible but needs substantial revision before the empirical claims can be accepted.

major comments (3)
  1. [Section VI and Table IV] The incompleteness precision and F1 reported in Table IV are not independent measurements of the LLM because the ground truth is revised after seeing model output. The paper states that of the 30 false positives in the 70 test cases, 26 are labeled complete in the ground truth but the model's predictions are 'reasonable,' suggesting 'annotators may have overlooked certain details.' If those 26 cases are true positives, precision would rise from the reported 0.571-0.588 to roughly 0.94, which would change the central RQ1 answer for incompleteness. Given that one annotation batch had an incompleteness Kappa of only 0.40 (Section III-B), the ground truth is not stable enough to arbitrate between the model and the original annotations. The authors should either formally re-adjudicate the disputed cases with a third annotator before computing metrics, or report both the original and the reconciled metrics explicitly and state which claim each supports.
  2. [Section V-A, Figures 4-5, Tables II-III] The shot counts and seeds are selected using test-set performance, which inflates the reported results and weakens the RQ2 comparison of prompting methods. The text notes that the F1 scores in Figures 4 and 5 correspond to the 'best-performing seed in each setting,' and the preferred shot counts (e.g., zero-shot for most ambiguity subclasses, 2-shot for lexical, 6-shot for reasoning) are identified from the test curves rather than from a held-out validation set. This makes the claim that 'the prompting method significantly impacts the model's performance' an exploratory finding rather than a validated comparison. The authors should use a nested or repeated validation scheme, report means and variance across all seeds rather than only the best seed, and either drop or clearly label the test-set-selected shot counts as preliminary.
  3. [Section IV-B, Section V-B, Table VI] The manual evaluation of generated clarification questions, which underpins the RQ3/RQ4 claim that the questions are 'highly effective,' is performed by two authors without any reported inter-annotator agreement or a detailed rubric. Table VI reports near-perfect counts of correct CQs (e.g., 69/70 for lexical ambiguity and 40/40 for incompleteness), but the reader cannot assess how reliable these judgments are because no Kappa or other agreement statistic is reported for the CQ evaluation, and the criteria described in Section IV-B are only two broad checks. The authors should report an agreement measure for the manual CQ evaluation, provide a concrete rubric or annotation guide, and ideally include developer ratings of a sample of generated CQs.
minor comments (4)
  1. [Figure 1 and Listing 6] There are copyediting errors: Figure 1 says 'Web Scrapped' instead of 'Web Scraped,' and Listing 6 is titled 'Incompletensss Detection' instead of 'Incompleteness Detection.'
  2. [Section V-B] The sentence 'Vagueness and incompleteness attain their highest instance-level F1 score at 4-shot' is confusing because Figure 6 is described as showing Cosine Similarity scores; please clarify whether the reported quantity is F1 or cosine similarity for those classes.
  3. [Section IV-C] There are minor grammatical errors in the interview description, including 'Each interview session include' and 'The interviews last an average of 42 minutes,' which should be 'includes' and 'lasted.'
  4. [Section VII] The threats-to-validity section states that each prompt is repeated 10 times and the average F1 is reported, but Tables II, III, and IV report individual seed values without averages or variance; please align the text with the actual reporting.

Circularity Check

1 steps flagged · score 4.0 of 10

Reported incompleteness precision is not independent: Section VI reclassifies 26 of 30 model false positives as reasonable, using the model to arbitrate the ground truth it is scored against.

  1. other [Section VI (Discussion), first paragraph answering RQ1, incompleteness discussion; cf. Table IV in Section V-A]
    "Of the remaining 30 requests, 26 are labeled as complete in the GT, but the model classifies them as incomplete, generating missing information. Upon review, we find the model’s predictions reasonable, suggesting annotators may have overlooked certain details."

    Precision in Table IV is computed against the human GT described in Section IV-A. After seeing GPT-4o's output, the authors declare 26 of its 30 disagreements with that GT 'reasonable' and attribute the disagreement to annotator oversight, without updating the GT or recalculating Table IV. This makes the model the arbiter of the benchmark on which it is measured: a false positive is effectively redefined as a true positive whenever the authors agree with the model. If those 26 cases were accepted as true positives, incompleteness precision would rise from 0.571-0.588 to roughly 66/70 = 0.94, so the reported 'moderate precision' is not a fixed, independent measurement of LLM performance.

full rationale

The core evaluation is a standard supervised comparison: an LLM is prompted to label feature requests, and its outputs are scored against a human-annotated corpus constructed in Step 2. That design is not circular in itself, and the 30/70 split prevents the few-shot examples from being scored on the same items used to build them. The circularity is confined to the incompleteness discussion in Section VI. There, the ground truth is treated as revisable after the model's output is seen: 26 of 30 false positives are called reasonable, and annotator oversight is inferred from the fact that the model disagreed. Because the same authors created both the GT and the post hoc 'reasonable' judgment, the evaluation criterion for those cases reduces to agreement with the model. Table IV is left unchanged, so the published 0.571-0.588 precision does not reflect the discussion's own reclassification. This is a localized circularity rather than a derivation-level one: it concerns the interpretation of one reported result, not the construction of the method or the CQ-generation claim. The ambiguity detection results, the cosine-similarity checks, and the developer interviews remain anchored to external or independent judgments. Score 4 reflects one partially self-confirming evaluation step in a broader empirical study whose central method does not reduce to its inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

All empirical claims rest on the annotated corpus, the taxonomy, and the assumption that GPT-4o stands in for LLMs generally. No new physical or formal entities are introduced; the only new artifacts are the dataset and prompt designs, which are not entities in the ledger sense. The main burden is the moderate inter-annotator agreement and the absence of a held-out model-selection protocol.

free parameters (3)
  • Number of in-context examples (shots) = 0, 2, 4, 6; best per defect class selected
    The best shot count is chosen from test-set F1, not a separate validation set, which makes reported performance optimistic.
  • Random seeds for data splits = 10, 20, 45
    Results vary across seeds; figures highlight the best-performing seed, and no seed averaging or significance testing is reported.
  • Training and test split ratio = 30/70
    Fixed split ratio chosen to balance few-shot training examples and test size; it is not tuned but influences variance.
assumptions (5)
  • domain assumption Issues labeled 'Feature Request' or 'Feature' on GitHub comprise genuine feature requests.
    Used in Section III-A to build the 476-request dataset; the paper later acknowledges some bug reports are mislabeled (Section VII).
  • domain assumption Berry et al.'s five ambiguity subclasses are an appropriate annotation scheme.
    Adopted in Section III-B to ensure consistency with prior work; subclasses may not map cleanly to user-generated OSS text.
  • domain assumption Annotators' reconciled labels are valid ground truth.
    The whole evaluation in Section IV-A depends on this; Cohen's Kappa is moderate, e.g., 0.40 for incompleteness in batch 2.
  • domain assumption GPT-4o represents LLMs generally.
    Only GPT-4o is tested but conclusions refer to LLMs throughout Sections V and VIII.
  • domain assumption Seven Signal developers' interview responses are representative of OSS developer perceptions.
    Interview findings in Section V-C are used to validate the approach; sample size is small and project-specific.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Demystifying Feature Requests: Leveraging LLMs to Refine Feature Requests in Open-Source Software." pith.science (2026). https://pith.science/paper/5TL62VVT

@misc{pith2026250713555,
  author       = {Pith},
  title        = {Pith review of: Demystifying Feature Requests: Leveraging LLMs to Refine Feature Requests in Open-Source Software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5TL62VVT}},
  note         = {Machine review of arXiv:2507.13555}
}
read the original abstract

The growing popularity and widespread use of software applications (apps) across various domains have driven rapid industry growth. Along with this growth, fast-paced market changes have led to constantly evolving software requirements. Such requirements are often grounded in feature requests and enhancement suggestions, typically provided by users in natural language (NL). However, these requests often suffer from defects such as ambiguity and incompleteness, making them challenging to interpret. Traditional validation methods (e.g., interviews and workshops) help clarify such defects but are impractical in decentralized environments like open-source software (OSS), where change requests originate from diverse users on platforms like GitHub. This paper proposes a novel approach leveraging Large Language Models (LLMs) to detect and refine NL defects in feature requests. Our approach automates the identification of ambiguous and incomplete requests and generates clarification questions (CQs) to enhance their usefulness for developers. To evaluate its effectiveness, we apply our method to real-world OSS feature requests and compare its performance against human annotations. In addition, we conduct interviews with GitHub developers to gain deeper insights into their perceptions of NL defects, the strategies they use to address these defects, and the impact of defects on downstream software engineering (SE) tasks.

Figures

Figures reproduced from arXiv: 2507.13555 by the authors.

Figure 1
Figure 1. Approach Overview ground truth (GT) for identifying and clarifying defects in the dataset through human annotations. In the third step, we utilize LLMs to detect NL defects. Finally, in step 4, we generate CQs to refine feature requests with defects. A. Step 1: Feature Requests Collection The goal of Step 1 is to create a feature request dataset. To this end, we select Mastodon1 and Signal2 , two well-known OSS proj… view at source ↗
Figure 2
Figure 2. Annotation Tool to define a display limit and determine what happens when the limit is reached. We also analyze the 15 requests to determine whether any of their comments addressed the NL defects identified during our annotations [20]. We observe mentions of difficulty in implementing the feature request, expressing the usefulness of the requested feature in solving an issue, requesting additional features, reportin… view at source ↗
Figure 3
Figure 3. Sampled Feature Requests for the Interviews [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Rouge-L F-1 Scores for Experiment 1.2 general, the interviewees perceive ambiguity and incomplete￾ness as challenges that result in wasted time, inefficiency, multiple ways for implementation, missing implementation details, overlooked edge cases, and unspecified use c…
Figure 6
Figure 6. Figure 6: Cosine Similarity Scores for CQs incompleteness. Table VII summarizes the key questions pro￾vided by the interviewees. TABLE VII: Sample CQs From the Interviews Topic Questions Understanding the Context and Need How did this need arise? Why do you want this feature? Wh…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 58 canonical work pages

  1. [1]

    Analysis of user comments: An approach for software requirements evolution,

    L. V . G. Carre ˜no and K. Winbladh, “Analysis of user comments: An approach for software requirements evolution,” in 2013 35th Interna- tional Conference on Software Engineering (ICSE) . IEEE, 2013, pp. 582–591

    work page 2013

  2. [2]

    Fame: supporting continuous requirements elicitation by combining user feedback and monitoring,

    M. Oriol, M. Stade, F. Fotrousi, S. Nadal, J. Varga, N. Seyff, A. Abello, X. Franch, J. Marco, and O. Schmidt, “Fame: supporting continuous requirements elicitation by combining user feedback and monitoring,” in 2018 Ieee 26th International Requirements Engineering Conference (RE). IEEE, 2018, pp. 217–227

    work page 2018

  3. [3]

    Re-swot: From user feedback to require- ments via competitor analysis,

    F. Dalpiaz and M. Parente, “Re-swot: From user feedback to require- ments via competitor analysis,” in International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 2019, pp. 55–70

    work page 2019

  4. [4]

    What would users change in my app? summarizing app reviews for recommending software changes,

    A. Di Sorbo, S. Panichella, C. V . Alexandru, J. Shimagaki, C. A. Visaggio, G. Canfora, and H. C. Gall, “What would users change in my app? summarizing app reviews for recommending software changes,” in Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering , 2016, pp. 499–510

    work page 2016

  5. [5]

    Bug report, feature request, or simply praise? on automatically classifying app reviews,

    W. Maalej and H. Nabil, “Bug report, feature request, or simply praise? on automatically classifying app reviews,” in 2015 IEEE 23rd International Requirements Engineering Conference (RE). IEEE, 2015, pp. 116–125

    work page 2015

  6. [6]

    Identification and classification of requirements from app user reviews

    H. Yang and P. Liang, “Identification and classification of requirements from app user reviews.” in SEKE, 2015, pp. 7–12

    work page 2015

  7. [7]

    From contract drafting to software specification: Linguistic sources of ambiguity,

    D. M. Berry, E. Kamsties, and M. M. Krieger, “From contract drafting to software specification: Linguistic sources of ambiguity,” Citeseer, 2003

    work page 2003

  8. [8]

    On the interplay between consistency, completeness, and correctness in requirements evolution,

    D. Zowghi and V . Gervasi, “On the interplay between consistency, completeness, and correctness in requirements evolution,” Information and Software Technology, vol. 45, no. 14, pp. 993–1009, 2003

    work page 2003

Show all 77 references
  1. [9]

    Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice,

    D. M. Fern ´andez, S. Wagner, M. Kalinowski, M. Felderer, P. Mafra, A. Vetr `o, T. Conte, M.-T. Christiansson, D. Greer, and C. Lassenius, “Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice,” Empirical Software Engineering, vol...

    2017

  2. [10]

    Software defect reduction top 10 list,

    B. Boehm and V . R. Basili, “Software defect reduction top 10 list,” Computer, vol. 34, no. 1, pp. 135–137, 2001

    2001

  3. [11]

    From ideas to expressed needs: an empirical study on the evolution of requirements during elicitation,

    S. Debnath, P. Spoletini, and A. Ferrari, “From ideas to expressed needs: an empirical study on the evolution of requirements during elicitation,” in 2021 IEEE 29th International Requirements Engineering Conference (RE). IEEE, 2021, pp. 233–244

    2021

  4. [12]

    The prevalence and severity of persistent ambiguity in software requirements specifications: Is a special effort needed to find them?

    C. Ribeiro and D. Berry, “The prevalence and severity of persistent ambiguity in software requirements specifications: Is a special effort needed to find them?” Science of Computer Programming , vol. 195, p. 102472, 2020

    2020

  5. [13]

    Two case studies of open source software development: Apache and mozilla,

    A. Mockus, R. T. Fielding, and J. D. Herbsleb, “Two case studies of open source software development: Apache and mozilla,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 11, no. 3, pp. 309–346, 2002

    2002

  6. [14]

    How do open source software (oss) developers practice and perceive requirements engineering? an empir- ical study,

    J. Kuriakose and J. Parsons, “How do open source software (oss) developers practice and perceive requirements engineering? an empir- ical study,” in 2015 IEEE Fifth International Workshop on Empirical Requirements Engineering (EmpiRE) . IEEE, 2015, pp. 49–56

    2015

  7. [15]

    Free/libre open- source software development: What we know and what we do not know,

    K. Crowston, K. Wei, J. Howison, and A. Wiggins, “Free/libre open- source software development: What we know and what we do not know,” ACM Computing Surveys (CSUR) , vol. 44, no. 2, pp. 1–35, 2008

    2008

  8. [16]

    A framework analysis of the open source development paradigm, 2000

    J. Feller and B. Fitzgerald, “A framework analysis of the open source development paradigm, 2000.”

    2000

  9. [17]

    Motivation, governance, and the viability of hybrid forms in open source software development,

    S. K. Shah, “Motivation, governance, and the viability of hybrid forms in open source software development,” Management science , vol. 52, no. 7, pp. 1000–1014, 2006

    2006

  10. [18]

    The social structure of free and open source software development,

    K. Crowston and J. Howison, “The social structure of free and open source software development,” 2005

    2005

  11. [19]

    Inquiry-based requirements analysis,

    C. Potts, K. Takahashi, and A. I. Anton, “Inquiry-based requirements analysis,” IEEE Software, vol. 11, no. 2, pp. 21–32, 1994

    1994

  12. [20]

    Data and resources,

    “Data and resources,” https://github.com/SoftEngineering11/RE2025- Repository, 2025

    2025

  13. [21]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  14. [22]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang et al. , “A survey on evaluation of large language models,” ACM transactions on intelligent systems and technology , vol. 15, no. 3, pp. 1–45, 2024

    2024

  15. [23]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, and Z. Dong, “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  16. [24]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

    2019

  17. [25]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–79, 2024

    2024

  18. [26]

    Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

    P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023

    2023

  19. [27]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  20. [28]

    A survey of prompt engineering meth- ods in large language models for different nlp tasks,

    S. Vatsal and H. Dubey, “A survey of prompt engineering meth- ods in large language models for different nlp tasks,” arXiv preprint arXiv:2407.12994, 2024

    2024 arXiv

  21. [29]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” arXiv preprint arXiv:2203.02155, 2022

    2022 arXiv

  22. [30]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, and A. Askell, “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  23. [31]

    A survey on in-context learning,

    Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu et al., “A survey on in-context learning,” arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  24. [32]

    True few-shot learning with language models,

    E. Perez, D. Kiela, and K. Cho, “True few-shot learning with language models,” Advances in neural information processing systems , vol. 34, pp. 11 054–11 070, 2021

    2021

  25. [33]

    C- icl: contrastive in-context learning for information extraction,

    Y . Mo, J. Liu, J. Yang, Q. Wang, S. Zhang, J. Wang, and Z. Li, “C- icl: contrastive in-context learning for information extraction,” arXiv preprint arXiv:2402.11254, 2024

    2024 arXiv

  26. [34]

    In-context example selection with influences,

    T. Nguyen and E. Wong, “In-context example selection with influences,” arXiv preprint arXiv:2302.11042 , 2023

    2023 arXiv

  27. [35]

    Requirements satisfiability with in-context learning,

    S. Santos, T. Breaux, T. Norton, S. Haghighi, and S. Ghanavati, “Requirements satisfiability with in-context learning,” arXiv preprint arXiv:2404.12576, 2024

    2024 arXiv

  28. [36]

    Pohl, Requirements engineering: An overview

    K. Pohl, Requirements engineering: An overview . Citeseer, 1996

    1996

  29. [37]

    Using domain-specific corpora for improved handling of ambiguity in requirements,

    S. Ezzini, S. Abualhaija, C. Arora, M. Sabetzadeh, and L. C. Briand, “Using domain-specific corpora for improved handling of ambiguity in requirements,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 1485–1497

    2021

  30. [38]

    Ambiguity in requirements specification,

    D. M. Berry and E. Kamsties, “Ambiguity in requirements specification,” in Perspectives on software requirements. Springer, 2004, pp. 7–44

    2004

  31. [39]

    Verifying and validating software requirements and design specifications,

    B. W. Boehm, “Verifying and validating software requirements and design specifications,” IEEE software, vol. 1, no. 1, p. 75, 1984

    1984

  32. [40]

    Ambiguity in Natural Language Software Requirements: A Case Study,

    F. de Bruijn and H. L. Dekkers, “Ambiguity in Natural Language Software Requirements: A Case Study,” in Requirements Engineering: Foundation for Software Quality: 16th International Working Confer- ence, REFSQ 2010, Essen, Germany, June 30–July 2, 2010. Proceedings

    2010

  33. [41]

    Springer, 2010, pp. 233–247

    2010

  34. [42]

    Requirement ambiguity not as important as expected—results of an empirical evaluation,

    E. J. Philippo, W. Heijstek, B. Kruiswijk, M. R. Chaudron, and D. M. Berry, “Requirement ambiguity not as important as expected—results of an empirical evaluation,” in Requirements Engineering: Foundation for Software Quality: 19th International Working Conference, REFSQ 2013,...

    2013

  35. [43]

    A framework for quality assessment of just-in-time requirements: the case of open source feature requests,

    P. Heck and A. Zaidman, “A framework for quality assessment of just-in-time requirements: the case of open source feature requests,” Requirements Engineering, vol. 22, pp. 453–473, 2017

    2017

  36. [44]

    Rule-based nlp vs chatgpt in ambiguity detection, a preliminary study,

    A. Fantechi, S. Gnesi, and L. Semini, “Rule-based nlp vs chatgpt in ambiguity detection, a preliminary study,” 2023

    2023

  37. [45]

    Automated handling of anaphoric ambiguity in requirements: A multi-solution study,

    S. Ezzini, S. Abualhaija, C. Arora, and M. Sabetzadeh, “Automated handling of anaphoric ambiguity in requirements: A multi-solution study,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 187–199

    2022

  38. [46]

    Addressing lexical and semantic ambiguity in natural language requirements,

    F. Zait and N. Zarour, “Addressing lexical and semantic ambiguity in natural language requirements,” in 2018 Fifth International Symposium on Innovation in Information and Communication Technology (ISIICT) . IEEE, 2018, pp. 1–7

    2018

  39. [47]

    Rapid quality assurance with requirements smells,

    H. Femmer, D. M. Fern ´andez, S. Wagner, and S. Eder, “Rapid quality assurance with requirements smells,” Journal of Systems and Software , vol. 123, pp. 190–213, 2017

    2017

  40. [48]

    Detecting requirements defects with nlp patterns: an industrial experience in the railway domain,

    A. Ferrari, G. Gori, B. Rosadini, I. Trotta, S. Bacherini, A. Fantechi, and S. Gnesi, “Detecting requirements defects with nlp patterns: an industrial experience in the railway domain,” Empirical Software Engineering , vol. 23, pp. 3684–3733, 2018

    2018

  41. [49]

    Automatic detection of nocuous coordination ambiguities in natural language requirements,

    H. Yang, A. Willis, A. De Roeck, and B. Nuseibeh, “Automatic detection of nocuous coordination ambiguities in natural language requirements,” in Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering , 2010, pp. 53–62

    2010

  42. [50]

    Ambiguity detection: Towards a tool explaining ambiguity sources,

    B. Gleich, O. Creighton, and L. Kof, “Ambiguity detection: Towards a tool explaining ambiguity sources,” in Requirements Engineering: Foun- dation for Software Quality: 16th International Working Conference, REFSQ 2010, Essen, Germany, June 30–July 2, 2010. Proceedings 16 . Sp...

    2010

  43. [51]

    Nero: A text-based tool for content annotation and detection of smells in feature requests,

    F. Mu, L. Shi, W. Zhou, Y . Zhang, and H. Zhao, “Nero: A text-based tool for content annotation and detection of smells in feature requests,” in 2020 IEEE 28th International Requirements Engineering Conference (RE). IEEE, 2020, pp. 400–403

    2020

  44. [52]

    Detecting bad smells in use case de- scriptions,

    Y . Seki, S. Hayashi, and M. Saeki, “Detecting bad smells in use case de- scriptions,” in 2019 IEEE 27th International Requirements Engineering Conference (RE). IEEE, 2019, pp. 98–108

    2019

  45. [53]

    An empirical study on the potential usefulness of domain models for completeness checking of requirements,

    C. Arora, M. Sabetzadeh, and L. C. Briand, “An empirical study on the potential usefulness of domain models for completeness checking of requirements,” Empirical Software Engineering , vol. 24, pp. 2509– 2539, 2019

    2019

  46. [54]

    Improving requirements completeness: Automated assistance through large language models,

    D. Luitel, S. Hassani, and M. Sabetzadeh, “Improving requirements completeness: Automated assistance through large language models,” Requirements Engineering, vol. 29, no. 1, pp. 73–95, 2024

    2024

  47. [55]

    Reqcompletion: Domain-enhanced automatic completion for software requirements,

    X. Lian, J. Ma, H. Lv, and L. Zhang, “Reqcompletion: Domain-enhanced automatic completion for software requirements,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 142–154

    2024

  48. [56]

    Automated smell detection and recommendation in natural language requirements,

    A. Veizaga, S. Y . Shin, and L. C. Briand, “Automated smell detection and recommendation in natural language requirements,” IEEE Transactions on Software Engineering , 2024

    2024

  49. [57]

    Free/libre open-source software development: What we know and what we do not know,

    K. Crowston, K. Wei, J. Howison, and A. Wiggins, “Free/libre open-source software development: What we know and what we do not know,” ACM Comput. Surv. , vol. 44, no. 2, Mar. 2008. [Online]. Available: https://doi.org/10.1145/2089125.2089127

    2008

  50. [58]

    An analysis of requirements evolution in open source projects: Recommendations for issue trackers,

    P. Heck and A. Zaidman, “An analysis of requirements evolution in open source projects: Recommendations for issue trackers,” in Proceedings of the 2013 International workshop on principles of software evolution , 2013, pp. 43–52

    2013

  51. [59]

    Towards utility-based prioritization of requirements in open source environments,

    A. Felfernig, M. Stettinger, M. Atas, R. Samer, J. Nerlich, S. Scholz, J. Tiihonen, and M. Raatikainen, “Towards utility-based prioritization of requirements in open source environments,” in 2018 IEEE 26th International Requirements Engineering Conference (RE). IEEE, 2018, pp. 406–411

    2018

  52. [60]

    How do open source software (oss) developers practice and perceive requirements engineering? an empir- ical study,

    J. Kuriakose and J. Parsons, “How do open source software (oss) developers practice and perceive requirements engineering? an empir- ical study,” in 2015 IEEE Fifth International Workshop on Empirical Requirements Engineering (EmpiRE) , 2015, pp. 49–56

    2015

  53. [61]

    Social networking meets software development: Perspectives from github, msdn, stack exchange, and topcoder,

    A. Begel, J. Bosch, and M.-A. Storey, “Social networking meets software development: Perspectives from github, msdn, stack exchange, and topcoder,” IEEE Software, vol. 30, no. 1, pp. 52–66, 2013

    2013

  54. [62]

    Using bug descriptions to reformulate queries during text-retrieval-based bug localization,

    O. Chaparro, J. M. Florez, and A. Marcus, “Using bug descriptions to reformulate queries during text-retrieval-based bug localization,” Empirical Software Engineering , vol. 24, pp. 2947–3007, 2019

    2019

  55. [63]

    Com- bining language and app ui analysis for the automated assessment of bug reproduction steps,

    J. Mahmud, A. Saha, O. Chaparro, K. Moran, and A. Marcus, “Com- bining language and app ui analysis for the automated assessment of bug reproduction steps,” arXiv preprint arXiv:2502.04251 , 2025

    2025 arXiv

  56. [64]

    Who should fix this bug?

    J. Anvik, L. Hiew, and G. C. Murphy, “Who should fix this bug?” in Proceedings of the 28th international conference on Software engineer- ing, 2006, pp. 361–370

    2006

  57. [65]

    What makes a good bug report?

    N. Bettenburg, S. Just, A. Schr ¨oter, C. Weiss, R. Premraj, and T. Zim- mermann, “What makes a good bug report?” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, 2008, pp. 308–318

    2008

  58. [66]

    SWE-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770 , 2023

    2023 arXiv

  59. [67]

    Basics of qualitative research: Techniques and procedures for developing grounded theory

    J. Corbin and A. Strauss, “Basics of qualitative research: Techniques and procedures for developing grounded theory.” Thousand Oaks, CA: Sage, 2008

    2008

  60. [68]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement , vol. 20, no. 1, pp. 37–46, 1960

    1960

  61. [69]

    Model generation with LLMs: From requirements to UML sequence diagrams,

    A. Ferrari, S. Abualhaijal, and C. Arora, “Model generation with LLMs: From requirements to UML sequence diagrams,” in 2024 IEEE 32nd International Requirements Engineering Conference Workshops (REW) . IEEE, 2024, pp. 291–300

    2024

  62. [70]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, and S. Brahma, “Scaling instruction-finetuned language models,” Journal of Machine Learning Research , vol. 25, no. 70, pp. 1–53, 2024

    2024

  63. [71]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, and A. Clark, “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  64. [72]

    Clam: Selective clarification for ambiguous questions with generative language models,

    L. Kuhn, Y . Gal, and S. Farquhar, “Clam: Selective clarification for ambiguous questions with generative language models,” arXiv preprint arXiv:2212.07769, 2022

    2022 arXiv

  65. [73]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys , vol. 55, no. 12, pp. 1–38, 2023

    2023

  66. [74]

    Revisiting automatic evalu- ation of extractive summarization task: Can we do better than rouge?

    M. Akter, N. Bansal, and S. K. Karmaker, “Revisiting automatic evalu- ation of extractive summarization task: Can we do better than rouge?” in ACL 2022, 2022, pp. 1547–1560

    2022

  67. [75]

    Toward regulatory compliance: A few-shot learning approach to extract processing activities,

    K. Pragyan, R. Ghandiparsi, R. Slavin, S. Ghanavati, T. Breaux, and M. B. Hosseini, “Toward regulatory compliance: A few-shot learning approach to extract processing activities,” in 2024 IEEE 32nd Interna- tional Requirements Engineering Conference Workshops (REW). IEEE, 2024,...

    2024

  68. [76]

    A large language model approach to code and privacy policy alignment,

    G. Morales, K. Pragyan, S. Jahan, M. B. Hosseini, and R. Slavin, “A large language model approach to code and privacy policy alignment,” in 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2024, pp. 79–90

    2024

  69. [77]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems , vol. 35, pp. 24 824–24 837, 2022

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.