REVIEW 3 major objections 4 minor 37 references
A RAG system that simulates novice-to-expert domain knowledge detects pragmatic ambiguities in natural-language requirements and produces candidate clarifications.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 19:11 UTC pith:DI6LJX2K
load-bearing objection Solid, reproducible RAG pipeline for pragmatic ambiguity in RE; the Wikipedia-as-expertise proxy is the soft spot, but the work is still worth a referee. the 3 major comments →
A Retrieval-Augmented Framework for Detecting and Resolving Pragmatic Ambiguities in Natural Language Requirements
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Retrieval-augmented generation over three domain knowledge bases that approximate increasing levels of stakeholder expertise can detect pragmatic ambiguities in natural-language requirements by measuring interpretation divergence and can generate candidate disambiguated requirements that domain experts judge relevant, clear and consistent with intended system functionality.
What carries the argument
Three graded domain knowledge bases (KN, KI, KE) built by WikiDoMiner at Wikipedia depths 0/1/2, combined with LLM-generated elucidation questions and pairwise cosine-similarity thresholding of the top retrieved chunks, form the detection engine; the expert base plus a controlled rewrite prompt then supplies candidate resolutions.
Load-bearing premise
Wikipedia articles retrieved at successive category depths are good enough proxies for the real differences in domain knowledge among novice, intermediate and expert stakeholders.
What would settle it
Annotate a fresh set of requirements from a non-transportation domain with the same expert protocol; if the method’s macro F2 drops below 0.5 while a simple lexical-ambiguity baseline remains higher, the claim that graded Wikipedia retrieval reliably surfaces pragmatic ambiguity is falsified.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a retrieval-augmented generation (RAG) framework that detects pragmatic ambiguities in natural-language requirements by generating elucidation questions (EQs) for potentially ambiguous terms, then comparing interpretations retrieved from three Wikipedia-derived domain knowledge bases (novice/intermediate/expert, constructed via WikiDoMiner depths 0/1/2). A requirement is flagged ambiguous if the requirements document itself does not answer the EQs and no triple of top-3 chunks from the three KBs has all pairwise cosine similarities above a calibrated threshold T. Candidate resolutions are generated from the expert KB plus the requirements KB and are intended for requirements-analyst validation. Evaluation on two PURE transportation documents (Clarus, VII DUAP) with four LLMs reports macro F2 up to 0.75 for detection and human ratings (relevance/clarity/consistency) for resolutions; a replication package is released.
Significance. If the operationalization holds, the work supplies a practical, LLM-based recommender that surfaces expertise-dependent misreadings of requirements—an underexplored class of ambiguity—while producing candidate rewrites that domain experts rate as relevant, clear and consistent. Strengths include a publicly released replication package (ground truth, code, outputs), transparent multi-model evaluation, high inter-annotator agreement on labels (κ=0.89) and resolution scores (weighted κ=0.77), explicit error-case analysis, and alignment of resolution metrics with ISO/IEC/IEEE 29148 quality characteristics. The approach usefully generalizes prior graph- and embedding-based pragmatic-ambiguity detectors by Ferrari et al. by treating cross-domain stakeholders as novices within a single graded-expertise continuum.
major comments (3)
- [§§3.1.2, 3.4] §§3.1.2 and 3.4 operationalize pragmatic ambiguity as pairwise cosine-similarity divergence among the top-3 chunks retrieved from KN/KI/KE (WikiDoMiner depths 0/1/2). The manuscript states that the intent is only to capture relative differences, yet the central claim is that the pipeline “simulates stakeholders with varying domain expertise.” No direct evidence is supplied that depth expansion systematically produces expertise-correlated differences that matter for the generated EQs; end-to-end macro-F2 (≤0.75) against a small ground-truth set is the sole support. Without an ablation or human validation that divergent chunks actually answer the same EQ differently, the detection numbers may reflect generic Wikipedia coverage divergence rather than genuine pragmatic signals.
- [§4.1] §4.1: Ground-truth labels for the 76 pragmatically ambiguous requirements were produced by two domain experts who were supplied with GPT-4-generated novice/intermediate/expert interpretations (Fig. 6) “only as guidance.” Although experts were instructed to rely on their own judgment and κ=0.89 is high, the guidance risks anchoring and circularity with the very multi-level interpretation idea the detector later uses. Combined with the modest hold-out size (40 % of two documents) and per-model threshold calibration on the complementary 60 %, the reported detection metrics rest on a fragile foundation that needs either independent re-annotation without model guidance or a larger multi-domain corpus.
- [§3.4, Table 3] The detection pipeline never verifies whether the retrieved domain chunks actually answer the EQ; it only checks pairwise embedding similarity of the chunks themselves (§3.4). Consequently a requirement can be declared unambiguous when three vague but mutually similar chunks fail to resolve the EQ, or ambiguous when the chunks differ for reasons orthogonal to the requirement (false-positive examples in Table 3). This gap between “chunk similarity” and “interpretation consistency with respect to the EQ” is load-bearing for the claim that the method detects pragmatic ambiguity.
minor comments (4)
- [Abstract, Table 2, §5.1] Table 2 reports macro-averaged metrics separately per document; the abstract and §5.1 quote a single “macro-averaged recall (0.75) and F2 score (0.75).” Clarify whether these are averages across documents or the best single-document figures.
- [§3.1] §3.1.1–3.1.2: chunk size (400 tokens) and overlap (20) are fixed without sensitivity analysis; a short ablation or justification would strengthen reproducibility claims.
- [§3.4] Figure 1 and Figure 4 are clear, but the 27-group enumeration in §3.4 would benefit from a brief complexity remark (3³ = 27) for readers less familiar with combinatorial retrieval.
- [§2.3] Related-work positioning (§2.3) correctly notes the infeasibility of direct comparison with Ferrari et al.; still, a qualitative side-by-side of a few shared requirements (if any exist in PURE) would help readers gauge complementarity.
Circularity Check
No significant circularity: empirical detector with hold-out threshold calibration and independent human labels, not a derivation that reduces to its inputs by construction.
full rationale
This is an empirical software-engineering paper proposing a RAG pipeline (EQ generation + multi-depth Wikipedia KBs + cosine-similarity consistency check) and evaluating it against human-annotated pragmatic-ambiguity labels on two PURE documents. The decision threshold T is selected by stratified 5-fold CV on a 60 % training split to maximise F2, then performance is reported on the held-out 40 % (standard supervised practice, not a fitted-input-called-prediction). Ground-truth labels were produced by independent domain experts (Cohen’s κ = 0.89) who were instructed to rely primarily on their own judgement; GPT-4 multi-level interpretations were supplied only as optional guidance and the experts are not co-authors. Candidate resolutions are likewise scored by the same independent experts on relevance/clarity/consistency. There is no self-definitional equation, no uniqueness theorem imported from the authors’ prior work, no ansatz smuggled via self-citation, and no renaming of a known result. The Wikipedia-depth proxy for expertise is an external-validity assumption, not a circular reduction. Consequently the derivation chain contains no step that is equivalent to its inputs by construction.
Axiom & Free-Parameter Ledger
free parameters (3)
- similarity threshold T =
0.87 (GPT-4o-mini), 0.85 (Llama), 0.82 (Mistral), 0.86 (Qwen)
- retrieval top-k and chunk size =
k=5/3, 400 tokens
- WikiDoMiner depth levels =
0=novice, 1=intermediate, 2=expert
axioms (3)
- ad hoc to paper Pragmatic ambiguity is adequately operationalized as pairwise cosine-similarity divergence among top-3 chunks retrieved from novice/intermediate/expert Wikipedia KBs once the requirements document itself fails to answer the generated EQs.
- domain assumption Wikipedia category/subcategory expansion systematically approximates increasing domain expertise.
- domain assumption LLM-generated elucidation questions surface the terms whose differing interpretations constitute pragmatic ambiguity.
invented entities (2)
-
graded domain knowledge bases KN/KI/KE
no independent evidence
-
elucidation questions (EQs)
no independent evidence
read the original abstract
Natural language requirements (NLRs) are essential for bridging communication gaps among diverse stakeholders in software development. However, the inherent ambiguity in NLRs can pose significant challenges. In particular, some requirements may be misinterpreted due to varying contextual knowledge and domain-specific expectations of the stakeholders, a phenomenon known as pragmatic ambiguity. This paper presents an approach for detecting and resolving pragmatic ambiguities in NLRs. The approach leverages retrieval-augmented generation techniques with novice, intermediate, and expert domain knowledge bases to simulate stakeholders with varying domain expertise and detect discrepancies in requirement interpretation. Candidate disambiguated requirements are generated using the expert domain knowledge base, with final validation by a requirements analyst required to ensure alignment with the intended functionality. We evaluate the approach on two requirements specification documents from the PUblic REquirements dataset, using four large language models: GPT-4o-mini, Mistral-7B, Llama-3.1-8B, and Qwen2.5-7B. Detection performance is assessed using macro-averaged accuracy, precision, recall, F1, and F2 scores. The resolution quality of the candidate disambiguated requirements is measured through human evaluation of relevance, clarity, and consistency. In this initial evaluation, results show that the proposed approach can detect pragmatic ambiguities and produce candidate disambiguated requirements that are relevant, clear, and consistent with the intended system functionality. Among the evaluated models, GPT-4o-mini achieved the highest macro-averaged recall (0.75) and F2 score (0.75) for pragmatic ambiguity detection. In the resolution task, GPT-4o-mini received the highest relevance scores from human evaluators, while Mistral-7B achieved the highest scores for clarity and consistency.
Figures
Reference graph
Works this paper leans on
-
[1]
ISO/IEC/IEEE 29148:2018 – Systems and Software Engineering — Life Cycle Processes — Requirements Engineering
2018. ISO/IEC/IEEE 29148:2018 – Systems and Software Engineering — Life Cycle Processes — Requirements Engineering. https://www.iso.org/standard/72089. html Accessed: 2025-09-27
2018
-
[2]
Mistral AI. 2023. Mistral 7B: A 7 Billion Parameter Language Model. https: //mistral.ai/news/announcing-mistral-7b Accessed: 2025-01-01
2023
-
[3]
D. M. Berry, E. Kamsties, and M. M. Krieger. 2003. From contract drafting to software specification: linguistic sources of ambiguity. https://cs.uwaterloo.ca/ ~dberry/handbook/ambiguityHandbook.pdf. Accessed: 2025-01-01
2003
-
[4]
2009.Software cost estimation with COCOMO II
Barry W Boehm, Chris Abts, A Winsor Brown, Sunita Chulani, Bradford K Clark, Ellis Horowitz, Ray Madachy, Donald J Reifer, and Bert Steece. 2009.Software cost estimation with COCOMO II. Prentice Hall Press
2009
-
[5]
Chantree, Bashar Nuseibeh, Anne De Roeck, and Alistair Willis
F. Chantree, Bashar Nuseibeh, Anne De Roeck, and Alistair Willis. 2006. Iden- tifying Nocuous Ambiguities in Natural Language Requirements.Proceedings of 14th IEEE International Requirements Engineering Conference (RE’06), 59 – 68. doi:10.1109/RE.2006.31
-
[6]
Michelene TH Chi, Paul J Feltovich, and Robert Glaser. 1981. Categorization and representation of physics problems by experts and novices.Cognitive science5, 2 (1981), 121–152
1981
-
[7]
1992.Issues in Requirements Elicitation
Michael Christel and Kyo Kang. 1992.Issues in Requirements Elicitation. Tech- nical Report CMU/SEI-92-TR-012. https://www.sei.cmu.edu/library/issues-in- requirements-elicitation/ Accessed: 2025-09-26
1992
-
[8]
D.A. Cruse. 1986.Lexical Semantics. Cambridge University Press. https://books. google.co.in/books?id=xDSBaet2uSsC
1986
-
[9]
Alan Davis, Scott Overmyer, Kathleen Jordan, Joseph Caruso, Fatma Dandashi, Anhtuan Dinh, Gary Kincaid, Glen Ledeboer, Patricia Reynolds, Pradip Sitaram, et al. 1993. Identifying and measuring quality in a software requirements specifi- cation. In[1993] Proceedings First International Software Metrics Symposium. Ieee, 141–152
1993
-
[10]
Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2022. Automated Handling of Anaphoric Ambiguity in Requirements: A Multi-solution Study. In2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE). 187–199. doi:10.1145/3510003.3510157
-
[11]
Saad Ezzini, Sallam Abualhaija, and Mehrdad Sabetzadeh. 2022. WikiDoMiner: wikipedia domain-specific miner. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Singapore, Singapore)(ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 1706–1710. doi:10.114...
-
[12]
Henning Femmer. 2018. Requirements Quality Defect Detection with the Qualicen Requirements Scout. InREFSQ Workshops
2018
-
[13]
Henning Femmer, Daniel Méndez Fernández, Stefan Wagner, and Sebastian Eder
-
[14]
Rapid quality assurance with Requirements Smells.Journal of Systems and Software123 (2017), 190–213. doi:10.1016/j.jss.2016.02.047
-
[15]
Alessio Ferrari and Andrea Esuli. 2019. An NLP approach for cross-domain am- biguity detection in requirements engineering.Automated Software Engineering 26 (2019), 559 – 598
2019
-
[16]
Alessio Ferrari, Gloria Gori, Benedetta Rosadini, Iacopo Trotta, Stefano Bacherini, Alessandro Fantechi, and Stefania Gnesi. 2018. Detecting requirements defects with NLP patterns: an industrial experience in the railway domain.Empirical Software Engineering23 (2018), 3684 – 3733
2018
-
[17]
Alessio Ferrari, Giuseppe Lipari, Stefania Gnesi, and Giorgio O. Spagnolo. 2014. Pragmatic ambiguity detection in natural language requirements. In2014 IEEE 1st International Workshop on Artificial Intelligence for Requirements Engineering (AIRE). 1–8. doi:10.1109/AIRE.2014.6894849
-
[18]
Alessio Ferrari, Giorgio Oronzo Spagnolo, and Stefania Gnesi. 2017. PURE: A Dataset of Public Requirements Documents.2017 IEEE 25th International Requirements Engineering Conference (RE)(2017), 502–505
2017
-
[19]
Stefania Gnesi. 2012. Using collective intelligence to detect pragmatic ambigui- ties.2012 20th IEEE International Requirements Engineering Conference, RE 2012 - Proceedings, 191–200. doi:10.1109/RE.2012.6345803
-
[20]
Vaibhav Jain, Ruchika Malhotra, Sanskar Jain, and Nishant Tanwar. 2020. Cross- Domain Ambiguity Detection using Linear Transformation of Word Embedding Spaces. arXiv:1910.12956 [cs.CL] https://arxiv.org/abs/1910.12956
Pith/arXiv arXiv 2020
-
[21]
Weld, Luke Zettlemoyer, and Omer Levy
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans.Transactions of the Association for Computational Linguistics8 (2020), 64–77. doi:10.1162/tacl_a_00300
-
[22]
2019.QuARS: A Pioneer Tool for NL Requirement Analysis
Giuseppe Lami, Mario Fusani, and Gianluca Trentanni. 2019.QuARS: A Pioneer Tool for NL Requirement Analysis. Springer International Publishing, Cham, 211–219. doi:10.1007/978-3-030-30985-5_13
-
[23]
Xiaoli Lian, Mona Rahimi, Jane Cleland-Huang, Li Zhang, Remo Ferrai, and Michael Smith. 2016. Mining Requirements Knowledge from Collections of Domain Documents. In2016 IEEE 24th International Requirements Engineering Conference (RE). 156–165. doi:10.1109/RE.2016.50
-
[24]
Aaron Massey, Richard Rutledge, Annie Antón, and Peter Swire. 2014. Identi- fying and classifying ambiguity for regulatory requirements.2014 IEEE 22nd International Requirements Engineering Conference, RE 2014 - Proceedings, 83–92. doi:10.1109/RE.2014.6912250
-
[25]
Meta. 2024. Llama 3.1: Our Most Capable Models to Date. https://ai.meta.com/ blog/meta-llama-3-1/ Accessed: 2025-01-01
2024
-
[26]
David Milne, Olena Medelyan, and Ian H Witten. 2006. Mining domain-specific thesauri from wikipedia: A case study. In2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006 Main Conference Proceedings)(WI’06). IEEE, 442–448
2006
-
[27]
Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic con- struction, evaluation and application of a wide-coverage multilingual semantic network.Artificial intelligence193 (2012), 217–250
2012
-
[28]
OpenAI. 2023. GPT-4 Technical Report.arXiv preprint arXiv:2303.08774(2023). https://cdn.openai.com/papers/gpt-4.pdf
Pith/arXiv arXiv 2023
-
[29]
OpenAI. 2024. GPT-4o mini: Advancing Cost-Efficient Intelligence. https:// openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Accessed: 2025-01-01
2024
-
[30]
OpenAI. 2024. New embedding models and API updates. https://openai.com/ index/new-embedding-models-and-api-updates/. Accessed: 2025-09-27
2024
-
[31]
2024.Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs
James Phoenix and Mike Taylor. 2024.Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs. O’Reilly Media, Sebastopol, CA
2024
-
[32]
Pinecone Systems, Inc. 2025. Pinecone: Scalable Vector Database. https://www. pinecone.io/. Accessed: 2025-09-27
2025
-
[33]
QwenLM. 2024. Qwen2.5: A Party of Foundation Models! https://qwenlm.github. io/blog/qwen2.5/ Accessed: 2025-01-01
2024
-
[34]
Danissa V. Rodriguez, Doris L. Carver, and Anas Mahmoud. 2018. An efficient wikipedia-based approach for better understanding of natural language text related to user requirements. In2018 IEEE Aerospace Conference. 1–16. doi:10. 1109/AERO.2018.8396645
arXiv 2018
-
[35]
Sri Fatimah Tjong and Daniel M. Berry. 2013. The Design of SREE — A Prototype Potential Ambiguity Finder for Requirements Specifications and Lessons Learned. InRequirements Engineering: Foundation for Software Quality, Joerg Doerr and Andreas L. Opdahl (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 80–95
2013
-
[36]
Óscar Dieste Tubío and Natalia Juristo Juzgado. 2011. Systematic review and aggregation of empirical studies on elicitation techniques.IEEE Transactions on Software Engineering37 (2011), 283–304
2011
-
[37]
2005.Requirements Elicitation: A Survey of Techniques, Approaches, and Tools
Didar Zowghi and Chad Coulin. 2005.Requirements Elicitation: A Survey of Techniques, Approaches, and Tools. Springer Berlin Heidelberg, Berlin, Heidelberg, 19–46. doi:10.1007/3-540-28244-0_2
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.