Pith. sign in

REVIEW 3 major objections 4 minor 37 references

A RAG system that simulates novice-to-expert domain knowledge detects pragmatic ambiguities in natural-language requirements and produces candidate clarifications.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-11 19:11 UTC pith:DI6LJX2K

load-bearing objection Solid, reproducible RAG pipeline for pragmatic ambiguity in RE; the Wikipedia-as-expertise proxy is the soft spot, but the work is still worth a referee. the 3 major comments →

arxiv 2607.04436 v1 pith:DI6LJX2K submitted 2026-07-05 cs.SE cs.AI

A Retrieval-Augmented Framework for Detecting and Resolving Pragmatic Ambiguities in Natural Language Requirements

classification cs.SE cs.AI
keywords Pragmatic AmbiguityRequirements EngineeringNatural Language RequirementsLarge Language ModelsRetrieval-Augmented GenerationAmbiguity DetectionRequirements Analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Natural-language software requirements are often read differently by people who hold different amounts of domain knowledge; those interpretation gaps are called pragmatic ambiguities and they later cause expensive rework. This paper shows that the gaps can be found automatically by building three Wikipedia-derived knowledge bases that approximate novice, intermediate and expert understanding, generating clarification questions for each requirement, and checking whether the answers retrieved from the three bases stay similar. When similarity falls below a calibrated threshold the requirement is flagged; the expert base then supplies material for an LLM to rewrite a clearer version that a human analyst still reviews. On two public transportation specifications the best model reaches macro-averaged F2 of 0.75 for detection, and domain experts rate the rewrites as relevant, clear and consistent. The result matters because it gives requirements analysts a concrete, early-warning recommender rather than leaving them to notice every hidden misreading by hand.

Core claim

Retrieval-augmented generation over three domain knowledge bases that approximate increasing levels of stakeholder expertise can detect pragmatic ambiguities in natural-language requirements by measuring interpretation divergence and can generate candidate disambiguated requirements that domain experts judge relevant, clear and consistent with intended system functionality.

What carries the argument

Three graded domain knowledge bases (KN, KI, KE) built by WikiDoMiner at Wikipedia depths 0/1/2, combined with LLM-generated elucidation questions and pairwise cosine-similarity thresholding of the top retrieved chunks, form the detection engine; the expert base plus a controlled rewrite prompt then supplies candidate resolutions.

Load-bearing premise

Wikipedia articles retrieved at successive category depths are good enough proxies for the real differences in domain knowledge among novice, intermediate and expert stakeholders.

What would settle it

Annotate a fresh set of requirements from a non-transportation domain with the same expert protocol; if the method’s macro F2 drops below 0.5 while a simple lexical-ambiguity baseline remains higher, the claim that graded Wikipedia retrieval reliably surfaces pragmatic ambiguity is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a retrieval-augmented generation (RAG) framework that detects pragmatic ambiguities in natural-language requirements by generating elucidation questions (EQs) for potentially ambiguous terms, then comparing interpretations retrieved from three Wikipedia-derived domain knowledge bases (novice/intermediate/expert, constructed via WikiDoMiner depths 0/1/2). A requirement is flagged ambiguous if the requirements document itself does not answer the EQs and no triple of top-3 chunks from the three KBs has all pairwise cosine similarities above a calibrated threshold T. Candidate resolutions are generated from the expert KB plus the requirements KB and are intended for requirements-analyst validation. Evaluation on two PURE transportation documents (Clarus, VII DUAP) with four LLMs reports macro F2 up to 0.75 for detection and human ratings (relevance/clarity/consistency) for resolutions; a replication package is released.

Significance. If the operationalization holds, the work supplies a practical, LLM-based recommender that surfaces expertise-dependent misreadings of requirements—an underexplored class of ambiguity—while producing candidate rewrites that domain experts rate as relevant, clear and consistent. Strengths include a publicly released replication package (ground truth, code, outputs), transparent multi-model evaluation, high inter-annotator agreement on labels (κ=0.89) and resolution scores (weighted κ=0.77), explicit error-case analysis, and alignment of resolution metrics with ISO/IEC/IEEE 29148 quality characteristics. The approach usefully generalizes prior graph- and embedding-based pragmatic-ambiguity detectors by Ferrari et al. by treating cross-domain stakeholders as novices within a single graded-expertise continuum.

major comments (3)
  1. [§§3.1.2, 3.4] §§3.1.2 and 3.4 operationalize pragmatic ambiguity as pairwise cosine-similarity divergence among the top-3 chunks retrieved from KN/KI/KE (WikiDoMiner depths 0/1/2). The manuscript states that the intent is only to capture relative differences, yet the central claim is that the pipeline “simulates stakeholders with varying domain expertise.” No direct evidence is supplied that depth expansion systematically produces expertise-correlated differences that matter for the generated EQs; end-to-end macro-F2 (≤0.75) against a small ground-truth set is the sole support. Without an ablation or human validation that divergent chunks actually answer the same EQ differently, the detection numbers may reflect generic Wikipedia coverage divergence rather than genuine pragmatic signals.
  2. [§4.1] §4.1: Ground-truth labels for the 76 pragmatically ambiguous requirements were produced by two domain experts who were supplied with GPT-4-generated novice/intermediate/expert interpretations (Fig. 6) “only as guidance.” Although experts were instructed to rely on their own judgment and κ=0.89 is high, the guidance risks anchoring and circularity with the very multi-level interpretation idea the detector later uses. Combined with the modest hold-out size (40 % of two documents) and per-model threshold calibration on the complementary 60 %, the reported detection metrics rest on a fragile foundation that needs either independent re-annotation without model guidance or a larger multi-domain corpus.
  3. [§3.4, Table 3] The detection pipeline never verifies whether the retrieved domain chunks actually answer the EQ; it only checks pairwise embedding similarity of the chunks themselves (§3.4). Consequently a requirement can be declared unambiguous when three vague but mutually similar chunks fail to resolve the EQ, or ambiguous when the chunks differ for reasons orthogonal to the requirement (false-positive examples in Table 3). This gap between “chunk similarity” and “interpretation consistency with respect to the EQ” is load-bearing for the claim that the method detects pragmatic ambiguity.
minor comments (4)
  1. [Abstract, Table 2, §5.1] Table 2 reports macro-averaged metrics separately per document; the abstract and §5.1 quote a single “macro-averaged recall (0.75) and F2 score (0.75).” Clarify whether these are averages across documents or the best single-document figures.
  2. [§3.1] §3.1.1–3.1.2: chunk size (400 tokens) and overlap (20) are fixed without sensitivity analysis; a short ablation or justification would strengthen reproducibility claims.
  3. [§3.4] Figure 1 and Figure 4 are clear, but the 27-group enumeration in §3.4 would benefit from a brief complexity remark (3³ = 27) for readers less familiar with combinatorial retrieval.
  4. [§2.3] Related-work positioning (§2.3) correctly notes the infeasibility of direct comparison with Ferrari et al.; still, a qualitative side-by-side of a few shared requirements (if any exist in PURE) would help readers gauge complementarity.

Circularity Check

0 steps flagged

No significant circularity: empirical detector with hold-out threshold calibration and independent human labels, not a derivation that reduces to its inputs by construction.

full rationale

This is an empirical software-engineering paper proposing a RAG pipeline (EQ generation + multi-depth Wikipedia KBs + cosine-similarity consistency check) and evaluating it against human-annotated pragmatic-ambiguity labels on two PURE documents. The decision threshold T is selected by stratified 5-fold CV on a 60 % training split to maximise F2, then performance is reported on the held-out 40 % (standard supervised practice, not a fitted-input-called-prediction). Ground-truth labels were produced by independent domain experts (Cohen’s κ = 0.89) who were instructed to rely primarily on their own judgement; GPT-4 multi-level interpretations were supplied only as optional guidance and the experts are not co-authors. Candidate resolutions are likewise scored by the same independent experts on relevance/clarity/consistency. There is no self-definitional equation, no uniqueness theorem imported from the authors’ prior work, no ansatz smuggled via self-citation, and no renaming of a known result. The Wikipedia-depth proxy for expertise is an external-validity assumption, not a circular reduction. Consequently the derivation chain contains no step that is equivalent to its inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central claim depends on a small set of free parameters (thresholds, retrieval depths, top-k) fitted or chosen experimentally, on the domain assumption that Wikipedia category expansion approximates expertise gradients, and on the invented operational entities (graded KBs, EQs) that have no independent existence outside the pipeline. No deep mathematical axioms are required; the work is engineering-empirical.

free parameters (3)
  • similarity threshold T = 0.87 (GPT-4o-mini), 0.85 (Llama), 0.82 (Mistral), 0.86 (Qwen)
    Calibrated per LLM by maximizing F2 on stratified 5-fold CV of the 60 % training split; final values 0.82–0.87. Directly decides the ambiguous/unambiguous label.
  • retrieval top-k and chunk size = k=5/3, 400 tokens
    Top-5 from KR, top-3 from each domain KB, 400-token chunks with 20-token overlap; chosen experimentally for performance/efficiency trade-off and affect which interpretations are compared.
  • WikiDoMiner depth levels = 0=novice, 1=intermediate, 2=expert
    Depths 0/1/2 define the three expertise strata; the mapping is a design choice, not derived from stakeholder data.
axioms (3)
  • ad hoc to paper Pragmatic ambiguity is adequately operationalized as pairwise cosine-similarity divergence among top-3 chunks retrieved from novice/intermediate/expert Wikipedia KBs once the requirements document itself fails to answer the generated EQs.
    Stated in Sections 3.3–3.4; no external validation that this divergence matches real stakeholder misinterpretation rates.
  • domain assumption Wikipedia category/subcategory expansion systematically approximates increasing domain expertise.
    Justified by citation to prior WikiDoMiner and graph-expansion literature (Section 3.1.2); treated as given rather than measured against actual experts.
  • domain assumption LLM-generated elucidation questions surface the terms whose differing interpretations constitute pragmatic ambiguity.
    EQGen prompt and examples (Figure 2); false-negative analysis later shows missed terms remain a failure mode.
invented entities (2)
  • graded domain knowledge bases KN/KI/KE no independent evidence
    purpose: Simulate stakeholders of three expertise levels so that interpretation discrepancies can be measured automatically.
    Constructed solely for this pipeline via WikiDoMiner depths; no independent existence or external falsifiable prediction.
  • elucidation questions (EQs) no independent evidence
    purpose: Intermediate representation that focuses retrieval and verification on specific ambiguous terms.
    Prompt-engineered artifact of the method; not a pre-existing linguistic object with independent measurement.

pith-pipeline@v1.1.0-grok45 · 21736 in / 3049 out tokens · 32572 ms · 2026-07-11T19:11:16.426245+00:00 · methodology

0 comments
read the original abstract

Natural language requirements (NLRs) are essential for bridging communication gaps among diverse stakeholders in software development. However, the inherent ambiguity in NLRs can pose significant challenges. In particular, some requirements may be misinterpreted due to varying contextual knowledge and domain-specific expectations of the stakeholders, a phenomenon known as pragmatic ambiguity. This paper presents an approach for detecting and resolving pragmatic ambiguities in NLRs. The approach leverages retrieval-augmented generation techniques with novice, intermediate, and expert domain knowledge bases to simulate stakeholders with varying domain expertise and detect discrepancies in requirement interpretation. Candidate disambiguated requirements are generated using the expert domain knowledge base, with final validation by a requirements analyst required to ensure alignment with the intended functionality. We evaluate the approach on two requirements specification documents from the PUblic REquirements dataset, using four large language models: GPT-4o-mini, Mistral-7B, Llama-3.1-8B, and Qwen2.5-7B. Detection performance is assessed using macro-averaged accuracy, precision, recall, F1, and F2 scores. The resolution quality of the candidate disambiguated requirements is measured through human evaluation of relevance, clarity, and consistency. In this initial evaluation, results show that the proposed approach can detect pragmatic ambiguities and produce candidate disambiguated requirements that are relevant, clear, and consistent with the intended system functionality. Among the evaluated models, GPT-4o-mini achieved the highest macro-averaged recall (0.75) and F2 score (0.75) for pragmatic ambiguity detection. In the resolution task, GPT-4o-mini received the highest relevance scores from human evaluators, while Mistral-7B achieved the highest scores for clarity and consistency.

Figures

Figures reproduced from arXiv: 2607.04436 by Pavithra PM Nair, Preethu Rose Anish.

Figure 1
Figure 1. Figure 1: Proposed Approach for Pragmatic Ambiguity De [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt Used for Verifying Whether Chunks Re [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Proposed Approach for Pragmatic Ambiguity Reso [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prompt Used for Generation of Candidate Require [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 8 canonical work pages

  1. [1]

    ISO/IEC/IEEE 29148:2018 – Systems and Software Engineering — Life Cycle Processes — Requirements Engineering

    2018. ISO/IEC/IEEE 29148:2018 – Systems and Software Engineering — Life Cycle Processes — Requirements Engineering. https://www.iso.org/standard/72089. html Accessed: 2025-09-27

  2. [2]

    Mistral AI. 2023. Mistral 7B: A 7 Billion Parameter Language Model. https: //mistral.ai/news/announcing-mistral-7b Accessed: 2025-01-01

  3. [3]

    D. M. Berry, E. Kamsties, and M. M. Krieger. 2003. From contract drafting to software specification: linguistic sources of ambiguity. https://cs.uwaterloo.ca/ ~dberry/handbook/ambiguityHandbook.pdf. Accessed: 2025-01-01

  4. [4]

    2009.Software cost estimation with COCOMO II

    Barry W Boehm, Chris Abts, A Winsor Brown, Sunita Chulani, Bradford K Clark, Ellis Horowitz, Ray Madachy, Donald J Reifer, and Bert Steece. 2009.Software cost estimation with COCOMO II. Prentice Hall Press

  5. [5]

    Chantree, Bashar Nuseibeh, Anne De Roeck, and Alistair Willis

    F. Chantree, Bashar Nuseibeh, Anne De Roeck, and Alistair Willis. 2006. Iden- tifying Nocuous Ambiguities in Natural Language Requirements.Proceedings of 14th IEEE International Requirements Engineering Conference (RE’06), 59 – 68. doi:10.1109/RE.2006.31

  6. [6]

    Michelene TH Chi, Paul J Feltovich, and Robert Glaser. 1981. Categorization and representation of physics problems by experts and novices.Cognitive science5, 2 (1981), 121–152

  7. [7]

    1992.Issues in Requirements Elicitation

    Michael Christel and Kyo Kang. 1992.Issues in Requirements Elicitation. Tech- nical Report CMU/SEI-92-TR-012. https://www.sei.cmu.edu/library/issues-in- requirements-elicitation/ Accessed: 2025-09-26

  8. [8]

    D.A. Cruse. 1986.Lexical Semantics. Cambridge University Press. https://books. google.co.in/books?id=xDSBaet2uSsC

  9. [9]

    Alan Davis, Scott Overmyer, Kathleen Jordan, Joseph Caruso, Fatma Dandashi, Anhtuan Dinh, Gary Kincaid, Glen Ledeboer, Patricia Reynolds, Pradip Sitaram, et al. 1993. Identifying and measuring quality in a software requirements specifi- cation. In[1993] Proceedings First International Software Metrics Symposium. Ieee, 141–152

  10. [10]

    Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2022. Automated Handling of Anaphoric Ambiguity in Requirements: A Multi-solution Study. In2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE). 187–199. doi:10.1145/3510003.3510157

  11. [11]

    Saad Ezzini, Sallam Abualhaija, and Mehrdad Sabetzadeh. 2022. WikiDoMiner: wikipedia domain-specific miner. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Singapore, Singapore)(ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 1706–1710. doi:10.114...

  12. [12]

    Henning Femmer. 2018. Requirements Quality Defect Detection with the Qualicen Requirements Scout. InREFSQ Workshops

  13. [13]

    Henning Femmer, Daniel Méndez Fernández, Stefan Wagner, and Sebastian Eder

  14. [14]

    doi:10.1016/j.jss.2016.02.047

    Rapid quality assurance with Requirements Smells.Journal of Systems and Software123 (2017), 190–213. doi:10.1016/j.jss.2016.02.047

  15. [15]

    Alessio Ferrari and Andrea Esuli. 2019. An NLP approach for cross-domain am- biguity detection in requirements engineering.Automated Software Engineering 26 (2019), 559 – 598

  16. [16]

    Alessio Ferrari, Gloria Gori, Benedetta Rosadini, Iacopo Trotta, Stefano Bacherini, Alessandro Fantechi, and Stefania Gnesi. 2018. Detecting requirements defects with NLP patterns: an industrial experience in the railway domain.Empirical Software Engineering23 (2018), 3684 – 3733

  17. [17]

    Spagnolo

    Alessio Ferrari, Giuseppe Lipari, Stefania Gnesi, and Giorgio O. Spagnolo. 2014. Pragmatic ambiguity detection in natural language requirements. In2014 IEEE 1st International Workshop on Artificial Intelligence for Requirements Engineering (AIRE). 1–8. doi:10.1109/AIRE.2014.6894849

  18. [18]

    Alessio Ferrari, Giorgio Oronzo Spagnolo, and Stefania Gnesi. 2017. PURE: A Dataset of Public Requirements Documents.2017 IEEE 25th International Requirements Engineering Conference (RE)(2017), 502–505

  19. [19]

    Stefania Gnesi. 2012. Using collective intelligence to detect pragmatic ambigui- ties.2012 20th IEEE International Requirements Engineering Conference, RE 2012 - Proceedings, 191–200. doi:10.1109/RE.2012.6345803

  20. [20]

    Vaibhav Jain, Ruchika Malhotra, Sanskar Jain, and Nishant Tanwar. 2020. Cross- Domain Ambiguity Detection using Linear Transformation of Word Embedding Spaces. arXiv:1910.12956 [cs.CL] https://arxiv.org/abs/1910.12956

  21. [21]

    Weld, Luke Zettlemoyer, and Omer Levy

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans.Transactions of the Association for Computational Linguistics8 (2020), 64–77. doi:10.1162/tacl_a_00300

  22. [22]

    2019.QuARS: A Pioneer Tool for NL Requirement Analysis

    Giuseppe Lami, Mario Fusani, and Gianluca Trentanni. 2019.QuARS: A Pioneer Tool for NL Requirement Analysis. Springer International Publishing, Cham, 211–219. doi:10.1007/978-3-030-30985-5_13

  23. [23]

    Xiaoli Lian, Mona Rahimi, Jane Cleland-Huang, Li Zhang, Remo Ferrai, and Michael Smith. 2016. Mining Requirements Knowledge from Collections of Domain Documents. In2016 IEEE 24th International Requirements Engineering Conference (RE). 156–165. doi:10.1109/RE.2016.50

  24. [24]

    Aaron Massey, Richard Rutledge, Annie Antón, and Peter Swire. 2014. Identi- fying and classifying ambiguity for regulatory requirements.2014 IEEE 22nd International Requirements Engineering Conference, RE 2014 - Proceedings, 83–92. doi:10.1109/RE.2014.6912250

  25. [25]

    Meta. 2024. Llama 3.1: Our Most Capable Models to Date. https://ai.meta.com/ blog/meta-llama-3-1/ Accessed: 2025-01-01

  26. [26]

    David Milne, Olena Medelyan, and Ian H Witten. 2006. Mining domain-specific thesauri from wikipedia: A case study. In2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006 Main Conference Proceedings)(WI’06). IEEE, 442–448

  27. [27]

    Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic con- struction, evaluation and application of a wide-coverage multilingual semantic network.Artificial intelligence193 (2012), 217–250

  28. [28]

    OpenAI. 2023. GPT-4 Technical Report.arXiv preprint arXiv:2303.08774(2023). https://cdn.openai.com/papers/gpt-4.pdf

  29. [29]

    OpenAI. 2024. GPT-4o mini: Advancing Cost-Efficient Intelligence. https:// openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Accessed: 2025-01-01

  30. [30]

    OpenAI. 2024. New embedding models and API updates. https://openai.com/ index/new-embedding-models-and-api-updates/. Accessed: 2025-09-27

  31. [31]

    2024.Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs

    James Phoenix and Mike Taylor. 2024.Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs. O’Reilly Media, Sebastopol, CA

  32. [32]

    Pinecone Systems, Inc. 2025. Pinecone: Scalable Vector Database. https://www. pinecone.io/. Accessed: 2025-09-27

  33. [33]

    QwenLM. 2024. Qwen2.5: A Party of Foundation Models! https://qwenlm.github. io/blog/qwen2.5/ Accessed: 2025-01-01

  34. [34]

    Rodriguez, Doris L

    Danissa V. Rodriguez, Doris L. Carver, and Anas Mahmoud. 2018. An efficient wikipedia-based approach for better understanding of natural language text related to user requirements. In2018 IEEE Aerospace Conference. 1–16. doi:10. 1109/AERO.2018.8396645

  35. [35]

    Sri Fatimah Tjong and Daniel M. Berry. 2013. The Design of SREE — A Prototype Potential Ambiguity Finder for Requirements Specifications and Lessons Learned. InRequirements Engineering: Foundation for Software Quality, Joerg Doerr and Andreas L. Opdahl (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 80–95

  36. [36]

    Óscar Dieste Tubío and Natalia Juristo Juzgado. 2011. Systematic review and aggregation of empirical studies on elicitation techniques.IEEE Transactions on Software Engineering37 (2011), 283–304

  37. [37]

    2005.Requirements Elicitation: A Survey of Techniques, Approaches, and Tools

    Didar Zowghi and Chad Coulin. 2005.Requirements Elicitation: A Survey of Techniques, Approaches, and Tools. Springer Berlin Heidelberg, Berlin, Heidelberg, 19–46. doi:10.1007/3-540-28244-0_2