Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Towards Requirements Engineering for RAG Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that retrieval requirements for RAG systems are elicited through iterative experimentation with expert users, and proposes a five-stage process model for doing so.

desk verdict A clear, honest single-case study that proposes a five-stage RE process for RAG systems; the model is plausible but rests entirely on user-provided correctness judgments, and the paper never squares that with the literature it cites. read the letter →

arxiv 2505.07553 v1 pith:6MFYMJRI submitted 2025-05-12 cs.SE cs.AI

Tor Sporsem , Rasmus Ulfsnes This is my paper · ORCID
classification cs.SEcs.AI
keywords RequirementsEngineeringRetrievalAugmentedGenerationRAGGenAIRE4AIcasestudymaritimeindustry
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This short paper argues that requirements for retrieval-augmented generation (RAG) systems—where an LLM pulls relevant stored documents or past answers into its prompt before answering—cannot be specified up front. Through a case study of a maritime service provider building a RAG system over 500,000 previously written expert answers, it shows that data scientists had to discover 'retrieval requirements' through iterative experimentation with the expert users who judge whether generated answers are correct. The paper presents an empirically grounded, five-stage iterative process model: knowledge modeling and experimentation, retrieval strategy, retrievable data management, monitoring and operation, and continuous expectation management. The contribution matters because RAG is a pragmatic way to integrate LLMs into organizations, yet software and requirements engineering research has lacked concrete industry guidance on how such systems' requirements are actually elicited.

What carries the argument

The central object is the 'retrieval requirement' (RR), defined in the model as a requirement about what should be retrievable from the knowledge base as input to the LLM. The mechanism carrying the argument is the iterative experimentation loop: data scientists release an imperfect RAG system, expert users judge output correctness, those judgments reveal gaps in retrievable data, and each gap is addressed by a compensation such as a filtering function or mandatory human review. That loop is organized by the paper's five-stage process model, which gives requirements engineering a concrete handle on a system class where the traditional 'specify first' approach breaks down.

What would settle it

If a comparable RAG deployment in another expert organization achieved equally correct outputs (as judged by independent experts) using pre-specified requirements and automated retrieval evaluation without iterative user experimentation or user-controlled filters, the claimed necessity of the five-stage user-experimentation process would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that RAG system correctness is not a property the data science team can define alone; it resides with experienced case handlers, who know current rules, strategic interpretation norms, and the infinite combinations of vessel characteristics that make context unique. Because the knowledge base of past answers cannot cover every context and becomes outdated as rules change, the data scientists repeatedly hit retrieval failures and compensated by giving users filtering controls (by year, vessel type, nationality, age, and similar factors) and by requiring a human check of every generated answer. Each such compensation is a 'retrieval requirement'—a decision about which parts of the knowledge base may legitimately be retrieved as input to the LLM. The paper generalizes these observations into a five-stage iterative process model in which requirements emerge from use, are refined through monitoring, and unfold under continuous—and in this case partly abandoned—expectation management.

Load-bearing premise

The whole process model depends on the assumption that the experienced users' judgments of whether an answer is correct are a valid and sufficient ground truth for deciding what the system should retrieve.

Editorial extensions

If this is right

  • RAG requirements engineering should treat user experimentation, not upfront specification, as the primary discovery mechanism.
  • Retrieval strategy and retrievable data management become first-class RE stages, because runtime control of what the LLM can access substitutes for retraining in traditional ML.
  • Monitoring live system use is itself a requirements elicitation activity: new retrieval requirements surface only after real users interact with early versions.
  • Expectation management must be planned as an ongoing activity, but the case shows it can fail under LLM hype, leaving users with unmet expectations even as the system improves.
  • The human check on every generated answer is a deliberate requirements decision, not just a safety feature, because interpretation norms and strategic motives cannot be captured in the corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same pattern—legacy corpus, dated rules, and context combinations too numerous to encode—is likely to appear in legal, medical, and financial RAG deployments, so the five stages may transfer across expert domains.
  • Beyond the paper: because correctness is delegated to expert users, the process inherits their blind spots; a systematic error shared by experts would be encoded into the retrieval requirements, suggesting organizations should add independent outcome checks.
  • Beyond the paper: the filter choices case handlers make could be logged and used to learn which context features predict retrieval relevance, potentially automating parts of the retrieval strategy stage in later deployments.
  • Beyond the paper: the observation that expectation management was abandoned implies the real constraint on RAG adoption may be organizational—managing hype—rather than technical retrieval quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a qualitative case study at a maritime company (Marcomp) that developed a RAG system over a knowledge base of 500,000 historical case-handler answers. Through observation and interviews, the authors describe how data scientists iteratively experimented with case handlers to identify what they call 'retrieval requirements,' eventually building filtering functions that let users constrain retrieval by year and ship characteristics. The paper proposes a five-stage iterative process model: knowledge modeling and experimentation, retrieval strategy, retrievable data management, monitoring and operation, and continuous expectation management. It claims that eliciting retrieval requirements is essential because users are the only ones who can determine correctness.

Significance. If the process model is accepted as a descriptive account, it is a useful early empirical contribution to requirements engineering for RAG systems, a topic with little industry evidence. The study's strengths are its substantial fieldwork (28 hours of observation plus multiple interviews), its transparent reporting of limitations, and its grounding in practitioner quotes and observed events. The proposed distinction between retrieval strategy and retrievable data management is a plausible and potentially reusable conceptual contribution. However, the normative force of the claims currently exceeds what the single-case evidence can support, because outcome data are absent and the reliability of user correctness judgments is not reconciled with the paper's own cited literature.

major comments (3)
  1. [§4.1, §5, Abstract] The central claim that data scientists must rely on case handlers because 'they are the ones who can determine correctness' is in tension with the paper's own citation of Lebovitz et al. [13], which documents that experts' 'know-what' can be biased, outdated, or inconsistent. The manuscript provides no independent validation of the case handlers' judgments, and Section 6 concedes that the system's effects have not been measured. As written, the evidence supports only that user judgments were used as a pragmatic proxy in this case, not that they are valid ground truth. Please either soften the normative claim or show how the proposed process would detect or correct biased user judgments.
  2. [§5, §6] The conclusion that eliciting 'retrieval requirements' is essential to ensure output correctness is not supported without outcome measurement. The authors state in Section 6 that they 'have not yet measured the effects of the RAG system' and 'cannot determine whether its development has been a success.' Without any outcome measure, the paper cannot establish that the five-stage process actually produces correct RAG outputs; it can only describe observed practice. The prescriptive conclusions should be reframed as a hypothesis or as a descriptive account of current practice until outcome data are collected.
  3. [§5, Figure 2] The internal structure of the proposed process model is presented inconsistently. The text in Section 5 says the authors 'identify four sequential steps' with expectation management as an 'underlying continuous activity,' while Figure 2 and the conclusion describe a 'five-stage iterative process.' Since the process model is the paper's main contribution, the relationship between expectation management and the other four stages must be clarified: is expectation management a stage, a cross-cutting activity, or both? This distinction affects how the model should be applied and evaluated.
minor comments (5)
  1. [Table 1] Table 1 does not specify the number of interviews conducted with the Data Architect versus the AI Solution Engineer; reporting the exact counts and timing of each interview round would aid reproducibility and transparency.
  2. [§3] The description of the analysis would benefit from reporting whether the temporal-bracketing phase boundaries were validated with participants (e.g., member checking), since the phase structure is central to the proposed model.
  3. [§4.2] There is a typographical error in the quote attribution: 'Architecht' should be 'Architect.' The same misspelling appears in the later quote from Magnus.
  4. [§5 heading] The heading 'TOW ARDS REQUIREMENTS ENGINEERING FOR RAG' appears to contain a line-break artifact; it should read 'TOWARDS REQUIREMENTS ENGINEERING FOR RAG.'
  5. [References] Reference [17] (Runeson and Höst) gives only the starting page '131'; it should include the full page range and, ideally, a DOI.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the process model is inductively derived from a case study; self-citations are not load-bearing.

full rationale

The paper proposes a five-stage iterative process model for eliciting 'retrieval requirements' in RAG systems, based on a qualitative case study using grounded theory, open coding, and temporal bracketing. The central claim—that data scientists must identify context-specific retrieval requirements through iterative experimentation with users because users can determine correctness—is a finding inductively derived from observations and interviews, not a derivation from a fitted parameter, a self-referential definition, or a uniqueness theorem. The only self-citations (Ulfsnes et al. 2024a, 2024b) are used for background claims about LLM outputs lacking a single correct answer and LLMs excelling at software engineering; these are not load-bearing for the proposed process model. The paper explicitly acknowledges its limitations: it is a single case study without measured effects, and construct validity is challenged by immature practitioner terminology. The skeptic's concern that user judgments may be biased or inconsistent (citing Lebovitz et al.) is a validity threat and an internal tension, but not circularity, because the paper's contribution is a descriptive process model, not a claim that its output is mathematically forced. No equation or construction reduces to its own inputs, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The paper makes no numeric fitting. It relies on the assumption that user judgments are valid correctness signals and that a single case can support a process model. It also introduces the novel construct 'retrieval requirements' without external validation.

assumptions (3)
  • domain assumption Case handlers can reliably judge the correctness of generated answers.
    The study treats users' perceptions of correctness as ground truth for retrieval requirements; Section 4.1 states data scientists needed case handlers to determine correctness.
  • domain assumption A single-case qualitative design can yield a transferable process model.
    Methodological assumption; acknowledged as a limitation in Section 6.
  • domain assumption Knowledge base diversity and completeness determine RAG output quality.
    Borrowed from prior ML literature (Vogelsang and Borg [26]) and used to interpret the case in Section 2.2.
invented entities (1)
  • Retrieval requirements (RR)
    purpose: Context-specific requirements for what a RAG system should retrieve and filter, elicited through user experimentation.
    The paper introduces this as a novel construct; it is derived from the single case and not yet validated or measured externally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Requirements Engineering for RAG Systems." pith.science (2026). https://pith.science/paper/6MFYMJRI

@misc{pith2026250507553,
  author       = {Pith},
  title        = {Pith review of: Towards Requirements Engineering for RAG Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6MFYMJRI}},
  note         = {Machine review of arXiv:2505.07553}
}
read the original abstract

This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific "retrieval requirements" through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these "retrieval requirements" and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.

Figures

Figures reproduced from arXiv: 2505.07553 by the authors.

Figure 1
Figure 1. The RAG system as developed by Marcomp. First, a [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. An iterative five-stage process model for eliciting [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

  1. [13]

    Lebovitz, S., Levine, N., and Lifshitz-Assaf, H. (2021). Is AI ground truth really true? The dangers of training and evaluating AI tools based on experts’ know-what. MIS Quarterly, 45(3, SI):1501–1526

    work page 2021

  2. [1]

    Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., and Grundy, J. (2023). Require- ments practices and gaps when engineering human-centered Artificial Intelligence systems. Applied Soft Computing, 143:110421

    work page 2023

  3. [2]

    Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019). Software Engineering for Machine Learning: A Case Study. In2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , pages 291–300

    work page 2019

  4. [3]

    Arslan, M., Ghanem, H., Munawar, S., and Cruz, C. (2024). A Survey on RAG with LLMs. Procedia Computer Science, 246:3781–3790. Publisher: Elsevier

    work page 2024

  5. [4]

    Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., and Abdelrazek, M. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, pages 194–199, Lisbon Portugal. ACM

    work page 2024

  6. [5]

    T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. (2023). Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv:2303.12712 [cs]

    arXiv 2023

  7. [6]

    Chen, J., Lin, H., Han, X., and Sun, L. (2024). Benchmarking Large Language Models in Retrieval-Augmented Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17754–17762. Number: 16

    work page 2024

  8. [7]

    R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., and Lakhani, K

    Dell’Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., and Lakhani, K. R. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of ai on knowledge worker productivity and quality

    work page 2023

Show all 28 references
  1. [8]

    Giray, G. (2021). A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software , 180:111031

    2021

  2. [9]

    and Yoshioka, N

    Ishikawa, F. and Yoshioka, N. (2019). How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems? - Questionnaire Survey. In 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop ...

    2019

  3. [10]

    Kalinowski, M., Mendez, D., Giray, G., Alves, A. P. S., Azevedo, K., Escovedo, T., Villamizar, H., Lopes, H., Baldassarre, T., Wagner, S., Biffl, S., Musil, J., Felderer, M., Lavesson, N., and Gorschek, T. (2024). Naming the Pain in Machine Learning-Enabled Systems Engineering...

    2024 arXiv

  4. [11]

    Kim, M., Zimmermann, T., DeLine, R., and Begel, A. (2018). Data Scientists in Software Teams: State of the Art and Challenges. IEEE Transactions on Software Engineering, 44(11):1024–1038

    2018

  5. [12]

    Langley, A. (1999). Strategies for Theorizing from Process Data. Academy of Management Review, 24(4):691–710. Publisher: Academy of Management

    1999

  6. [14]

    C., Sim, S

    Lethbridge, T. C., Sim, S. E., and Singer, J. (2005). Studying Software Engineers: Data Collection Techniques for Software Field Studies.Empirical Software Engineering, 10(3):311–341

    2005

  7. [15]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., and Rocktäschel, T. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems , 33:9459–9474

    2020

  8. [16]

    Lipton, Z. C. (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31–57

    2018

  9. [17]

    and Höst, M

    Runeson, P. and Höst, M. (2008). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2):131

    2008

  10. [18]

    Seaman, C. B. (1999). Qualitative methods in empirical studies of software engineering. IEEE Transactions on software engineering , 25(4):557–572

    1999

  11. [19]

    Stol, K.-J. (2024). Teaching Theorizing in Software Engineering Research. In Mendez, D., Avgeriou, P., Kalinowski, M., and Ali, N. B., editors,Handbook on Teaching Empirical Software Engineering, pages 31–69. Springer Nature Switzerland, Cham

    2024

  12. [20]

    Stol, K.-J., Ralph, P., and Fitzgerald, B. (2016). Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering , pages 120–131, Austin Texas. ACM

    2016

  13. [21]

    M., and Dreier, S

    Tanweer, A., Gade, E., Krafft, P. M., and Dreier, S. (2021). Why the data revolution needs qualitative thinking. Harvard Data Science Review , 3

    2021

  14. [22]

    Ulfsnes, R., Mikalsen, M., and Barbala, A. M. (2024a). From generation to applica- tion: Exploring knowledge workers’ relations with GenAI. In ICIS 2024 Proceedings

    2024

  15. [23]

    B., Stray, V., and Skarpen, M

    Ulfsnes, R., Moe, N. B., Stray, V., and Skarpen, M. (2024b). Transforming Software Development with Generative AI: Empirical Insights on Collaboration and Workflow. In Nguyen-Duc, A., Abrahamsson, P., and Khomh, F., editors, Generative AI for Effective Software Development, pa...

    2024

  16. [24]

    van den Broek, E., Sergeeva, A., and Huysman, M. (2021). When the machine meets the expert: an ethnography of developing AI for hiring. MIS Quarterly, 45(3, SI):1557–1580

    2021

  17. [25]

    L., Tripto, N

    Veturi, S., Vaichal, S., Jagadheesh, R. L., Tripto, N. I., and Yan, N. (2024). RAG based Question-Answering for Contextual Response Prediction System. arXiv:2409.03708

    2024 arXiv

  18. [26]

    and Borg, M

    Vogelsang, A. and Borg, M. (2019). Requirements Engineering for Machine Learn- ing: Perspectives from Data Scientists. In 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), pages 245–251

    2019

  19. [27]

    and Huysman, M

    Waardenburg, L. and Huysman, M. (2022). From coexistence to co-creation: Blurring boundaries in the age of AI. Information and Organization , 32(4):100432. Publisher: Elsevier

    2022

  20. [28]

    Wan, Z., Xia, X., Lo, D., and Murphy, G. C. (2021). How does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering , 47(9):1857–1871

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.