A Conceptual Framework for Requirements Engineering of Pretrained-Model-Enabled Systems

Dongming Jin; Linyu Li; Xiaohong Chen; Zhi Jin

REVIEW 3 major objections 7 minor 16 references

A Conceptual Framework for Requirements Engineering of Pretrained-Model-Enabled Systems

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that large pretrained models force requirements engineering to abandon its core assumptions and reorganize around six activities.

desk verdict A clear, honest research agenda for RE with pretrained models, but the 'paradigm shift' framing oversells a repackaging of known RE-for-ML ideas; still deserves a serious look. read the letter →

arxiv 2507.13095 v1 pith:A2RM35G3 submitted 2025-07-17 cs.SE

Dongming Jin , Zhi Jin , Linyu Li , Xiaohong Chen This is my paper

classification cs.SE

keywords pretrainedmodelsrequirementsengineeringlargelanguageprompt-as-specificationmodelcapabilitydiscoveryintent-capabilityalignmentnon-functionalartifactsmanagement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that requirements engineering, as practiced for deterministic software, will not carry over to systems whose behavior is driven by large pretrained models. It identifies three emergent properties—ambiguous capability boundaries, context-sensitive behavior, and continuous evolution—and argues they break the assumptions that requirements can be decomposed, predicted, and exhaustively verified. To replace the broken assumptions, it proposes a six-activity lifecycle ranging from model capability discovery to model artifacts management. If the argument holds, requirements engineering for such systems will center on prompt specifications, statistical validation, and continuous evolution governance rather than on fixed feature sets.

What carries the argument

Key machinery is the reorganized six-activity lifecycle, with the Prompt-as-Specification activity at its center: a prompt is treated as a first-class executable requirements specification that has explicit structure and defined semantics, created by filling model prompt templates from a user intent list. Around it sit the capability catalog—a structured repository of model capabilities, limitations, risks, and costs built from model-card information—and the mapping engine that pairs intents to catalog capabilities. The framework's claim to work is that every other activity feeds or checks these specifications: capability discovery grounds elicitation, intent-capability mapping decides where a model is needed, validation checks prompt quality statistically, and artifacts management version-stamps prompts, checkpoints, and data so evolution stays auditable.

What would settle it

Run a conventional requirements process on a deployed pretrained-model chatbot, enumerate expected inputs and outputs, then update the underlying model several times. If behavior stays within the enumerated boundaries and the original specification remains testable and complete across updates, the paper's central claim that pretrained-model-enabled systems break traditional requirements assumptions would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that software systems whose core behavior comes from pretrained models possess three emergent properties—ambiguous capability boundaries, context-sensitive behavior, and continuous evolution—and that these properties invalidate the long-standing requirements-engineering assumptions of functional decomposability and behavioral predictability. From this it concludes that requirements engineering for such systems cannot be a one-time specification activity but must become a continuous lifecycle. The proposed response is a six-activity framework: discover and catalog model capabilities, elicit user intent against that catalog, map intents to candidate models, turn prompts into first-class executable specifications, validate those specifications statistically and with human review, and version and trace prompts, checkpoints, and data. The paper presents this as a foundation for research, not as a validated method.

Load-bearing premise

The load-bearing premise is that a prompt can serve as a stable, executable requirements specification with explicit structure and defined semantics; if prompt behavior cannot be pinned down well enough to specify and verify, the central activity of the framework does not have a foundation.

Editorial extensions

If this is right

Requirements for pretrained-model-enabled systems will be written as structured prompts plus quality checks, not as exhaustive functional specifications.
Validation shifts from exhaustive input-output testing to statistical testing, scenario simulation, and human-in-the-loop review.
Model updates, prompts, and tuning data must be versioned and traced like code, to prevent configuration drift and keep audits possible.
New non-functional requirements—hallucination rate, fairness, explainability—need standard definitions and metrics before they can be engineered.
Selection of models becomes an explicit requirements activity: intents are matched to documented capabilities before any prompt is written.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The framework is presented as a research agenda, and the authors do not evaluate it end-to-end; an immediate extension is to instantiate all six activities on one product and measure whether prompt drift, rollback effort, and audit gaps decrease.
One consequence the authors leave implicit is that requirement satisfaction becomes probabilistic: a prompt specification can only promise behavior within a tolerance, so acceptance criteria should be statistical bounds rather than yes/no tests.
The capability catalog could in principle be built by automatically scraping and normalizing model cards, which would turn intent-capability mapping into a search-and-recommendation problem rather than an expert judgment task.
If prompt behavior cannot be given stable, verifiable semantics, the Prompt-as-Specification activity would need to be reframed as continuous test generation, and the rest of the framework would have to lean more heavily on validation and artifact management.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a position/vision paper, not a technical result. It reorganizes the requirements engineering lifecycle into six activities for pretrained-model-enabled systems and argues these systems break traditional RE assumptions. The writing is clear, the challenges are real, and the authors are honest that most of the machinery is future work. If you need a concise framing of why RE for pretrained models is hard, this is a decent pointer.

What's actually new: not much as a result. Model Capability Discovery, User Intent Elicitation, Intent-Capability Mapping, Prompt-as-Specification, Behavior-Driven Validation, and Model Artifacts Management map almost one-to-one onto elicitation, analysis, specification, validation, and management from standard RE. Prior RE-for-ML surveys (citations [12], [13], [15]) already cover capability uncertainty, specification gaps, and validation difficulties. The paper's contribution is packaging these into one framework and giving each activity a name. That has some organizing value, but it is not a paradigm shift; it is a relabeling with a few new artifacts like capability catalogs and prompt specifications.

The paper does several things well. It motivates the challenges concretely, with examples like model cards lacking domain jargon detail. It cites the relevant survey literature. It does not overclaim empirical validation — there is none, but the paper presents itself as a research agenda, so that is acceptable. The six activities hang together coherently, and the figure is useful.

Soft spots, in proportion. First, the central claim that these characteristics 'fundamentally challenge' and 'necessitate a paradigm shift' is asserted, not demonstrated. A skeptic could say that capability uncertainty and context sensitivity are quantitative extensions of existing RE problems, not a break from them. The paper does not engage with that counterargument. Second, the stress-test concern about Prompt-as-Specification is real: Section III-D says prompts can have 'defined semantics' but never says how, and Section IV lists prompt specification languages as future research. That is not an internal contradiction — the paper admits the gap — but it means the framework's linchpin activity is currently aspirational. Validation and artifact management inherit that open problem. This is a limitation, not a fatal flaw, for a vision paper.

Who this is for: researchers new to RE for ML or pretrained models will get a useful orientation. Practitioners looking for methods will not find anything usable yet. As a peer reviewer, I would not reject it for being unvalidated — it is explicitly a framework proposal — but I would ask the authors to soften the paradigm-shift language and to position the work more explicitly as a synthesis of existing RE-for-ML research rather than a break from it.

Recommendation: send it to peer review. It is a competent, honest vision paper in a growing area. A good referee will push for sharper novelty claims and a more careful treatment of prompt semantics, but the paper deserves that referee time.

Referee Report

3 major / 7 minor

Summary. The paper argues that software systems powered by large pretrained models have three distinctive characteristics—ambiguous capability boundaries, context-sensitive behavior, and continuous evolution—that fundamentally challenge traditional requirements engineering assumptions such as functional decomposability and behavioral predictability. It identifies six open challenges spanning elicitation, alignment, specification, validation, non-functional requirements, and artifact governance, then proposes a conceptual framework with six interconnected activities: Model Capability Discovery, User Intent Elicitation, Intent-Capability Alignment, Prompt-as-Specification, Prompt Specification Validation, and Model Artifacts Management. The paper also outlines five future research directions. The contribution is conceptual and prescriptive: no empirical evaluation, case study, or formal derivation is provided, and the framework's central Prompt-as-Specification activity relies on prompt semantics that the paper itself acknowledges are not yet defined.

Significance. If read as a research agenda, the paper provides a useful synthesis of known difficulties in requirements engineering for pretrained-model-enabled systems and a structured vocabulary for future work. It honestly names open problems such as prompt specification languages and emergent quality metrics, which is a strength. However, the paper does not ship machine-checked proofs, reproducible code, parameter-free derivations, or falsifiable predictions; its value depends on whether the proposed framework can be operationalized. The load-bearing assumption that prompts can serve as first-class executable requirements specifications is left undefined, so the contribution is a plausible starting point rather than a demonstrated method.

major comments (3)

[III-D and IV] The central activity of the framework is 'Prompt-as-Specification,' which the paper describes as elevating prompts to 'first-class and executable requirements specifications' with 'explicit structure and defined semantics' (Section III-D). However, the observable behavior of a pretrained-model-enabled system depends not only on the prompt text but also on the model checkpoint, decoding parameters, and runtime context—a dependence the paper itself emphasizes in Section I. Section III-D does not specify which of these factors is part of the prompt specification or how its semantics are defined, and Section IV lists 'prompt specification languages' as future research, which confirms that stable semantics do not yet exist. Because Sections III-E and III-F validate and version the prompt specification as a unit, the entire framework inherits this gap. This is honestly framed as an open problem, but it is load-bearing: without a definition of prompt semantics, the framework's main innovation is a placeholder. I recommend either clarifying the semantics relative to a fixed model version and inference configuration or repositioning prompts as requirements artifacts whose behavior must be jointly versioned with the model, rather than as standalone executable specifications.
[I and II] The paper's central claim is that the three characteristics 'fundamentally challenge' traditional requirements engineering and 'necessitate a paradigm shift' (Sections I and V). This is an empirical claim about the prevalence and severity of the six challenges listed in Section II, but no survey, case study, systematic literature review, or other evidence is presented to support it. The cited literature (e.g., [12]) documents problems in requirements engineering for machine learning, but the paper does not show that these problems are widespread for pretrained-model-enabled systems specifically or that the proposed framework addresses them better than existing approaches. For a conceptual/vision paper this may be acceptable if the claims are softened, but as written the 'necessitate' language is stronger than the evidence. Please either add evidence (e.g., a case study or an expert evaluation of the framework) or reframe the contribution as a research agenda that motivates rather than demonstrates the paradigm shift.
[III-A] The Model Capability Discovery activity assumes that model cards can be scraped, parsed, and normalized into a capability catalog (Section III-A). However, model cards are known in the literature to be incomplete, inconsistent, and often outdated; the paper does not discuss how the catalog should handle missing, conflicting, or stale information, nor what evidence a capability claim must carry. Since the capability catalog feeds Intent-Capability Mapping (III-C) and prompt selection (III-D), the reliability of the whole framework depends on the output of this activity. The paper should either acknowledge this limitation explicitly or describe a validation mechanism for the catalog itself.

minor comments (7)

[III-A] This catalog 'servers as' should be 'serves as.'
[III-C] 'If no required' should be 'If not required.'
[III-E] 'examples provides in the prompt' should be 'examples provided in the prompt.'
[III-F] 'pretrained models checkpoints' should be 'pretrained model checkpoints.'
[I] Section references such as 'In section II' and 'In section III' should capitalize 'Section.'
[Figure 1] Figure 1 labels 'Intent-PromptValidation' and 'Intent-Capability Mapping' differ from the corresponding section titles 'Prompt Specification Validation' and 'Intent-Capability Alignment'; the labels should be aligned with the text.
[IV] 'from the following aspect' should be 'from the following aspects.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a speculative conceptual framework whose claims do not derive from fitted parameters, definitions, or load-bearing self-citations.

full rationale

This is a position/vision paper with no equations, no fitted parameters, and no empirical derivation. The central claim—that pretrained-model-enabled systems exhibit ambiguous capability boundaries, context-sensitive behavior, and continuous evolution—is supported by qualitative argument and by external references (e.g., [8] for prompt sensitivity, [9] for model updating), not by a chain that reduces to its own inputs. The paper does contain several self-citations ([3], [11], [14], [16]), but these are used only as background illustrations or as examples of prior work: LLM agent roles, a RE textbook, feature-tree construction, and Linux package governance. None of these citations is load-bearing for the proposed framework's validity. The framework's activities (capability discovery, intent elicitation, mapping, prompt-as-specification, validation, artifact management) are presented as a conceptual reorganization, not as results derived from assumptions. The one potentially fragile premise—that prompts can be elevated to specifications with explicit structure and defined semantics (Section III-D)—is explicitly treated by the paper itself as an open problem: Section IV lists 'Creating standardized, syntax-aware prompt languages with semantic guarantees' as future research. Thus the paper does not claim to have established those semantics; it identifies a gap and proposes a direction. That is an unsupported premise or an open problem, not circularity. No self-definitional reduction, no fitted-input-renamed-as-prediction, no imported uniqueness theorem, and no ansatz smuggled via citation appears in the manuscript. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its central claim rests on domain assumptions about pretrained models and prompt behavior, which are stated without empirical validation.

assumptions (3)

domain assumption Pretrained-model-enabled systems exhibit ambiguous capability boundaries, context-sensitive behavior, and continuous evolution.
Stated in Section I as defining characteristics, but no empirical evidence is provided; the entire framework is designed to cope with these assumptions.
domain assumption Prompts can be treated as first-class executable requirements specifications with explicit structure and defined semantics.
Section III-D asserts this capability; it is a core premise for the Prompt-as-Specification activity but is not demonstrated.
domain assumption Traditional requirements engineering relies on deterministic, predictable, and decomposable behavioral assumptions.
Section I and II claim this, citing references [10] and [11], but the universality of this assumption is not established; some modern RE methods already address uncertainty.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Conceptual Framework for Requirements Engineering of Pretrained-Model-Enabled Systems." pith.science (2026). https://pith.science/paper/A2RM35G3

@misc{pith2026250713095,
  author       = {Pith},
  title        = {Pith review of: A Conceptual Framework for Requirements Engineering of Pretrained-Model-Enabled Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A2RM35G3}},
  note         = {Machine review of arXiv:2507.13095}
}

read the original abstract

Recent advances in large pretrained models have led to their widespread integration as core components in modern software systems. The trend is expected to continue in the foreseeable future. Unlike traditional software systems governed by deterministic logic, systems powered by pretrained models exhibit distinctive and emergent characteristics, such as ambiguous capability boundaries, context-dependent behavior, and continuous evolution. These properties fundamentally challenge long-standing assumptions in requirements engineering, including functional decomposability and behavioral predictability. This paper investigates this problem and advocates for a rethinking of existing requirements engineering methodologies. We propose a conceptual framework tailored to requirements engineering of pretrained-model-enabled software systems and outline several promising research directions within this framework. This vision helps provide a guide for researchers and practitioners to tackle the emerging challenges in requirements engineering of pretrained-model-enabled systems.

Figures

Figures reproduced from arXiv: 2507.13095 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

[12]

Status quo and problems of requirements engineering for machine learning: Results from an international survey,

A. P. S. Alves, M. Kalinowski, G. Giray, D. Mendez, N. Lavesson, K. Azevedo, H. Villamizar, T. Escovedo, H. Lopes, S. Bifflet al., “Status quo and problems of requirements engineering for machine learning: Results from an international survey,” in International Conference on Product-Focused Software Process Improvement . Springer, 2023, pp. 159–174

work page 2023
[13]

Requirements engineering for machine learning: A review and reflection,

Z. Pei, L. Liu, C. Wang, and J. Wang, “Requirements engineering for machine learning: A review and reflection,” in 2022 IEEE 30th International Requirements Engineering Conference Workshops (REW) . IEEE, 2022, pp. 166–175

work page 2022
[15]

Causal Models in Requirement Specifications for Machine Learning: A vision

H.-M. Heyn, Y . Mao, R. Weiss, and E. Knauss, “Causal models in requirement specifications for machine learning: A vision,” arXiv preprint arXiv:2502.11629, 2025

work page Pith review arXiv 2025
[1]

Large language models as software components: A taxonomy for llm-integrated applications,

I. Weber, “Large language models as software components: A taxonomy for llm-integrated applications,” arXiv preprint arXiv:2406.10300, 2024

arXiv 2024
[2]

Large language model agent: A survey on methodology, applications and challenges,

J. Luo, W. Zhang, Y . Yuan, Y . Zhao, J. Yang, Y . Gu, B. Wu, B. Chen, Z. Qiao, Q. Long et al. , “Large language model agent: A survey on methodology, applications and challenges,” arXiv preprint arXiv:2503.21460, 2025

arXiv 2025
[3]

Mare: Multi-agents col- laboration framework for requirements engineering,

D. Jin, Z. Jin, X. Chen, and C. Wang, “Mare: Multi-agents col- laboration framework for requirements engineering,” arXiv preprint arXiv:2405.03256, 2024

arXiv 2024
[4]

Social simulacra: Creating populated prototypes for social computing systems,

J. S. Park, L. Popowski, C. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Social simulacra: Creating populated prototypes for social computing systems,” in Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology , 2022, pp. 1–18

work page 2022
[5]

Can large language models transform computational social science?

C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang, “Can large language models transform computational social science?” Computational Linguistics, vol. 50, no. 1, pp. 237–291, 2024

work page 2024

Show all 16 references

[6]

Self-collaboration code generation via chatgpt,

Y . Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via chatgpt,” ACM Transactions on Software Engineering and Method- ology, vol. 33, no. 7, pp. 1–38, 2024

2024
[7]

Architectural tactics to achieve quality attributes of machine-learning-enabled systems: A systematic literature review,

V . Indykov, D. Strüber, and R. Wohlrab, “Architectural tactics to achieve quality attributes of machine-learning-enabled systems: A systematic literature review,” Journal of Systems and Software , p. 112373, 2025

2025
[8]

What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering,

F. Errica, G. Siracusano, D. Sanvito, and R. Bifulco, “What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering,” arXiv preprint arXiv:2406.12334 , 2024

2024 arXiv
[9]

Lora: Low-rank adaptation of large language models

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

2022
[10]

A comparison between three sdlc models waterfall model, spiral model, and incremental/iterative model,

A. Alshamrani and A. Bahattab, “A comparison between three sdlc models waterfall model, spiral model, and incremental/iterative model,” International Journal of Computer Science Issues (IJCSI), vol. 12, no. 1, p. 106, 2015

2015
[11]

J. Zhi, L. Liu, X. Chen, and T. Li, Software Requirements Engineering: Methods and Practice . Tsinghua University Press, 2023

2023
[14]

Automatic multi- level feature tree construction for domain-specific reusable artifacts management,

D. Jin, Z. Jin, N. Li, K. Yang, L. Li, and S. Guan, “Automatic multi- level feature tree construction for domain-specific reusable artifacts management,” arXiv preprint arXiv:2506.03946 , 2025

2025 arXiv
[16]

A first look at package- to-group mechanism: An empirical study of the linux distributions,

D. Jin, N. Li, K. Yang, M. Zhou, and Z. Jin, “A first look at package- to-group mechanism: An empirical study of the linux distributions,” in 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2025, pp. 299–309

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [12]

Status quo and problems of requirements engineering for machine learning: Results from an international survey,

A. P. S. Alves, M. Kalinowski, G. Giray, D. Mendez, N. Lavesson, K. Azevedo, H. Villamizar, T. Escovedo, H. Lopes, S. Bifflet al., “Status quo and problems of requirements engineering for machine learning: Results from an international survey,” in International Conference on Product-Focused Software Process Improvement . Springer, 2023, pp. 159–174

work page 2023

[2] [13]

Requirements engineering for machine learning: A review and reflection,

Z. Pei, L. Liu, C. Wang, and J. Wang, “Requirements engineering for machine learning: A review and reflection,” in 2022 IEEE 30th International Requirements Engineering Conference Workshops (REW) . IEEE, 2022, pp. 166–175

work page 2022

[3] [15]

Causal Models in Requirement Specifications for Machine Learning: A vision

H.-M. Heyn, Y . Mao, R. Weiss, and E. Knauss, “Causal models in requirement specifications for machine learning: A vision,” arXiv preprint arXiv:2502.11629, 2025

work page Pith review arXiv 2025

[4] [1]

Large language models as software components: A taxonomy for llm-integrated applications,

I. Weber, “Large language models as software components: A taxonomy for llm-integrated applications,” arXiv preprint arXiv:2406.10300, 2024

arXiv 2024

[5] [2]

Large language model agent: A survey on methodology, applications and challenges,

J. Luo, W. Zhang, Y . Yuan, Y . Zhao, J. Yang, Y . Gu, B. Wu, B. Chen, Z. Qiao, Q. Long et al. , “Large language model agent: A survey on methodology, applications and challenges,” arXiv preprint arXiv:2503.21460, 2025

arXiv 2025

[6] [3]

Mare: Multi-agents col- laboration framework for requirements engineering,

D. Jin, Z. Jin, X. Chen, and C. Wang, “Mare: Multi-agents col- laboration framework for requirements engineering,” arXiv preprint arXiv:2405.03256, 2024

arXiv 2024

[7] [4]

Social simulacra: Creating populated prototypes for social computing systems,

J. S. Park, L. Popowski, C. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Social simulacra: Creating populated prototypes for social computing systems,” in Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology , 2022, pp. 1–18

work page 2022

[8] [5]

Can large language models transform computational social science?

C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang, “Can large language models transform computational social science?” Computational Linguistics, vol. 50, no. 1, pp. 237–291, 2024

work page 2024