{"id":"79d8a9b6-46df-4a3e-92b3-7c30c477f607","arxiv_id":"2507.13095","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A conceptual framework reorganizes requirements engineering for pretrained-model-enabled systems into six activities, based on identified challenges of opaque capabilities, context sensitivity, and continuous evolution.","lead":"This paper proposes a conceptual framework for requirements engineering of systems built around large pretrained models, organizing the process into six activities such as capability discovery and prompt-as-specification. It is a vision paper that identifies open challenges and research directions, without empirical validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt-as-specification lacks the defined semantics it claims: the same prompt text can behave differently across model versions or inference settings, so validation and artifact management inherit an unsupported foundation.","rationale":"I read the paper as a research agenda rather than a claimed implementation; its central thesis is that ambiguous capability boundaries, context sensitivity, and continuous evolution make traditional requirements engineering assumptions insufficient. For the six-activity framework to serve as the foundation for a paradigm shift, the specification activity must be coherent. The weakest link is Section III-D's assertion that prompts can be executable specifications with explicit structure and defined semantics. The paper provides no semantic definition and no argument that one exists; its own research directions defer prompt specification languages to future work. This matters because the validation activity checks a 'prompt specification' against quality criteria, and the artifact management activity versions prompts as first-class artifacts. If the same prompt text can yield different behavior across model updates or inference settings, then neither validation nor management can be grounded in the prompt alone. A controlled reproducibility test on a current open model would settle whether the worry is real. I agree with the reader's identification of this assumption as the weakest. The concern does not overturn the paper's value as a vision; it supports a conditional verdict requiring the authors either to specify the semantics or to reposition Prompt-as-Specification as an explicitly open research challenge rather than a fully defined framework activity. Therefore I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":5830,"tokens_out":3900,"duration_ms":47882,"concrete_test":"Take the legal-document summarization example from Section III-D. Construct a fixed prompt specification and a fixed input suite of, say, 50 legal clauses. Execute the prompt at temperature 0 with a fixed seed against two consecutive revisions of the same open pretrained model, for instance Llama-3.1-8B before and after a LoRA fine-tune, or two released checkpoints in the same model series. Measure the fraction of outputs that differ on the same inputs. If materially different outputs occur, prompt text alone cannot carry defined semantics; the framework must either index Prompt-as-Specification by model version and inference configuration, or weaken its 'defined semantics' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's central activity, Prompt-as-Specification in Section III-D, requires that a prompt be elevated to a first-class, executable requirements specification with explicit structure and defined semantics. For that to be true, the prompt text must determine or at least reliably constrain the observable behavior of the system. But in a pretrained-model-enabled system, behavior is a function of the prompt, the model checkpoint, decoding parameters, and the runtime context. The paper itself emphasizes context sensitivity and continuous evolution in Section I, so the same prompt can produce different outputs after a model update or under different inference settings. Section III-D never specifies which of these factors is part of the 'prompt specification' or how the semantics are defined; Section IV lists 'prompt specification languages' as future research, confirming that stable semantics do not yet exist. Because validation in Section III-E and artifact management in Section III-F both operate on prompt specifications, the entire framework inherits this gap. This is not an internal contradiction; the paper honestly calls it an open problem. It is nevertheless the load-bearing assumption on which the framework's usefulness rests, since the paradigm shift is grounded in treating prompts as the executable expression of requirements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that software systems powered by large pretrained models have three distinctive characteristics—ambiguous capability boundaries, context-sensitive behavior, and continuous evolution—that fundamentally challenge traditional requirements engineering assumptions such as functional decomposability and behavioral predictability. It identifies six open challenges spanning elicitation, alignment, specification, validation, non-functional requirements, and artifact governance, then proposes a conceptual framework with six interconnected activities: Model Capability Discovery, User Intent Elicitation, Intent-Capability Alignment, Prompt-as-Specification, Prompt Specification Validation, and Model Artifacts Management. The paper also outlines five future research directions. The contribution is conceptual and prescriptive: no empirical evaluation, case study, or formal derivation is provided, and the framework's central Prompt-as-Specification activity relies on prompt semantics that the paper itself acknowledges are not yet defined.","tokens_in":6149,"tokens_out":4800,"duration_ms":55169,"significance":"If read as a research agenda, the paper provides a useful synthesis of known difficulties in requirements engineering for pretrained-model-enabled systems and a structured vocabulary for future work. It honestly names open problems such as prompt specification languages and emergent quality metrics, which is a strength. However, the paper does not ship machine-checked proofs, reproducible code, parameter-free derivations, or falsifiable predictions; its value depends on whether the proposed framework can be operationalized. The load-bearing assumption that prompts can serve as first-class executable requirements specifications is left undefined, so the contribution is a plausible starting point rather than a demonstrated method.","major_comments":[{"comment":"The central activity of the framework is 'Prompt-as-Specification,' which the paper describes as elevating prompts to 'first-class and executable requirements specifications' with 'explicit structure and defined semantics' (Section III-D). However, the observable behavior of a pretrained-model-enabled system depends not only on the prompt text but also on the model checkpoint, decoding parameters, and runtime context—a dependence the paper itself emphasizes in Section I. Section III-D does not specify which of these factors is part of the prompt specification or how its semantics are defined, and Section IV lists 'prompt specification languages' as future research, which confirms that stable semantics do not yet exist. Because Sections III-E and III-F validate and version the prompt specification as a unit, the entire framework inherits this gap. This is honestly framed as an open problem, but it is load-bearing: without a definition of prompt semantics, the framework's main innovation is a placeholder. I recommend either clarifying the semantics relative to a fixed model version and inference configuration or repositioning prompts as requirements artifacts whose behavior must be jointly versioned with the model, rather than as standalone executable specifications.","section":"III-D and IV"},{"comment":"The paper's central claim is that the three characteristics 'fundamentally challenge' traditional requirements engineering and 'necessitate a paradigm shift' (Sections I and V). This is an empirical claim about the prevalence and severity of the six challenges listed in Section II, but no survey, case study, systematic literature review, or other evidence is presented to support it. The cited literature (e.g., [12]) documents problems in requirements engineering for machine learning, but the paper does not show that these problems are widespread for pretrained-model-enabled systems specifically or that the proposed framework addresses them better than existing approaches. For a conceptual/vision paper this may be acceptable if the claims are softened, but as written the 'necessitate' language is stronger than the evidence. Please either add evidence (e.g., a case study or an expert evaluation of the framework) or reframe the contribution as a research agenda that motivates rather than demonstrates the paradigm shift.","section":"I and II"},{"comment":"The Model Capability Discovery activity assumes that model cards can be scraped, parsed, and normalized into a capability catalog (Section III-A). However, model cards are known in the literature to be incomplete, inconsistent, and often outdated; the paper does not discuss how the catalog should handle missing, conflicting, or stale information, nor what evidence a capability claim must carry. Since the capability catalog feeds Intent-Capability Mapping (III-C) and prompt selection (III-D), the reliability of the whole framework depends on the output of this activity. The paper should either acknowledge this limitation explicitly or describe a validation mechanism for the catalog itself.","section":"III-A"}],"minor_comments":[{"comment":"This catalog 'servers as' should be 'serves as.'","section":"III-A"},{"comment":"'If no required' should be 'If not required.'","section":"III-C"},{"comment":"'examples provides in the prompt' should be 'examples provided in the prompt.'","section":"III-E"},{"comment":"'pretrained models checkpoints' should be 'pretrained model checkpoints.'","section":"III-F"},{"comment":"Section references such as 'In section II' and 'In section III' should capitalize 'Section.'","section":"I"},{"comment":"Figure 1 labels 'Intent-PromptValidation' and 'Intent-Capability Mapping' differ from the corresponding section titles 'Prompt Specification Validation' and 'Intent-Capability Alignment'; the labels should be aligned with the text.","section":"Figure 1"},{"comment":"'from the following aspect' should be 'from the following aspects.'","section":"IV"}],"recommendation":"major_revision","confidential_remarks":"This is a vision/position paper rather than an empirical study. The editor may wish to consider whether such conceptual contributions are in scope for the journal; if so, the revision should still address the prompt-semantics gap and temper the paradigm-shift claim. Several references are from the same research group (e.g., [3], [14], [16]); this is not problematic, but the novelty relative to existing RE-for-ML work (e.g., [12]–[15]) should be made explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a position/vision paper, not a technical result. It reorganizes the requirements engineering lifecycle into six activities for pretrained-model-enabled systems and argues these systems break traditional RE assumptions. The writing is clear, the challenges are real, and the authors are honest that most of the machinery is future work. If you need a concise framing of why RE for pretrained models is hard, this is a decent pointer.\n\nWhat's actually new: not much as a result. Model Capability Discovery, User Intent Elicitation, Intent-Capability Mapping, Prompt-as-Specification, Behavior-Driven Validation, and Model Artifacts Management map almost one-to-one onto elicitation, analysis, specification, validation, and management from standard RE. Prior RE-for-ML surveys (citations [12], [13], [15]) already cover capability uncertainty, specification gaps, and validation difficulties. The paper's contribution is packaging these into one framework and giving each activity a name. That has some organizing value, but it is not a paradigm shift; it is a relabeling with a few new artifacts like capability catalogs and prompt specifications.\n\nThe paper does several things well. It motivates the challenges concretely, with examples like model cards lacking domain jargon detail. It cites the relevant survey literature. It does not overclaim empirical validation — there is none, but the paper presents itself as a research agenda, so that is acceptable. The six activities hang together coherently, and the figure is useful.\n\nSoft spots, in proportion. First, the central claim that these characteristics 'fundamentally challenge' and 'necessitate a paradigm shift' is asserted, not demonstrated. A skeptic could say that capability uncertainty and context sensitivity are quantitative extensions of existing RE problems, not a break from them. The paper does not engage with that counterargument. Second, the stress-test concern about Prompt-as-Specification is real: Section III-D says prompts can have 'defined semantics' but never says how, and Section IV lists prompt specification languages as future research. That is not an internal contradiction — the paper admits the gap — but it means the framework's linchpin activity is currently aspirational. Validation and artifact management inherit that open problem. This is a limitation, not a fatal flaw, for a vision paper.\n\nWho this is for: researchers new to RE for ML or pretrained models will get a useful orientation. Practitioners looking for methods will not find anything usable yet. As a peer reviewer, I would not reject it for being unvalidated — it is explicitly a framework proposal — but I would ask the authors to soften the paradigm-shift language and to position the work more explicitly as a synthesis of existing RE-for-ML research rather than a break from it.\n\nRecommendation: send it to peer review. It is a competent, honest vision paper in a growing area. A good referee will push for sharper novelty claims and a more careful treatment of prompt semantics, but the paper deserves that referee time.","headline":"A clear, honest research agenda for RE with pretrained models, but the 'paradigm shift' framing oversells a repackaging of known RE-for-ML ideas; still deserves a serious look.","tokens_in":6484,"tokens_out":1106,"would_cite":false,"duration_ms":14697,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that large pretrained models force requirements engineering to abandon its core assumptions and reorganize around six activities.","keywords":["pretrained models","requirements engineering","large language models","prompt-as-specification","model capability discovery","intent-capability alignment","non-functional requirements","model artifacts management"],"falsifier":"Run a conventional requirements process on a deployed pretrained-model chatbot, enumerate expected inputs and outputs, then update the underlying model several times. If behavior stays within the enumerated boundaries and the original specification remains testable and complete across updates, the paper's central claim that pretrained-model-enabled systems break traditional requirements assumptions would be contradicted.","tokens_in":5657,"feed_emoji":"🧠","tokens_out":6823,"duration_ms":75886,"temperature":0.7,"pith_summary":"The paper tries to establish that requirements engineering, as practiced for deterministic software, will not carry over to systems whose behavior is driven by large pretrained models. It identifies three emergent properties—ambiguous capability boundaries, context-sensitive behavior, and continuous evolution—and argues they break the assumptions that requirements can be decomposed, predicted, and exhaustively verified. To replace the broken assumptions, it proposes a six-activity lifecycle ranging from model capability discovery to model artifacts management. If the argument holds, requirements engineering for such systems will center on prompt specifications, statistical validation, and continuous evolution governance rather than on fixed feature sets.","feed_headline":"Pretrained-model systems need a new requirements lifecycle","feed_subtitle":"Paper proposes six activities that center on prompt specifications, capability catalogs, and versioned model artifacts.","key_machinery":"Key machinery is the reorganized six-activity lifecycle, with the Prompt-as-Specification activity at its center: a prompt is treated as a first-class executable requirements specification that has explicit structure and defined semantics, created by filling model prompt templates from a user intent list. Around it sit the capability catalog—a structured repository of model capabilities, limitations, risks, and costs built from model-card information—and the mapping engine that pairs intents to catalog capabilities. The framework's claim to work is that every other activity feeds or checks these specifications: capability discovery grounds elicitation, intent-capability mapping decides where a model is needed, validation checks prompt quality statistically, and artifacts management version-stamps prompts, checkpoints, and data so evolution stays auditable.","core_discovery":"The paper's central claim is that software systems whose core behavior comes from pretrained models possess three emergent properties—ambiguous capability boundaries, context-sensitive behavior, and continuous evolution—and that these properties invalidate the long-standing requirements-engineering assumptions of functional decomposability and behavioral predictability. From this it concludes that requirements engineering for such systems cannot be a one-time specification activity but must become a continuous lifecycle. The proposed response is a six-activity framework: discover and catalog model capabilities, elicit user intent against that catalog, map intents to candidate models, turn prompts into first-class executable specifications, validate those specifications statistically and with human review, and version and trace prompts, checkpoints, and data. The paper presents this as a foundation for research, not as a validated method.","pith_inferences":["The framework is presented as a research agenda, and the authors do not evaluate it end-to-end; an immediate extension is to instantiate all six activities on one product and measure whether prompt drift, rollback effort, and audit gaps decrease.","One consequence the authors leave implicit is that requirement satisfaction becomes probabilistic: a prompt specification can only promise behavior within a tolerance, so acceptance criteria should be statistical bounds rather than yes/no tests.","The capability catalog could in principle be built by automatically scraping and normalizing model cards, which would turn intent-capability mapping into a search-and-recommendation problem rather than an expert judgment task.","If prompt behavior cannot be given stable, verifiable semantics, the Prompt-as-Specification activity would need to be reframed as continuous test generation, and the rest of the framework would have to lean more heavily on validation and artifact management."],"forward_implications":["Requirements for pretrained-model-enabled systems will be written as structured prompts plus quality checks, not as exhaustive functional specifications.","Validation shifts from exhaustive input-output testing to statistical testing, scenario simulation, and human-in-the-loop review.","Model updates, prompts, and tuning data must be versioned and traced like code, to prevent configuration drift and keep audits possible.","New non-functional requirements—hallucination rate, fairness, explainability—need standard definitions and metrics before they can be engineered.","Selection of models becomes an explicit requirements activity: intents are matched to documented capabilities before any prompt is written."],"supporting_citations":[{"why":"Establishes that large language models are now treated as software components, which is the premise for the paper's object of study.","marker":"[1]"},{"why":"Characterizes the deterministic-logic assumptions of traditional systems that the paper argues no longer hold.","marker":"[7]"},{"why":"Documents sensitivity to prompt changes, providing evidence for the context-sensitive behavior at the heart of the paper's argument.","marker":"[8]"},{"why":"Supplies the view of requirements as deterministic specifications that the paper challenges.","marker":"[11]"},{"why":"Surveys existing problems in requirements engineering for machine learning, motivating the six open challenges.","marker":"[12]"},{"why":"Reviews requirements engineering for machine learning and supports the described intent-capability misalignment.","marker":"[13]"},{"why":"Motivates the specification-quality gap by proposing causal models as a remedy for machine-learning requirement specifications.","marker":"[15]"}],"fun_headline_variants":["Pretrained-model systems need continuous requirements engineering","Rethinking requirements for AI-model-powered software","Model-driven systems demand a new requirements lifecycle","Requirements engineering must adapt to pretrained models","Beyond deterministic specs: RE for model-based systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a prompt can serve as a stable, executable requirements specification with explicit structure and defined semantics; if prompt behavior cannot be pinned down well enough to specify and verify, the central activity of the framework does not have a foundation.","fun_headline_variants_meta":{"raw":{"variants":["Pretrained-model systems need continuous requirements engineering","Rethinking requirements for AI-model-powered software","Model-driven systems demand a new requirements lifecycle","Requirements engineering must adapt to pretrained models","Beyond deterministic specs: RE for model-based systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000467,"raw_usage":{"total_tokens":2263,"prompt_tokens":817,"completion_tokens":1446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":1379}},"tokens_in":433,"tokens_out":1446,"duration_ms":12176,"temperature":1.0,"reasoning_tokens":1379,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:30:11.552555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a conventional requirements process on a deployed pretrained-model chatbot, enumerate expected inputs and outputs, then update the underlying model several times. If behavior stays within the enumerated boundaries and the original specification remains testable and complete across updates, the paper's central claim that pretrained-model-enabled systems break traditional requirements assumptions would be contradicted.","supporting_citations":[{"cited_title":"Architectural tactics to achieve quality attributes of machine-learning-enabled systems: A systematic literature review,","cited_arxiv_id":null,"evidence_quote":"Characterizes the deterministic-logic assumptions of traditional systems that the paper argues no longer hold."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the view of requirements as deterministic specifications that the paper challenges."},{"cited_title":"Status quo and problems of requirements engineering for machine learning: Results from an international survey,","cited_arxiv_id":null,"evidence_quote":"Surveys existing problems in requirements engineering for machine learning, motivating the six open challenges."},{"cited_title":"Requirements engineering for machine learning: A review and reflection,","cited_arxiv_id":null,"evidence_quote":"Reviews requirements engineering for machine learning and supports the described intent-capability misalignment."},{"cited_title":"Causal Models in Requirement Specifications for Machine Learning: A vision","cited_arxiv_id":"2502.11629","evidence_quote":"Motivates the specification-quality gap by proposing causal models as a remedy for machine-learning requirement specifications."}],"review_version":1}