REVIEW 3 cited by
Exploring LLMs for Verifying Technical System Specifications Against Requirements
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Requirements engineering is a knowledge intensive process and crucial for the success of engineering projects. The field of knowledge-based requirements engineering (KBRE) aims to support engineers by providing knowledge to assist in the elicitation, validation, and management of system requirements. The advent of large language models (LLMs) opens new opportunities in the field of KBRE. This work experimentally investigates the potential of LLMs in requirements verification. Therein, LLMs are provided with a set of requirements and a textual system specification and are prompted to assess which requirements are fulfilled by the system specification. Different experimental variables such as system specification complexity, the number of requirements, and prompting strategies were analyzed. Formal rule-based systems serve as a benchmark to compare LLM performance to. Requirements and system specifications are derived from the smart-grid domain. Results show that advanced LLMs, like GPT-4o and Claude 3.5 Sonnet, achieved f1-scores between 79 % and 94 % in identifying non-fulfilled requirements, indicating potential for LLMs to be leveraged for requirements verification.
Forward citations
Cited by 3 Pith papers
-
Can LLMs Generate User Stories and Assess Their Quality?
LLMs generate user stories with human-like coverage and style but lower diversity, and they assess semantic quality well when given explicit codebook criteria, though human oversight is still needed.
-
Leveraging LLMs for Formal Software Requirements -- Challenges and Prospects
LLM-based formalisation of software requirements is promising but faces five persistent challenges; the proposed VERIFAI framework plans to address them with human-in-the-loop and tool-neutral pipelines.
-
A Short Survey on Formalising Software Requirements using Large Language Models
A survey summarizing 35 papers on using LLMs to formalize software requirements, but it contains no new experimental results and its classification tables have errors.
Discussion (0). Sign in to comment.