Work in Progress: AI-Powered Engineering-Bridging Theory and Practice

Ilya Dikman; Michael Winokur; Natan Levy; Oz Levy

REVIEW 4 major objections 5 minor 17 references

Work in Progress: AI-Powered Engineering-Bridging Theory and Practice

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes that generative AI can classify system requirements against the systems-engineering 'good requirement' criteria as reliably as experienced engineers, and lays out a blind expert-comparison experiment to test that claim.

desk verdict A clear, honest WIP proposal with no results yet; the missing expert-expert agreement check is a real design gap, but the plan is sensible for a workshop track. read the letter →

arxiv 2502.04256 v1 pith:2M2LALHM submitted 2025-02-06 eess.SY cs.SEcs.SY

Oz Levy , Ilya Dikman , Natan Levy , Michael Winokur This is my paper · ORCID

classification eess.SYcs.SEcs.SY

keywords generativeAIsystemsengineeringrequirementsgoodrequirementcriteriaclassificationfunctionalandnon-functionalCohen'skappaevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This work-in-progress paper aims to establish that generative AI can analyze system requirements against the systems-engineering 'good requirement' criteria (essential, independent, unambiguous, complete, singular, feasible, verifiable) and classify each requirement as functional or non-functional at a level that agrees with experienced systems engineers. The authors argue that if this holds, an AI tool could flag poorly written requirements with explanations, generate test specifications from the classifications, and serve as a teaching aid for engineering students. To test the claim, they propose running three large language models (GPT-4, Claude, and Llama) on 107 real requirements from a medical-equipment tracking system and comparing the models' outputs with a blind panel of human experts using Cohen's kappa. The paper reports no results yet; its current contribution is the evaluation design and the argument that measuring human-AI agreement is the correct way to judge whether AI matches or only assists an engineer's judgment.

What carries the argument

The load-bearing mechanism is a two-rater comparison design. Three large language models (GPT-4, Claude, and Llama) and a panel of experienced systems engineers independently classify the same 107 requirements, 31 stakeholder and 76 optimized system requirements from the DR TOOL medical-equipment tracking project, against the seven 'good requirement' criteria and the functional/non-functional split; Cohen's kappa measures agreement between each model and the human benchmark, and discrepancies feed an iterative refinement loop.

What would settle it

Run the described experiment: have GPT-4, Claude, and Llama classify all 107 DR TOOL requirements against the seven 'good requirement' criteria and the functional/non-functional split, have a blind panel of experienced systems engineers do the same, and compute Cohen's kappa for each model-expert pair; if all three kappas are at or near zero, the claim that AI can match a systems engineer's judgment fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that large language models can apply the seven 'good requirement' criteria and the functional/non-functional distinction to real engineering requirements, producing classifications and justifications that agree with experienced systems engineers. The authors expect some agreement but predict that AI will show hallucinations and contextual misunderstandings that engineers catch intuitively; the proposed experiment is designed to measure exactly where agreement breaks down. The intended discovery, if the experiment goes as hoped, is that a generative-AI workflow can serve as a reliable screening and teaching tool for requirements quality, with human engineers remaining the benchmark.

Load-bearing premise

The 107 requirements from one medical-equipment tracking project are treated as representative enough of system requirements in general that agreement with experts on this dataset would tell us something general about AI's classification ability.

Editorial extensions

If this is right

If the comparison shows strong agreement, systems engineers could delegate the first pass of requirements-quality screening to an AI and spend their time on the flagged items.
If AI can reliably separate functional from non-functional requirements, test specifications can be generated earlier in the design process, before detailed system design is locked.
If the AI's justifications match expert reasoning, the same tool can be used in engineering education to show students why a requirement fails a quality criterion.
If agreement is weak, the refinement loop is intended to show which criteria and which requirement types need model tuning or human oversight.
Organizations could use the measured agreement to decide where AI assistance is safe and where a human expert must stay in the loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the same protocol could be applied to requirements from defense, transportation, or software projects to test whether performance on the medical-equipment dataset generalizes.
A future extension could separate label agreement from explanation agreement, since a model and an expert can choose the same criterion and still disagree about the underlying defect.
If the method proves reliable, it could scale to large requirements documents where manual expert review is expensive, turning the comparison into a practical audit tool.
The educational claim could be tested by measuring whether students who train on AI-flagged requirements write better requirements than a control group.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a work-in-progress proposal, not a paper with results. The one thing to know up front: nothing is evaluated or reported, so every claim about AI performance is promissory. That is not a fatal flaw given the explicit WIP framing, but it means the paper should not be judged as a completed study.

What is actually useful: the authors propose combining three current LLMs (GPT-4, Claude, Llama) with INCOSE's seven "good requirement" criteria, and they plan to validate the approach on a real dataset from their prior DR TOOL project. That dataset, 107 requirements from an RFID-based medical equipment tracking system, gives the plan some concreteness. The design is straightforward: AI classifies, a panel of experts classifies blind, and agreement is measured with Cohen's kappa. The paper is clearly written, cites prior ML-based requirements classification work (Tamai and Anzai, Cheligeer et al.), and is transparent that this is an expected contribution rather than a completed one.

The soft spots are the usual ones for a proposal, plus one the authors have not yet addressed. First, there is no data and no analysis. For a WIP that is acceptable, but the abstract overstates the case by using present tense ("The AI does not just classify requirements but also explains why...") when no AI has run yet. Second, the protocol lacks an expert-expert agreement measure. Cohen's kappa between AI and experts is only interpretable if the experts agree with each other; without an inter-rater reliability check on the expert panel, a low AI-expert kappa could mean the AI is wrong or the reference standard is unstable. That is a fixable design gap, but it should be added before data collection. Third, the dataset is a single project in one domain, so any quantitative result will be domain-specific. That is fine for a pilot, but the paper should not imply general conclusions without cross-domain validation.

The citation pattern looks fine. Using their own prior DR TOOL dataset is transparent and not a problem. The self-citation in [11] is to the source of the dataset, which is appropriate.

Who is this for? Someone running a WIP/emerging-results session or a graduate seminar on applying LLMs to requirements engineering could get value from the framing. I would not cite it in my own work yet, because there is no result to cite. If the authors bring this back with actual experimental data and the expert-expert kappa added, it could become a legitimate empirical paper. As it stands, I would not send it to a full research track, but it is appropriate for a workshop or a WIP track that explicitly solicits proposals.

Referee Report

4 major / 5 minor

Summary. The manuscript is a work-in-progress proposal for using generative AI models (GPT-4, Claude Sonnet, Llama) to automate the classification and quality assessment of systems engineering requirements according to INCOSE's 'good requirement' criteria, to distinguish functional from non-functional requirements, and to generate test specifications. The proposed method involves a dataset of 107 requirements from a single RFID-based medical-equipment tracking project (DR TOOL), independent blind classification by experienced systems engineers, and comparison of AI and expert classifications using Cohen's kappa, followed by iterative refinement of the AI models. The paper reports no experimental results, no data, and no computed statistics; all evaluation steps are described as planned future work.

Significance. If the proposed evaluation were carried out and the results were positive, the work would address a practical gap in requirements engineering: automated, explainable classification of requirement quality against an established standard such as INCOSE's. The study's strengths are its use of a real-world requirements dataset, the explicit choice of three state-of-the-art LLMs, and a planned blind expert benchmark with a statistical agreement measure. However, the paper currently provides no evidence for its central claim, only a research design. The discussion sections (IV) and conclusion (VI) go beyond what the manuscript supports by describing the study as 'demonstrating' a practical transition that has not yet occurred. The planned benchmark also needs a critical addition—inter-rater reliability among the expert panel—without which any future AI-expert kappa is not interpretable.

major comments (4)

[Section II-B (Steps 1–5) and Section III] The paper reports no experimental results. The central claim that generative AI can automate requirement analysis and that this can be validated by comparing AI outputs with expert judgments is not supported by any data, model output, or computed agreement statistic. All five steps in the research design are described in future/proposed terms, and Section III itself states only 'expected' contributions. For a full journal paper this is a blocking issue: the manuscript must either present completed experiments with at least one AI model and a comparison to expert classifications, or be explicitly reframed as a research proposal/vision paper with the scope limited to presenting a protocol.
[Section II-B, Steps 3 and 4] The validation protocol does not specify any measure of expert-expert agreement. Step 3 says a panel of experienced systems engineers will independently classify the dataset, and Step 4 compares AI classifications with expert classifications using Cohen's kappa. Cohen's kappa is defined for two raters, not for a panel, and more importantly, without an expert-expert agreement measure (e.g., Fleiss' kappa or pairwise Cohen's kappa) the expert benchmark is not established as a stable ground truth. INCOSE criteria such as 'unambiguous' and 'complete' are inherently interpretive; if experts disagree among themselves, a low AI-expert kappa does not indicate AI failure, and a high AI-expert kappa could be misleading. Please add an explicit expert-expert agreement analysis and a stratification of the dataset by expert difficulty.
[Section II-C] The dataset consists of 107 requirements from a single RFID-based medical equipment tracking project (DR TOOL). The paper generalizes its claims to 'AI-powered engineering' without justifying the representativeness of this dataset. A positive or negative result on this small, domain-specific corpus would not support conclusions about systems engineering in general. Either the dataset needs to be expanded to multiple projects/domains, or the paper must explicitly restrict its claims to this domain and present the work as a case study rather than a general evaluation.
[Section II-A and II-B] The protocol does not operationalize how each INCOSE criterion is assessed. For instance, is each requirement rated as pass/fail or on a Likert scale? What prompt templates and few-shot examples are used for each LLM? How is a 'hallucination' identified and recorded? Without these details, the proposed comparison is not reproducible and the future AI-expert comparison cannot be interpreted. This load-bearing detail should be specified before the experiment is run.

minor comments (5)

[Section II-C, first sentence] The verb tense shifts from the future tense of Step 2 ('will analyze') to the past tense in Section II-C ('The models were applied'), creating ambiguity about whether the experiment has already been conducted; please standardize the tense.
[Figures 1 and 2] The captions for Fig. 1 and Fig. 2 are present, but the figures themselves are not included in the text; please provide the actual diagrams or remove the references.
[References] Reference [13] lacks a year and appears to be a course report; please provide a permanent citation or remove it. Reference [15] is a preliminary review of ChatGPT rather than a technical description of GPT-4; citing the GPT-4 technical report or model card would be more appropriate.
[Section II-A] The paper states that the study focuses on seven out of nine INCOSE parameters but does not name the two omitted parameters or explain the selection; please add a brief justification.
[Section IV-A] The phrase 'this study demonstrates how AI can effectively transition from a research tool to a practical asset' overclaims relative to the absence of results; please soften to 'proposes' or 'aims to demonstrate.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper is a proposal with no derived results, and the only self-citation supplies input data rather than evidential support.

full rationale

The paper is explicitly a work-in-progress proposal and reports no experimental results, equations, fitted parameters, or derived predictions that could reduce to their own inputs. The DR TOOL dataset is introduced in Section II-C and cited to the authors' prior work [11], but it is used solely as a test input for the proposed evaluation, not as evidence that AI classification works; the proposed benchmark is an independent blind classification by experienced systems engineers (Section II-B, step 3). Therefore the self-citation is not load-bearing in any argumentative sense. The lack of an expert-expert agreement measure and the small single-project dataset are genuine validity concerns, but they concern the future experiment's conclusiveness and generalizability, not circularity. No circular step can be exhibited from the paper's text, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or free parameters. It rests on domain assumptions about INCOSE standards, the representativeness of a self-created dataset, and the undemonstrated capability of LLMs. No data or derivations are provided.

assumptions (4)

domain assumption The INCOSE 'good requirement' criteria are a valid standard for evaluating requirement quality
The paper derives its evaluation framework from INCOSE [8] without questioning its validity as ground truth.
domain assumption Experienced systems engineers' classifications provide an unbiased benchmark
The comparative analysis in Section II-B.4 assumes expert classification is the gold standard for AI comparison.
domain assumption The DR TOOL dataset (107 requirements) is representative of real-world system requirements
Section II-C states the dataset comes from a single project, which may limit generalizability.
domain assumption State-of-the-art LLMs can perform text classification tasks with sufficient accuracy for this use
The paper assumes GPT-4, Claude, and Llama can analyze requirements, but this is the very hypothesis to be tested, so assuming feasibility is a premise of the study design.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Work in Progress: AI-Powered Engineering-Bridging Theory and Practice." pith.science (2026). https://pith.science/paper/2M2LALHM

@misc{pith2026250204256,
  author       = {Pith},
  title        = {Pith review of: Work in Progress: AI-Powered Engineering-Bridging Theory and Practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2M2LALHM}},
  note         = {Machine review of arXiv:2502.04256}
}

read the original abstract

This paper explores how generative AI can help automate and improve key steps in systems engineering. It examines AI's ability to analyze system requirements based on INCOSE's "good requirement" criteria, identifying well-formed and poorly written requirements. The AI does not just classify requirements but also explains why some do not meet the standards. By comparing AI assessments with those of experienced engineers, the study evaluates the accuracy and reliability of AI in identifying quality issues. Additionally, it explores AI's ability to classify functional and non-functional requirements and generate test specifications based on these classifications. Through both quantitative and qualitative analysis, the research aims to assess AI's potential to streamline engineering processes and improve learning outcomes. It also highlights the challenges and limitations of AI, ensuring its safe and ethical use in professional and academic settings.

Figures

Figures reproduced from arXiv: 2502.04256 by the authors.

**Figure 2.** The Dr. Tool System Diagram security. The evaluation not only enables a direct comparison between AI-generated classifications and those performed manually by experienced systems engineers but also provides insights into areas where AI models can be refined and improved. Furthermore, this process assists system engineers by identifying problematic requirements, offering justifications and actionable insights to enh… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

[11]

Management and Detection System for Medical Surgical Equipment

A. Hadar, N. Levy, and M. Winokur, ”Management and Detec- tion System for Medical Surgical Equipment,” *arXiv preprint* arXiv:2211.02351, 2022

work page Pith review arXiv 2022
[1]

[Online]

Stanford University, *AI Index 2024 Annual Report*, Stanford Institute for Human- Centered Artificial Intelligence (HAI), 2024. [Online]. Available: https://aiindex.stanford.edu/wp-content/uploads/2024/05/HAI AI-Index-Report-2024.pdf

work page 2024
[2]

G. Tesla. ”Artificial intelligence at Tesla” [Online]. Available: https://www.tesla.com/AI

work page
[3]

[Online]

NASA, ”Artificial intelligence”. [Online]. Available: https://www.nasa.gov/artificial- intelligence/

work page
[4]

O. J. Adeyeye and I. Akanbi, ”Artificial intelligence for systems engi- neering complexity: A review on the use of AI and machine learning algorithms,” *Comput. Sci. IT Res. J.*, vol. 5, no. 4, pp. 787-808, 2024

work page 2024
[5]

Baidya, S

S. Baidya, S. K. Das, M. H. Uddin, C. Kosek, and C. Summers,”Digital twin in safety- critical robotics applications: Opportunities and chal- lenges”, in *2022 IEEE Int. Performance, Computing, and Communi- cations Conf. (IPCCC)*, Nov. 2022, pp. 101- 107

work page 2022
[6]

Cheligeer, J

C. Cheligeer, J. Huang, G. Wu, N. Bhuiyan, Y . Xu, and Y . Zeng, ”Machine learning in requirements elicitation: A literature review,” *AI EDAM*, vol. 36, e32, 2022

work page 2022
[7]

Tamai and T

T. Tamai and T. Anzai, ”Quality Requirements Analysis with Machine Learning,” in *ENASE*, Mar. 2018, pp. 241-248

work page 2018

Show all 17 references

[8]

John Wiley and Sons, 2023

INCOSE, Ed., *INCOSE systems engineering handbook*. John Wiley and Sons, 2023

2023
[9]

Maleki, B

N. Maleki, B. Padmanabhan, and K. Dutta, ”AI hallucinations: a mis- nomer worth clarifying,” in *2024 IEEE Conf. on Artificial Intelligence (CAI)*, June 2024, pp. 133-138

2024
[10]

Mart ´ınez, J

G. Mart ´ınez, J. Conde, P. Reviriego, E. Merino-G´omez, J. A. Hern´andez, and F. Lombardi, ”How many words does ChatGPT know? The answer is ChatWords,” *arXiv preprint* arXiv:2309.16777, 2023

2023 arXiv
[12]

Perez-Cerrolaza, J

J. Perez-Cerrolaza, J. Abella, M. Borg, C. Donzella, J. Cerquides, F. J. Cazorla, and J. L. Flores, ”Artificial intelligence for safety-critical systems in industrial and transportation domains: A survey,” *ACM Comput. Surveys*, vol. 56, no. 7, pp. 1-40, 2024. doi: 10.1145/3626314

2024 doi
[13]

Mahajann, M

S. Mahajann, M. Mulchandani, and S. Ajani, ”Automating Test Case Creation Using Natural Language Processing,” Dept. Comput. Sci. Eng., Jhulelal Inst. Technol., RTMNU, Nagpur, [n.d.]
[14]

Ricca, A

F. Ricca, A. Marchetto, and A. Stocco,”AI-based test automation: A grey literature analysis,” in *2021 IEEE Int. Conf. Software Testing, Verification and Validation Workshops (ICSTW)*, Apr. 2021, pp. 263- 270

2021
[15]

K. I. Roumeliotis and N. D. Tselikas, ”ChatGPT and Open-AI Models: A Preliminary Review,” Future Internet, vol. 15, no. 6, p. 192, 2023

2023
[16]

2023, revised Jun

Anthropic, ”Claude 3.5 Sonnet Model Card Addendum,” Dec. 2023, revised Jun. 2024. [Online].https://tinyurl.com/mr3vtvya

2023
[17]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al., ”Llama: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302.13971, 2023

2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [11]

Management and Detection System for Medical Surgical Equipment

A. Hadar, N. Levy, and M. Winokur, ”Management and Detec- tion System for Medical Surgical Equipment,” *arXiv preprint* arXiv:2211.02351, 2022

work page Pith review arXiv 2022

[2] [1]

[Online]

Stanford University, *AI Index 2024 Annual Report*, Stanford Institute for Human- Centered Artificial Intelligence (HAI), 2024. [Online]. Available: https://aiindex.stanford.edu/wp-content/uploads/2024/05/HAI AI-Index-Report-2024.pdf

work page 2024

[3] [2]

G. Tesla. ”Artificial intelligence at Tesla” [Online]. Available: https://www.tesla.com/AI

work page

[4] [3]

[Online]

NASA, ”Artificial intelligence”. [Online]. Available: https://www.nasa.gov/artificial- intelligence/

work page

[5] [4]

O. J. Adeyeye and I. Akanbi, ”Artificial intelligence for systems engi- neering complexity: A review on the use of AI and machine learning algorithms,” *Comput. Sci. IT Res. J.*, vol. 5, no. 4, pp. 787-808, 2024

work page 2024

[6] [5]

Baidya, S

S. Baidya, S. K. Das, M. H. Uddin, C. Kosek, and C. Summers,”Digital twin in safety- critical robotics applications: Opportunities and chal- lenges”, in *2022 IEEE Int. Performance, Computing, and Communi- cations Conf. (IPCCC)*, Nov. 2022, pp. 101- 107

work page 2022

[7] [6]

Cheligeer, J

C. Cheligeer, J. Huang, G. Wu, N. Bhuiyan, Y . Xu, and Y . Zeng, ”Machine learning in requirements elicitation: A literature review,” *AI EDAM*, vol. 36, e32, 2022

work page 2022

[8] [7]

Tamai and T

T. Tamai and T. Anzai, ”Quality Requirements Analysis with Machine Learning,” in *ENASE*, Mar. 2018, pp. 241-248

work page 2018