Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Description and Comparative Analysis of QuRE: A New Industrial Requirements Quality Dataset

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read QuRE: a new public dataset of 2,111 real industrial requirements, each annotated with a weak-word label and an expert defect judgment.

desk verdict QuRE is a genuinely useful public industrial requirements dataset, but the label-quality claim rests on an unmeasured assertion; still worth refereeing. read the letter →

arxiv 2508.08868 v1 pith:LZWI5GNX submitted 2025-08-12 cs.SE

Henning Femmer , Frank Houdek , Max Unterbusch , Andreas Vogelsang This is my paper · ORCID
classification cs.SE
keywords requirementsqualitydatasetnaturallanguagesmellsweakwordsindustrialbenchmarksyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QuRE is a new public dataset of 2,111 natural-language requirements drawn from real automotive specifications, each annotated with a weak-word label and an expert judgment of whether that word marks a quality defect. The labels come from an industrial review process used in a contractual setting for five years, which the authors describe as giving the annotations industry quality. The paper argues that QuRE is the largest requirements-quality dataset publicly available to date and that it is linguistically typical of real-world requirements, unlike three sets of LLM-generated requirements that are markedly simpler, shorter, and less dense. If that claim holds, researchers gain a common, realistic benchmark for training and evaluating automatic quality-assessment tools, and a reference point for judging whether synthetic requirements can stand in for industrial ones.

What carries the argument

The dataset itself is the central artifact: 2,187 rows covering 2,111 unique requirements, each paired with one of 23 weak-word labels and a binary ok/defect judgment. The weak-word taxonomy—originally a 103-word catalog reduced to 23 umbrella categories—carries the analytic load, since it converts the abstract notion of a requirements smell into a concrete, labeled signal that tools can be trained and tested on. The defect/ok column is what makes it a quality benchmark rather than just a text corpus.

What would settle it

Have two or more independent experts re-annotate all 2,111 requirements under the same 23 weak-word categories and compare their defect/ok judgments; if agreement is near chance or systematically opposite to the original labels, the benchmark's ground truth collapses.

Watch

Extended reading notes

Core claim

On its own terms, the paper's contribution is the release of QuRE: a CSV dataset of 2,111 unique English requirements from a single automotive domain, with four columns—id, requirement text, weak_word (an umbrella label such as 'detail' or 'close'), and defect (ok or defect). The weak-word labels derive from a catalog used to build regression tests for requirements-smell detectors, and the defect judgments were made by up to three industry experts, with unclear cases negotiated. The dataset was used for five years under a service-level agreement as a benchmark that automated analyses had to match, which the authors cite as evidence of industry-grade label quality. A comparative linguistic an

Load-bearing premise

The dataset's value as a benchmark rests on the assumption that the industrial defect/ok labels are accurate and consistent enough to serve as ground truth, yet the paper reports no inter-rater agreement metrics and acknowledges individual labels may be debatable or incorrect from an academic viewpoint.

Editorial extensions

If this is right

  • Automated requirements-quality tools can be benchmarked on a common, realistic set instead of small or private collections.
  • Machine-learning models for detecting weak-word misuse and related lexical smells can be trained and validated on 2,111 industry-authored sentences.
  • Researchers can run replication studies and compare new taxonomies of requirements defects against the existing labels.
  • The reported contrast between QuRE and the three synthetic sets gives a quantitative baseline for asking when LLM-generated requirements are realistic enough for experiments.
  • The dataset's documented provenance and context make it a candidate seed for a community-driven requirements-dataset repository.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the authors could not retrieve inter-rater agreement metrics, benchmarks built on QuRE should report how results shift under label noise; a re-annotation study would be the natural next experiment.
  • The sampling deliberately over-represents requirements containing weak words and defects, so QuRE should not be used to estimate how often quality defects occur in ordinary automotive specifications.
  • The framework of 23 weak-word categories suggests a testable extension: prompt LLMs with examples from QuRE and measure whether the generated requirements move toward real-world lexical density and parse-tree depth.
  • If QuRE's labels are accepted as ground truth, the same annotation schema could be applied to requirements in avionics, defense, or healthcare, testing how far automotive writing style transfers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces QuRE, a publicly released dataset of 2,111 industrial requirements from Mercedes-Benz, each annotated with a weak-word indicator and a human-assigned defect/ok label. The authors describe the dataset's provenance, including a five-year use in an industrial SLA, provide descriptive linguistic statistics, and compare QuRE with existing requirements datasets and LLM-generated synthetic requirements. They argue that QuRE is the largest requirements-quality dataset to date, that its labels are of 'industry quality,' and that its linguistic properties align with other real-world requirements datasets while synthetic data are linguistically distinct.

Significance. If the dataset and its annotations are reliable, QuRE is a valuable community resource: it is substantially larger than most publicly available requirements-quality datasets, it comes from a real industrial context, and it is accompanied by release code and documentation. The paper is transparent about provenance and limitations, and the authors explicitly call for community-driven dataset infrastructure. The comparative analysis of synthetic versus real requirements is a useful starting point for researchers evaluating LLM-generated test data. However, the central value Proposition—usable ground-truth defect labels—rests on expert annotations whose consistency is not measured, and the dataset covers only weak-word-related defects from a specific sampling strategy. These caveats do not invalidate the resource, but they bound what can be claimed.

major comments (4)
  1. [Section III-B] The statement 'the annotations have industry quality' is presented as a conclusion from the dataset's industrial use, but the section immediately acknowledges that no inter-rater agreement metrics could be retrieved. Contractual use in an SLA demonstrates that the labels were useful for regression testing in a specific industrial setting, but it does not measure label accuracy, consistency, or agreement with any external standard. Since the dataset's primary purpose is to serve as ground truth for evaluating quality-assessment tools, this missing evidence is load-bearing. Please either provide inter-rater agreement statistics (if recoverable), or soften the claim from 'industry quality' to 'labels resulting from a negotiated industrial review process' and explicitly state in the abstract and conclusion that label reliability is unquantified.
  2. [Section III-A Step 4] The preprocessing step excluded 31 rows with inconsistent data (a defect label with no weak word) and 33 rows where the human labeler was undecided, totaling 64 rows (<3%). While the exclusion is disclosed, its effect on benchmark conclusions is not analyzed. These excluded cases may indicate real label noise or ambiguity; removing them could systematically improve apparent label quality. Please report the characteristics of the excluded rows and, if possible, analyze how inclusion/exclusion of these cases affects downstream benchmarking (e.g., agreement with a simple weak-word heuristic).
  3. [Section V-C and Section VI] The paper acknowledges in Section V-C that the dataset 'has a much higher defect density than regular requirements' and is 'not suited for any sort of distribution and frequency analysis' due to the weak-word-based sampling. Yet Section VI claims the dataset is 'the largest requirements quality dataset to date' without repeating this scope restriction. The claim may be true for weak-word-focused quality datasets, but it is misleading if interpreted as a general requirements-quality benchmark. Please qualify the headline claim in the conclusion and abstract to make explicit that the dataset is a weak-word-centric, deliberately oversampled set of defect and non-defect instances, not a representative sample of automotive requirements.
  4. [Section IV-C and Table II] The comparison of linguistic characteristics across datasets is purely descriptive. Table II reports point estimates for lexical, syntactic, and readability metrics, but no uncertainty quantification (e.g., bootstrap confidence intervals) or inferential tests are provided. The abstract's claim that 'QURE is linguistically similar to existing ones' relies on visual inspection of Figure 2 and Figure 3. Because the datasets are samples of different sizes and from different domains, some differences may be within sampling variation. Please add at least bootstrap confidence intervals or effect sizes for the key metrics to support the similarity/difference claims.
minor comments (6)
  1. [Author list] The second affiliation contains 'V ogelsang' with a space; should be 'Vogelsang'.
  2. [Throughout] The dataset is called QURE in the title and abstract but QuRE in most of the text and figures. Please standardize the spelling.
  3. [Section III-A Step 4] Typo: 'weak work' should be 'weak word' in the description of excluded rows.
  4. [Table II] The row label 'Promise' should be 'PROMISE' to match the dataset name used elsewhere.
  5. [Section IV-A] The selection of PURE* from Kaggle is a subset of the original PURE; the difference is acknowledged, but it would be helpful to state explicitly that the comparison is with this subset, not the complete PURE, in the main text of Section IV-C as well.
  6. [Section V-C] The limitation that the dataset 'focuses on weak words' is important, but it appears only in the discussion. Consider mentioning this scope in the abstract or introduction, as it affects whether the 'largest requirements quality dataset' claim is interpreted correctly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: QuRE is an empirical dataset release; its claims are descriptive and not derived from its inputs by construction.

full rationale

The paper's contributions are a released dataset and descriptive comparative statistics. There is no derivation chain whose conclusion equals an input: the defect labels are human annotations, the weak-word taxonomy is imported from prior independent work (Krisch's dissertation, Femmer et al.'s smell catalog) and is not derived from the dataset; the 'largest dataset' claim is an empirical counting claim; the linguistic comparisons are computed directly from text via standard metrics and do not presuppose the conclusions. The only passages that could look like a circularity issue are the assertions of 'industry quality' (Section III-B) and the admitted absence of inter-rater agreement (Section III-B) and possible label debatability (Section V-C). Those are threats to validity and evidentiary support, not instances of a result reducing to its inputs by definition or by self-citation. Self-citations (e.g., [5], [16], [20], [21]) are contextual history and not load-bearing justification for the dataset's central claims. Therefore score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No mathematical derivation is present. The load-bearing assumptions are about the meaning and reliability of the labels and the validity of linguistic proxies for requirements quality.

assumptions (4)
  • domain assumption Weak word categories from Krisch's catalogue are meaningful indicators of potential quality defects.
    The entire dataset's defect labels are defined through weak words; this taxonomy is taken from prior work and is not validated within this paper.
  • domain assumption Expert labels from an industrial review process constitute ground truth for requirement defects.
    The paper asserts industry quality but reports no inter-rater agreement metrics; the labels are treated as reliable enough for a benchmark.
  • domain assumption Linguistic metrics such as lexical diversity, parse tree depth, and Flesch-Kincaid grade level are appropriate proxies for requirements quality and complexity.
    The comparative analysis relies on these proxies to claim similarity between datasets, without validating that they capture quality-relevant differences.
  • domain assumption A sample of requirements with oversampled weak words is representative of real-world weak-word usage for regression testing purposes.
    The sampling strategy intentionally targeted weak words and maturity stages, making the dataset unrepresentative for frequency analysis; this is disclosed in Section V-C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Description and Comparative Analysis of QuRE: A New Industrial Requirements Quality Dataset." pith.science (2026). https://pith.science/paper/LZWI5GNX

@misc{pith2026250808868,
  author       = {Pith},
  title        = {Pith review of: Description and Comparative Analysis of QuRE: A New Industrial Requirements Quality Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZWI5GNX}},
  note         = {Machine review of arXiv:2508.08868}
}
read the original abstract

Requirements quality is central to successful software and systems engineering. Empirical research on quality defects in natural language requirements relies heavily on datasets, ideally as realistic and representative as possible. However, such datasets are often inaccessible, small, or lack sufficient detail. This paper introduces QuRE (Quality in Requirements), a new dataset comprising 2,111 industrial requirements that have been annotated through a real-world review process. Previously used for over five years as part of an industrial contract, this dataset is now being released to the research community. In this work, we furthermore provide descriptive statistics on the dataset, including measures such as lexical diversity and readability, and compare it to existing requirements datasets and synthetically generated requirements. In contrast to synthetic datasets, QuRE is linguistically similar to existing ones. However, this dataset comes with a detailed context description, and its labels have been created and used systematically and extensively in an industrial context over a period of close to a decade. Our goal is to foster transparency, comparability, and empirical rigor by supporting the development of a common gold standard for requirements quality datasets. This, in turn, will enable more sound and collaborative research efforts in the field.

Figures

Figures reproduced from arXiv: 2508.08868 by the authors.

Figure 1
Figure 1. Distribution of weak words in QURE time, then ...”, the word “long” is problematic, whereas in “The long range radar shall send ...”, the same word is appropriate in that context and, therefore, not considered a defect. The defect column was labeled by Mercedes￾Benz employees, specifically software testers. If individ￾ual cases were unclear, the labels were negotiated by up to three experts. Unfortunately, we could … view at source ↗
Figure 2
Figure 2. Distribution of syntactic complexity metrics across sentences [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Flesch-Kincaid Grade Level between datasets [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Stakeholder and Product Requirements: An Empirical Study of Requirement Engineering in the Automotive Industry

    cs.SE 2026-07 conditional novelty 6.0 of 10

    In industrial automotive chip projects, stakeholder-to-product refinement complexity is driven mainly by architectural scope and missing context rather than linguistic verbosity, while acceptance is dominated by speci...

Reference graph

Works this paper leans on

28 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Requirements quality is quality in use,

    H. Femmer and A. V ogelsang, “Requirements quality is quality in use,” IEEE Software, vol. 36, no. 3, pp. 83–91, 2019

    work page 2019

  2. [2]

    Does quality of requirements specifications matter? combined results of two empirical studies,

    J. Mund, D. Mendez Fernandez, H. Femmer, and J. Eckhardt, “Does quality of requirements specifications matter? combined results of two empirical studies,” inACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 2015, pp. 1– 10

    work page 2015

  3. [3]

    Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice,

    D. M. Fernández, S. Wagner, M. Kalinowski, M. Felderer, P. Mafra, A. Vetrò, T. Conte, M.-T. Christiansson, D. Greer, C. Lassenius, T. Män- nistö, M. Nayabi, M. Oivo, B. Penzenstadler, D. Pfahl, R. Prikladnicki, G. Ruhe, A. Schekelmann, S. Sen, R. Spinola, A. Tuzcu, J. L. de la Vara, and R. Wieringa, “Naming the pain in requirements engineering: Contempora...

    work page 2016

  4. [4]

    Empirical research on requirements quality: a systematic mapping study,

    L. Montgomery, D. Fucci, A. Bouraffa, L. Scholz, and W. Maalej, “Empirical research on requirements quality: a systematic mapping study,”Requirements Engineering, vol. 27, no. 2, pp. 183–209, 2022

    work page 2022

  5. [5]

    Rapid quality assurance with requirements smells,

    H. Femmer, D. Méndez Fernández, S. Wagner, and S. Eder, “Rapid quality assurance with requirements smells,”Journal of Systems and Software, vol. 123, pp. 190–213, 2017

    work page 2017

  6. [6]

    Research artifacts in software engineering publications: Status and trends,

    M. Liu, X. Huang, W. He, Y . Xie, J. M. Zhang, X. Jing, Z. Chen, and Y . Ma, “Research artifacts in software engineering publications: Status and trends,”Journal of Systems and Software, vol. 213, 2024

    work page 2024

  7. [7]

    PURE: A dataset of public requirements documents,

    A. Ferrari, G. O. Spagnolo, and S. Gnesi, “PURE: A dataset of public requirements documents,” inIEEE 25th International Requirements Engineering Conference (RE). IEEE, 2017, pp. 502–505

    work page 2017

  8. [8]

    Software engi- neering repositories: Expanding the PROMISE database,

    M. Lima, V . Valle, E. Costa, F. Lira, and B. Gadelha, “Software engi- neering repositories: Expanding the PROMISE database,” inBrazilian Symposium on Software Engineering (SBES). ACM, 2019, pp. 427–436

    work page 2019

Show all 28 references
  1. [9]

    Requirements classification with interpretable machine learning and dependency pars- ing,

    F. Dalpiaz, D. Dell’Anna, F. B. Aydemir, and S. Cevikol, “Requirements classification with interpretable machine learning and dependency pars- ing,” inIEEE 27th International Requirements Engineering Conference (RE). IEEE, 2019, pp. 142–152

    2019

  2. [10]

    Intralogistics requirements data set,

    A. Wohlgemuth, “Intralogistics requirements data set,” 2024. [Online]. Available: https://zenodo.org/doi/10.5281/zenodo.14513612

    2024 doi

  3. [11]

    Requirements data sets (user stories),

    F. Dalpiaz, “Requirements data sets (user stories),” 2018. [Online]. Available: https://data.mendeley.com/datasets/7zbk8zsd8y/1

    2018

  4. [12]

    Knauss, S

    E. Knauss, S. H. Houmb, S. Islam, J. Jürjens, and K. Schneider, “SecReq,” 2021. [Online]. Available: https://zenodo.org/doi/10.5281/ zenodo.4530182

    2021

  5. [13]

    Requirements quality research artifacts: Recovery, analysis, and management guideline,

    J. Frattini, L. Montgomery, D. Fucci, M. Unterkalmsteiner, D. Mendez, and J. Fischbach, “Requirements quality research artifacts: Recovery, analysis, and management guideline,”Journal of Systems and Software, vol. 216, p. 112120, 2024

    2024

  6. [14]

    Inconsistency detection in natural language requirements using ChatGPT: a prelim- inary evaluation,

    A. Fantechi, S. Gnesi, L. Passaro, and L. Semini, “Inconsistency detection in natural language requirements using ChatGPT: a prelim- inary evaluation,” inIEEE 31st International Requirements Engineering Conference (RE). IEEE, 2023

    2023

  7. [15]

    Experience with variability management in requirement specifications,

    E. Boutkova, “Experience with variability management in requirement specifications,” in15th International Software Product Line Conference (SPLC). IEEE, 2011, pp. 303–312

    2011

  8. [16]

    Sprachliche Kontrolle von Anforderungsdokumenten,

    J. Krisch, “Sprachliche Kontrolle von Anforderungsdokumenten,” Ph.D. dissertation, Universitätsverlag Hildesheim Hildesheim, 2017

    2017

  9. [17]

    DeNom: a tool to find problematic nominalizations using NLP,

    M. Landhausser, S. J. Korner, W. F. Tichy, J. Keim, and J. Krisch, “DeNom: a tool to find problematic nominalizations using NLP,” in IEEE Second International Workshop on Artificial Intelligence for Requirements Engineering (AIRE). IEEE, 2015, pp. 1–8

    2015

  10. [18]

    A lexical resource for the identification of “weak words

    J. Krisch, M. Dick, R. Jauch, and U. Heid, “A lexical resource for the identification of “weak words” in German specification documents,” in Tenth International Conference on Language Resources and Evaluation (LREC’16), N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Gro- ...

    2016

  11. [19]

    The myth of bad passive voice and weak words: an empirical investigation in the automotive industry,

    J. Krisch and F. Houdek, “The myth of bad passive voice and weak words: an empirical investigation in the automotive industry,” inIEEE 23rd International Requirements Engineering Conference (RE). IEEE, 2015

    2015

  12. [20]

    Automatic requirements reviews - potentials, limitations and practical tool support,

    H. Femmer, “Automatic requirements reviews - potentials, limitations and practical tool support,” inProduct-Focused Software Process Im- provement (PROFES). Springer International Publishing, 2017, pp. 617–620

    2017

  13. [21]

    Requirements quality defect detection with the Qualicen Require- ments Scout,

    ——, “Requirements quality defect detection with the Qualicen Require- ments Scout,” inJoint Proceedings of REFSQ-2018 Workshops, Doctoral Symposium, Live Studies Track, and Poster Track, ser. CEUR Work- shop Proceedings, K. Schmid, P. Spoletini, E. B. Charrada, Y . Chisik, F. ...

    2018

  14. [22]

    A prompt pattern cat- alog to enhance prompt engineering with ChatGPT,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. El- nashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern cat- alog to enhance prompt engineering with ChatGPT,”arXiv preprint arXiv:2302.11382, 2023

    2023 arXiv

  15. [23]

    W. N. Francis and H. Kucera,Brown corpus manual, Brown University, 1979

    1979

  16. [24]

    Is this sentence difficult? do you agree?

    D. Brunato, L. De Mattei, F. Dell’Orletta, B. Iavarone, G. Venturiet al., “Is this sentence difficult? do you agree?” inConference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2018, pp. 2690–2699

    2018

  17. [25]

    Automatic analysis of syntactic complexity in second language writing,

    X. Lu, “Automatic analysis of syntactic complexity in second language writing,”International journal of corpus linguistics, vol. 15, no. 4, pp. 474–496, 2010

    2010

  18. [26]

    Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel,

    J. P. Kincaid, R. P. Fishburne Jr, R. L. Rogers, and B. S. Chissom, “Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel,” Institute for Simulation and Training, University of Central Flori...

    1975

  19. [27]

    Requirements quality research: a harmonized theory, evaluation, and roadmap,

    J. Frattini, L. Montgomery, J. Fischbach, D. Mendez, D. Fucci, and M. Unterkalmsteiner, “Requirements quality research: a harmonized theory, evaluation, and roadmap,”Requirements engineering, vol. 28, no. 4, pp. 507–520, 2023

    2023

  20. [28]

    Automated han- dling of anaphoric ambiguity in requirements: a multi-solution study,

    S. Ezzini, S. Abualhaija, C. Arora, and M. Sabetzadeh, “Automated han- dling of anaphoric ambiguity in requirements: a multi-solution study,” in 44th International Conference on Software Engineering (ICSE). ACM, 2022, pp. 187–199

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.