{"id":"06730980-854e-4169-a303-4d183a52cff3","arxiv_id":"2508.08868","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new public dataset of 2,111 labeled industrial requirements is released and shown to be linguistically closer to real datasets than to naive synthetic ones.","lead":"This paper introduces QuRE, a new public dataset of 2,111 industrial car requirements labeled for weak-word quality defects by engineers. It compares QuRE with other requirements datasets and with ChatGPT-generated requirements, showing that real requirements are linguistically more complex.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QuRE's defect labels are asserted to be 'industry quality' but no inter-rater agreement is reported; the benchmark's ground truth is therefore unverified.","rationale":"The paper's contribution is a dataset intended as a benchmark for requirements quality tools. The size and linguistic realism are well documented and the dataset is a genuine new artifact. However, the dataset's usefulness as ground truth hinges on the validity of its defect labels. The paper explicitly states that no inter-rater agreement metrics could be retrieved, and it concedes that labels may be debatable or incorrect from an academic viewpoint. This is the weakest link in the central claim because it directly undermines the 'industry quality' assertion, which is the basis for treating QuRE as a gold standard. The reader's weakest assumption identified the same issue, and I agree. My concrete test would either provide the missing evidence or demonstrate that the concern is real. The verdict should remain CONDITIONAL: the dataset is valuable and transparently described, but its label reliability must be independently verified before it can be recommended as a benchmark without caveats. No other concern—such as the sampling bias or the lack of uncertainty bounds in comparative statistics—is as load-bearing, since these are either acknowledged limitations or less directly tied to the dataset's primary purpose.","tokens_in":10073,"tokens_out":3240,"duration_ms":32683,"concrete_test":"Independently re-annotate a stratified random sample of 200 rows from QuRE (e.g., 50 each from the four most frequent weak-word labels) using the same defect/ok labels and a written annotation guideline derived from Section III-B. Have two annotators who were not involved in the original labeling process label all 200 rows. Compute pairwise Cohen's kappa between the original labels and each new annotator, and between the two new annotators, overall and per weak-word category. If overall kappa is below 0.6, or if per-category kappa varies by more than 0.2, the 'industry quality' claim is not supported and the dataset should be presented with an explicit warning that label reliability is unverified. Also compare the per-category defect rates in the sample to the original distribution; a significant shift would indicate labeling bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central value of QuRE as a benchmark depends on the defect labels being sufficiently accurate to serve as ground truth. Section III-B states that labels were negotiated by up to three experts, but immediately concedes that 'we could not retrieve any information on the consistency of the experts' labeling, such as inter-rater agreement metrics.' The claim in Section III-B that 'the annotations have industry quality' is therefore an assertion based on the dataset's use in a five-year SLA, not a measurement. Contractual use shows the labels were useful for a specific regression-testing purpose, but it does not establish that they are correct or consistent in the sense required for evaluating generic quality-assessment tools. Section V-C further concedes that 'single labels may be debatable or even incorrect from an academic point of view' and that the dataset has a higher defect density than typical requirements due to sampling. If original labelers systematically disagreed with each other or with an independent standard, any tool benchmarked on QuRE is scored against a noisy reference; reported performance would be upper-bounded by agreement with the original labelers, not by true detection quality. Because no agreement metric is reported, the strongest claim—'industry-quality' ground truth—is not supported by the evidence presented. Additionally, the dataset only contains requirements that contain at least one weak word from the Mercedes-Benz catalog; this selection is acknowledged, but it means QuRE is not a random sample of requirements, further limiting the generality of any quality-assessment benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QuRE, a publicly released dataset of 2,111 industrial requirements from Mercedes-Benz, each annotated with a weak-word indicator and a human-assigned defect/ok label. The authors describe the dataset's provenance, including a five-year use in an industrial SLA, provide descriptive linguistic statistics, and compare QuRE with existing requirements datasets and LLM-generated synthetic requirements. They argue that QuRE is the largest requirements-quality dataset to date, that its labels are of 'industry quality,' and that its linguistic properties align with other real-world requirements datasets while synthetic data are linguistically distinct.","tokens_in":10353,"tokens_out":2696,"duration_ms":28632,"significance":"If the dataset and its annotations are reliable, QuRE is a valuable community resource: it is substantially larger than most publicly available requirements-quality datasets, it comes from a real industrial context, and it is accompanied by release code and documentation. The paper is transparent about provenance and limitations, and the authors explicitly call for community-driven dataset infrastructure. The comparative analysis of synthetic versus real requirements is a useful starting point for researchers evaluating LLM-generated test data. However, the central value Proposition—usable ground-truth defect labels—rests on expert annotations whose consistency is not measured, and the dataset covers only weak-word-related defects from a specific sampling strategy. These caveats do not invalidate the resource, but they bound what can be claimed.","major_comments":[{"comment":"The statement 'the annotations have industry quality' is presented as a conclusion from the dataset's industrial use, but the section immediately acknowledges that no inter-rater agreement metrics could be retrieved. Contractual use in an SLA demonstrates that the labels were useful for regression testing in a specific industrial setting, but it does not measure label accuracy, consistency, or agreement with any external standard. Since the dataset's primary purpose is to serve as ground truth for evaluating quality-assessment tools, this missing evidence is load-bearing. Please either provide inter-rater agreement statistics (if recoverable), or soften the claim from 'industry quality' to 'labels resulting from a negotiated industrial review process' and explicitly state in the abstract and conclusion that label reliability is unquantified.","section":"Section III-B"},{"comment":"The preprocessing step excluded 31 rows with inconsistent data (a defect label with no weak word) and 33 rows where the human labeler was undecided, totaling 64 rows (<3%). While the exclusion is disclosed, its effect on benchmark conclusions is not analyzed. These excluded cases may indicate real label noise or ambiguity; removing them could systematically improve apparent label quality. Please report the characteristics of the excluded rows and, if possible, analyze how inclusion/exclusion of these cases affects downstream benchmarking (e.g., agreement with a simple weak-word heuristic).","section":"Section III-A Step 4"},{"comment":"The paper acknowledges in Section V-C that the dataset 'has a much higher defect density than regular requirements' and is 'not suited for any sort of distribution and frequency analysis' due to the weak-word-based sampling. Yet Section VI claims the dataset is 'the largest requirements quality dataset to date' without repeating this scope restriction. The claim may be true for weak-word-focused quality datasets, but it is misleading if interpreted as a general requirements-quality benchmark. Please qualify the headline claim in the conclusion and abstract to make explicit that the dataset is a weak-word-centric, deliberately oversampled set of defect and non-defect instances, not a representative sample of automotive requirements.","section":"Section V-C and Section VI"},{"comment":"The comparison of linguistic characteristics across datasets is purely descriptive. Table II reports point estimates for lexical, syntactic, and readability metrics, but no uncertainty quantification (e.g., bootstrap confidence intervals) or inferential tests are provided. The abstract's claim that 'QURE is linguistically similar to existing ones' relies on visual inspection of Figure 2 and Figure 3. Because the datasets are samples of different sizes and from different domains, some differences may be within sampling variation. Please add at least bootstrap confidence intervals or effect sizes for the key metrics to support the similarity/difference claims.","section":"Section IV-C and Table II"}],"minor_comments":[{"comment":"The second affiliation contains 'V ogelsang' with a space; should be 'Vogelsang'.","section":"Author list"},{"comment":"The dataset is called QURE in the title and abstract but QuRE in most of the text and figures. Please standardize the spelling.","section":"Throughout"},{"comment":"Typo: 'weak work' should be 'weak word' in the description of excluded rows.","section":"Section III-A Step 4"},{"comment":"The row label 'Promise' should be 'PROMISE' to match the dataset name used elsewhere.","section":"Table II"},{"comment":"The selection of PURE* from Kaggle is a subset of the original PURE; the difference is acknowledged, but it would be helpful to state explicitly that the comparison is with this subset, not the complete PURE, in the main text of Section IV-C as well.","section":"Section IV-A"},{"comment":"The limitation that the dataset 'focuses on weak words' is important, but it appears only in the discussion. Consider mentioning this scope in the abstract or introduction, as it affects whether the 'largest requirements quality dataset' claim is interpreted correctly.","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid dataset-description contribution, and the authors' transparency about provenance and limitations is commendable. The main issue from my perspective is that the term 'industry quality' is used as a warranty for label correctness without supporting measurement, and the companion claim of being 'the largest requirements quality dataset' is not adequately scoped. These are fixable through rephrasing and additional analysis. I would also gently note that the industrial contract history involves a company founded by one of the authors; this is disclosed implicitly but could be made more explicit to avoid any perception of bias in the 'industry quality' assertion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"QuRE is a genuinely new and useful resource: 2,111 industrial requirements from Mercedes-Benz, labeled by practitioners, released publicly with code. That alone moves the field forward, because the alternatives are tiny, inaccessible, or synthetic.\n\nThe paper does most things right. The provenance is described in unusual detail (PhD thesis, SLA, cleaning steps). The analysis is honest: they compare against several real and synthetic datasets, and they clearly flag that the dataset is not suitable for frequency/distribution claims, that defect density is inflated by sampling, and that labels are industry-grade but might be debatable academically (Sections V-B and V-C).\n\nThe soft spot is exactly the one the reader flags: the central claim that labels have 'industry quality' is an assertion, not a measurement. No inter-rater agreement is reported (Section III-B), and the five-year SLA only proves the labels were useful for regression testing a specific tool, not that they are correct in any generalizable sense. If a benchmark inherits systematic label noise, tool comparisons built on it inherit that noise. This is a real limitation, but it is not fatal: the dataset is still useful as a realistic, labeled corpus, as long as users understand the labels are one industrial team's judgment. The paper would be stronger with a small re-labeling study on a subsample to bound label noise.\n\nThe comparison with synthetic data is descriptive and lacks uncertainty bounds, but for a dataset paper that is acceptable. The prompt-based generation is simple, but the observation that even context/role-prompting stays far from real requirements is a useful side result.\n\nThis paper deserves a serious referee. The dataset is important enough, and the paper is transparent enough, that a careful review could verify the artifact and push for a modest label-consistency analysis. I would accept it for review, with a request to make the label-quality caveat front-and-center in the abstract.","headline":"QuRE is a genuinely useful public industrial requirements dataset, but the label-quality claim rests on an unmeasured assertion; still worth refereeing.","tokens_in":10823,"tokens_out":1399,"would_cite":true,"duration_ms":16120,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QuRE: a new public dataset of 2,111 real industrial requirements, each annotated with a weak-word label and an expert defect judgment.","keywords":["requirements quality","dataset","natural language requirements","requirements smells","weak words","industrial requirements","benchmark","synthetic data"],"falsifier":"Have two or more independent experts re-annotate all 2,111 requirements under the same 23 weak-word categories and compare their defect/ok judgments; if agreement is near chance or systematically opposite to the original labels, the benchmark's ground truth collapses.","tokens_in":9980,"feed_emoji":"🚗","tokens_out":4016,"duration_ms":40397,"temperature":0.7,"pith_summary":"QuRE is a new public dataset of 2,111 natural-language requirements drawn from real automotive specifications, each annotated with a weak-word label and an expert judgment of whether that word marks a quality defect. The labels come from an industrial review process used in a contractual setting for five years, which the authors describe as giving the annotations industry quality. The paper argues that QuRE is the largest requirements-quality dataset publicly available to date and that it is linguistically typical of real-world requirements, unlike three sets of LLM-generated requirements that are markedly simpler, shorter, and less dense. If that claim holds, researchers gain a common, realistic benchmark for training and evaluating automatic quality-assessment tools, and a reference point for judging whether synthetic requirements can stand in for industrial ones.","feed_headline":"2,111 industrial requirements become a public quality benchmark","feed_subtitle":"Real automotive specs with defect labels from a decade of industrial review give quality tools a realistic test set.","key_machinery":"The dataset itself is the central artifact: 2,187 rows covering 2,111 unique requirements, each paired with one of 23 weak-word labels and a binary ok/defect judgment. The weak-word taxonomy—originally a 103-word catalog reduced to 23 umbrella categories—carries the analytic load, since it converts the abstract notion of a requirements smell into a concrete, labeled signal that tools can be trained and tested on. The defect/ok column is what makes it a quality benchmark rather than just a text corpus.","core_discovery":"On its own terms, the paper's contribution is the release of QuRE: a CSV dataset of 2,111 unique English requirements from a single automotive domain, with four columns—id, requirement text, weak_word (an umbrella label such as 'detail' or 'close'), and defect (ok or defect). The weak-word labels derive from a catalog used to build regression tests for requirements-smell detectors, and the defect judgments were made by up to three industry experts, with unclear cases negotiated. The dataset was used for five years under a service-level agreement as a benchmark that automated analyses had to match, which the authors cite as evidence of industry-grade label quality. A comparative linguistic an","pith_inferences":["Because the authors could not retrieve inter-rater agreement metrics, benchmarks built on QuRE should report how results shift under label noise; a re-annotation study would be the natural next experiment.","The sampling deliberately over-represents requirements containing weak words and defects, so QuRE should not be used to estimate how often quality defects occur in ordinary automotive specifications.","The framework of 23 weak-word categories suggests a testable extension: prompt LLMs with examples from QuRE and measure whether the generated requirements move toward real-world lexical density and parse-tree depth.","If QuRE's labels are accepted as ground truth, the same annotation schema could be applied to requirements in avionics, defense, or healthcare, testing how far automotive writing style transfers."],"forward_implications":["Automated requirements-quality tools can be benchmarked on a common, realistic set instead of small or private collections.","Machine-learning models for detecting weak-word misuse and related lexical smells can be trained and validated on 2,111 industry-authored sentences.","Researchers can run replication studies and compare new taxonomies of requirements defects against the existing labels.","The reported contrast between QuRE and the three synthetic sets gives a quantitative baseline for asking when LLM-generated requirements are realistic enough for experiments.","The dataset's documented provenance and context make it a candidate seed for a community-driven requirements-dataset repository."],"supporting_citations":[{"why":"Supplies the weak-word catalog, the original sampling strategy, and the linguistic rules separating problematic from unproblematic weak-word usage.","marker":"[16]"},{"why":"Defines the concept of requirements smells that the weak-word defect labels operationalize.","marker":"[5]"},{"why":"Provides the PURE dataset and the PURE* sample used as the main real-world comparison baseline.","marker":"[7]"},{"why":"Documents that most requirements-quality datasets are small and inaccessible, establishing the scarcity claim QuRE addresses.","marker":"[13]"},{"why":"Empirically grounds the distinction between problematic and unproblematic weak words in automotive requirements, motivating the ok/defect labeling.","marker":"[19]"},{"why":"Expands the PROMISE dataset used in the linguistic comparison with QuRE.","marker":"[8]"},{"why":"Supplies the Brown corpus as a generic-English baseline for the linguistic metrics.","marker":"[23]"}],"fun_headline_variants":["QuRE: 2,111 industrial requirement reviews, now public","A decade of industrial requirement quality, released as QuRE","Real requirements, real defects: QuRE dataset for benchmarking","QuRE: 2,111 real-world requirements with expert defect labels"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The dataset's value as a benchmark rests on the assumption that the industrial defect/ok labels are accurate and consistent enough to serve as ground truth, yet the paper reports no inter-rater agreement metrics and acknowledges individual labels may be debatable or incorrect from an academic viewpoint.","fun_headline_variants_meta":{"raw":{"variants":["QuRE: 2,111 industrial requirement reviews, now public","A decade of industrial requirement quality, released as QuRE","Real requirements, real defects: QuRE dataset for benchmarking","QuRE: 2,111 real-world requirements with expert defect labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1168,"prompt_tokens":727,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":471,"tokens_out":441,"duration_ms":4719,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:20:04.824746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more independent experts re-annotate all 2,111 requirements under the same 23 weak-word categories and compare their defect/ok judgments; if agreement is near chance or systematically opposite to the original labels, the benchmark's ground truth collapses.","supporting_citations":[{"cited_title":"Sprachliche Kontrolle von Anforderungsdokumenten,","cited_arxiv_id":null,"evidence_quote":"Supplies the weak-word catalog, the original sampling strategy, and the linguistic rules separating problematic from unproblematic weak-word usage."},{"cited_title":"Rapid quality assurance with requirements smells,","cited_arxiv_id":null,"evidence_quote":"Defines the concept of requirements smells that the weak-word defect labels operationalize."},{"cited_title":"PURE: A dataset of public requirements documents,","cited_arxiv_id":null,"evidence_quote":"Provides the PURE dataset and the PURE* sample used as the main real-world comparison baseline."},{"cited_title":"Requirements quality research artifacts: Recovery, analysis, and management guideline,","cited_arxiv_id":null,"evidence_quote":"Documents that most requirements-quality datasets are small and inaccessible, establishing the scarcity claim QuRE addresses."},{"cited_title":"The myth of bad passive voice and weak words: an empirical investigation in the automotive industry,","cited_arxiv_id":null,"evidence_quote":"Empirically grounds the distinction between problematic and unproblematic weak words in automotive requirements, motivating the ok/defect labeling."},{"cited_title":"Software engi- neering repositories: Expanding the PROMISE database,","cited_arxiv_id":null,"evidence_quote":"Expands the PROMISE dataset used in the linguistic comparison with QuRE."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Brown corpus as a generic-English baseline for the linguistic metrics."}],"review_version":1}