{"id":"69cc6153-1e0b-4697-b67e-0fe227dac630","arxiv_id":"2607.16564","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A traceable pipeline (ReqGenX) can decompose legacy SRS documents into atomic statements and typed pre-SRS artifacts with high grounding, but information is partially lost along the way.","lead":"This paper builds a pipeline that breaks old software requirements documents into small, traceable pieces, then converts those pieces into typed requirements artifacts with source links. The finding is that LLMs can do this faithfully most of the time, but some information is still lost, making a trade-off between faithfulness and completeness visible.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All central faithfulness/grounding numbers depend on AlignScore thresholds never validated against human judgments on SRS text; if AlignScore is miscalibrated for requirements language, the feasibility claim's quantitative support collapses.","rationale":"The reader identified the same weakest assumption, and I agree. Every headline quantitative claim—atom faithfulness, artifact grounding, reconstruction support—is measured by AlignScore (and SBERT for semantic retention) without a human calibration study against SRS text. The judge LLM is human-calibrated, but that does not validate the alignment metric used for the headline numbers. This is the most load-bearing link in the argument because the paper's central claim is that the transformation yields faithful, grounded, traceable artifacts; if the metric that establishes faithfulness is unreliable, the transformation's feasibility is not established. I do not think this warrants rejection: the study is transparent, ships artifacts, and explicitly calls the metrics proxies. But because the abstract and findings present the AlignScore numbers as evidence of faithfulness, acceptance should be conditional on either (a) providing the stratified human calibration test described above, or (b) visibly rephrasing all such numbers as 'per AlignScore' and moving the feasibility claim to the provenance/artifact release rather than the metric-based quality evidence. The concrete check would settle the question.","tokens_in":17626,"tokens_out":10281,"duration_ms":116774,"concrete_test":"Stratified human validation: sample about 150 atoms and 150 artifacts across the seven SRSs, spanning the full AlignScore distribution. Have two requirements-engineering annotators independently label each sample as supported/unsupported relative to its source (binary), adjudicate disagreements, then compute AlignScore's AUC and its confusion matrix at the 0.80 threshold (precision/recall and false-acceptance rate). If AlignScore fails to discriminate human-supported from human-unsupported items (e.g., AUC < 0.8 or >20% false acceptance at 0.80), the headline quantitative claims should be reframed as AlignScore-relative and the feasibility conclusion weakened; if AlignScore passes, the claims stand.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical support for 'faithful atoms' and 'grounded artifacts' is AlignScore medians (0.96–0.99 for atoms, 0.80–0.94 for artifacts) and unsupported rates at a fixed 0.80 threshold (§3.2, §4.2, §4.3). The judge LLM (Gemma 3) is selected against 291 reconciled human triples (§4.1), but AlignScore itself is never calibrated against human judgments of support in the requirements domain. AlignScore is a general alignment/NLI metric; SRS text is deontic, conditional, and domain-specific, and atom/artifact units are short, so its training distribution may not match. If AlignScore is systematically lenient on paraphrase that drops a qualifier, or harsh on legitimate compact rephrasing, the reported medians and '<3% unsupported' claims would not establish genuine faithfulness. The same issue propagates to reconstruction support values in Table 5. The threats section (§7) labels AlignScore as approximate, but the abstract and findings present these numbers as evidence that the transformation 'does not break down'; without domain-level validation the core quantitative claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReqGenX, a controlled three-stage pipeline that converts legacy SRS documents into traceable synthetic pre-SRS artifacts: section-level chunks are decomposed into atomic, source-grounded statements; atoms are semantically routed to IEEE 29148-inspired artifact types via multi-LLM plurality voting with human-in-the-loop review; and typed artifacts are generated under constrained prompts with judge-guided refinement. The evaluation uses seven text-based PURE SRS documents and multiple LLMs (Claude, GPT-4o, GPT-OSS, etc.) to study five research questions: reliability of LLM-based routing and judging, quality of atoms, quality of artifacts, downstream SRS reconstruction, and information loss. The reported results show high atom faithfulness (median AlignScore 0.96–0.99, unsupported rates mostly below 3%), strong artifact grounding (mean AlignScore 0.80–0.94), high internal judge pass rates after refinement, and partial recoverability in a four-dataset reconstruction case study (SBERT means 0.69–0.75). The authors conclude that traceable synthetic pre-SRS artifacts can support fine-grained evaluation of LLM-based SRS generation while exposing tradeoffs among faithfulness, coverage, and artifact completeness.","tokens_in":17980,"tokens_out":4468,"duration_ms":50758,"significance":"If the quantitative claims hold, this is a useful and timely contribution: it provides a reproducible, traceable evaluation substrate for LLM-based SRS generation and a stage-wise characterization of where information is lost. The paper has several genuine strengths. It ships a replication package with processed data, prompts, codebooks, traceability metadata, and evaluation outputs. It calibrates the judge LLM against 291 human-annotated triples (κ=0.84 for atoms, κ=0.97 for artifacts), which is a substantially stronger grounding than is common in LLM-as-a-judge work. The authors are appropriately careful to label the study as a feasibility analysis, not a deployment claim, and they explicitly discuss external validity limitations. The stage-wise analysis of information loss and failure patterns is insightful and goes beyond simple end-task accuracy. However, the central quantitative evidence depends on AlignScore and embedding-similarity thresholds that are not validated for SRS text, which is the main barrier to accepting the feasibility conclusion as stated.","major_comments":[{"comment":"The central quantitative claims—atoms are faithful (median AlignScore 0.96–0.99, unsupported rates mostly below 3%), artifacts are grounded (AlignScore 0.80–0.94), and reconstruction preserves support (AlignScore medians 0.76–0.84)—all rest on AlignScore with a fixed 0.80 support threshold. No evidence is given that AlignScore is calibrated for SRS language, which is deontic, conditional, and domain-specific, or for the short atom/artifact units used here. A general alignment/NLI model may be systematically lenient about dropped qualifiers or harsh about legitimate compact rephrasing. The threats section (§7) labels AlignScore as approximate, but that does not establish that the threshold separates supported from unsupported content in this domain. Please add a domain-level calibration: human annotation of support for a sample of atoms/artifacts against their sources, an error analysis,","section":"§3.2, §4.2–§4.4, Figs. 4/6, Table 5"},{"comment":"The information-retention and loss percentages (e.g., 'Claude retains 52.7–84.2% end-to-end') rely on a source-coverage threshold of 0.75 cosine similarity over text-embedding-3-large. This threshold is also not validated against human judgments of whether a source unit is genuinely retained after atomization/artifact generation. A faithfully compressed but paraphrased unit may fall below 0.75 and be counted as lost, while a unit with overlapping vocabulary but altered meaning may pass. This directly affects RQ5's conclusion that information loss is stage-specific and reflects evidence granularity rather than generation failure. Please add a sensitivity analysis and a small human check of the covered/lost classifications to demonstrate that the reported tradeoffs are not artifacts of the chosen threshold.","section":"§3.2, §4.5, Fig. 7"}],"minor_comments":[{"comment":"Finding 2 states that 'most unsupported rates remain below 3%,' but QHeadache with Claude (9.2%) and VUB with GPT-4o/GPT-OSS (8.9/8.6%) are notable exceptions. Please name them in the text so readers do not have to extract them from the figure.","section":"§4.2, Fig. 4"},{"comment":"Table 5 reports only four datasets; the text says this is a limited case study, but the caption should state the inclusion criterion and the distribution of the 473 accepted artifact-backed atoms across datasets.","section":"§4.4, Table 5"},{"comment":"The internal judge pass ratio is close to 1.00 by construction, because the loop iterates until the artifact passes or the refinement budget is exhausted. Please state this explicitly in the metric definition and interpret judge pass ratio as a process check, not an independent quality signal. The Prometheus results already serve this purpose, but the current wording blurs the distinction.","section":"§2.3, §4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical paper with a credible replication package and human-anchored judge selection. The main barrier is that the headline numbers all depend on AlignScore and embedding thresholds that have not been validated for SRS text. I would not reject: a calibration study or a thorough threshold-sensitivity analysis with qualitative error analysis would be enough to address the concern. The small, hand-filtered dataset and single-run design are acknowledged limitations and are acceptable for a feasibility study in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead ReqGenX. The useful core is the pipeline: decompose legacy SRS sections into source-grounded atomic claims, route those atoms to artifact families via plurality voting, then generate typed artifacts with provenance links. That gives requirements engineering evaluation something it has been missing—a traceable substrate for checking whether an LLM-generated SRS is actually supported by source material. The paper also contributes real empirical findings: semantic tags are more stable than fine-grained requirement labels (Claude κ=0.83 vs 0.80), and the human-calibrated judge-selection procedure with 291 annotated triples is a solid methodological addition.\n\nWhat's good: the authors ship a replication package, state their assumptions, and hedge their conclusions. They don't oversell. The stage-wise information-loss analysis is genuinely useful—most loss is evidence-granularity, not hallucination or parsing failure.\n\nThe soft spots are real but proportionate. The dataset is seven hand-filtered text-only PURE documents; small and selected for tractability. AlignScore's support threshold (0.80) and coverage threshold (0.75) are used as ground truth for faithfulness and retention, and the paper never calibrates AlignScore against human judgments on SRS text. That is a genuine gap: if AlignScore is miscalibrated on deontic, conditional requirements language, the medians in the abstract (0.96–0.99 for atoms, 0.80–0.94 for artifacts) would overstate the case. The stress-test note is right to flag this. But I don't think it is fatal. The central claim is feasibility, and the paper triangulates with human-calibrated judge scores, Prometheus, and qualitative revision patterns. The exact AlignScore numbers should be treated as approximate, and the authors should either calibrate AlignScore on SRS data or soften the abstract's wording. There's also no baseline comparison; the authors argue existing baselines lack intermediate provenance, which is fair, but it limits what they can claim about ReqGenX's relative merit. Single-run generation at temperature 1.0 is another caveat they acknowledge.\n\nBottom line: this is a serious, honest empirical study. It deserves a real referee. The flaws are limitations, not load-bearing errors. I'd send it out and ask for a short AlignScore calibration check or explicit domain-validation caveat before acceptance. Anyone working on LLM-based requirements evaluation should read it and probably cite it.","headline":"A careful, well-scoped empirical study that makes a plausible case for using traceable synthetic pre-SRS artifacts in LLM-based requirements evaluation; the main quantitative claims rest on uncalibrated AlignScore thresholds, but the paper's transparency and human-grounded judge selection keep it honest.","tokens_in":18410,"tokens_out":3611,"would_cite":true,"duration_ms":38769,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Legacy SRS documents can be broken into source-grounded atomic statements, regenerated into typed pre-SRS artifacts that stay traceable to the original requirements, and used to measure how well LLMs preserve requirements information throug","keywords":["requirements engineering","SRS generation","LLM evaluation","atomic decomposition","traceability","artifact generation","information loss","synthetic benchmark"],"falsifier":"If a human-annotated sample of generated atoms, artifacts, and reconstructed SRS sentences shows that items scored above 0.80 by AlignScore often contain unsupported or distorted claims—or if reconstructing SRSs from the same artifacts with a different compiler model yields SBERT similarities below 0.5 for most atoms—then the central claim about faithfulness and recoverability would be undermined.","tokens_in":17572,"feed_emoji":"🧩","tokens_out":2968,"duration_ms":33795,"temperature":0.7,"pith_summary":"The paper claims that ordinary legacy software requirements specifications (SRS) can be turned into a traceable evaluation substrate for LLM-based SRS generation. Instead of judging a generated SRS only by document-level fluency, the proposed pipeline splits each SRS section into atomic claims, routes those atoms to standards-inspired artifact types (use case briefs, quality constraints, context notes), and generates artifacts that remain linked to their source sentences. The study shows this transformation keeps most atoms faithful and artifacts grounded, but downstream reconstruction preserves only part of the original information. A sympathetic reader would care because it offers a way to test whether an LLM actually used the source requirements rather than just producing coherent prose.","feed_headline":"Break legacy SRS into traceable atoms to test LLM generation","feed_subtitle":"A three-stage pipeline links every generated artifact back to its source claim, exposing where requirements information survives and where i","key_machinery":"The central object is the atom: a minimal, source-supported proposition extracted from an SRS section. The study uses an atomization step to split sections into such propositions, a semantic-routing step that assigns each atom to an artifact family via plurality voting among LLMs, artifact templates that keep the atom as the source of truth, and a judge-guided refinement loop that enforces faithfulness, support, contextuality, structural alignment, and sparsity. Provenance metadata links each artifact back to its source atom and original SRS location. Two automated metrics carry the evaluation: AlignScore for factual support between generated text and its source, and SBERT cosine similarity","core_discovery":"The central claim is that a three-stage pipeline—decomposing SRS sections into atomic statements, routing those atoms to artifact types through multi-LLM plurality voting, and generating artifacts with constrained prompts plus iterative judge-guided refinement—can create traceable synthetic pre-SRS artifacts from legacy SRS documents. On seven public SRS documents, the resulting atoms show high faithfulness (median alignment scores 0.96–0.99) and low unsupported rates, while generated artifacts remain strongly grounded in their source atoms (alignment scores 0.80–0.94). Downstream SRS reconstruction from those artifacts is only partially successful: semantic similarity scores are moderate, a","pith_inferences":["An editorial extension: the same traceability links could enable automatic defect localization in generated SRSs, flagging each generated sentence to the atoms it failed to preserve.","The atom-and-artifact schema is not limited to SRSs; the same decomposition-routing-generation pattern could be applied to other documentation types, such as change requests or regulatory texts, where claim-level traceability is valuable.","The reliance on embedding-based and alignment-based metrics as proxies for human-judged faithfulness needs calibration against domain-specific requirements language; the paper itself treats these metrics as proxy quality, not complete correctness.","A testable extension would be to vary the artifact templates and see whether more permissive templates improve downstream reconstruction without increasing unsupported content, directly probing the faithfulness-completeness tradeoff the paper identifies."],"forward_implications":["If the pipeline works as claimed, evaluation of LLM-based SRS generation can move from document-level fluency to claim-level traceability, letting researchers see exactly which source statements support each generated requirement.","Public SRS documents can be converted into reproducible benchmarks of synthetic pre-SRS artifacts, reducing the need for manually curated requirements datasets.","Stage-wise information-loss analysis can guide future systems to preserve implicit or weakly structured statements rather than discarding them during atomization.","The observed gap between high grounding and lower Prometheus pass rates means artifact quality checks must go beyond faithfulness to cover sparsity, completeness, and structural validity.","Downstream reconstruction studies should verify factuality across the full chain—source SRS, atoms, artifacts, reconstructed SRS—instead of only checking the final document."],"fun_headline_variants":["Transform legacy SRS into traceable atoms for fine-grained LLM eval","Three-stage pipeline links generated SRS artifacts to source claims","Atomic decomposition makes LLM SRS generation easier to verify","Legacy SRS become testable via atomic decomposition and artifact regeneration","From legacy SRS to traceable atoms: a pipeline for better LLM testing"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The automatic evaluation metrics—AlignScore for factual support and the fixed similarity/threshold values used for coverage and retention—are accurate enough for requirements text that the reported high faithfulness and low unsupported rates reflect genuine semantic support rather than artifact misjudgment.","fun_headline_variants_meta":{"raw":{"variants":["Transform legacy SRS into traceable atoms for fine-grained LLM eval","Three-stage pipeline links generated SRS artifacts to source claims","Atomic decomposition makes LLM SRS generation easier to verify","Legacy SRS become testable via atomic decomposition and artifact regeneration","From legacy SRS to traceable atoms: a pipeline for better LLM testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1316,"prompt_tokens":845,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":589,"tokens_out":471,"duration_ms":5506,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T20:33:22.358274+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a human-annotated sample of generated atoms, artifacts, and reconstructed SRS sentences shows that items scored above 0.80 by AlignScore often contain unsupported or distorted claims—or if reconstructing SRSs from the same artifacts with a different compiler model yields SBERT similarities below 0.5 for most atoms—then the central claim about faithfulness and recoverability would be undermined.","supporting_citations":[],"review_version":1}