{"id":"c26f0fda-f062-47c7-8d28-857bbea95735","arxiv_id":"2507.14969","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RequireCEG combines large language models with causal-effect graphs to elicit and self-review Gherkin requirements from natural language narratives, reporting improved quality, diversity, and consistency over baselines.","lead":"This paper presents RequireCEG, an agent that turns natural language app descriptions into structured testable requirement scenarios using causal-effect graphs and large language models. It reports higher quality, diversity, and consistency than prior methods, with 87% coverage of real website features in a small field test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline numbers rest on a closed-loop evaluation: RGPair narratives are ChatGPT summaries of the target Feature files and the quality/diversity judges are GPT-4.1-mini, so the reported gains may not survive contact with real end-user narratives.","rationale":"RequireCEG has genuine strengths: the architecture is clearly specified, the three-phase pipeline is novel, the ablation isolates the feature-tree and CEG contributions, and the authors release code and data. The self-healing CEG loop is a plausible mechanism for detecting missing preconditions and branches, and the field study's inter-rater agreement (Cohen's kappa > 0.82) is decent. However, the central empirical claim cannot be separated from the evaluation protocol. The reader's weakest_assumption correctly identifies RGPair's synthetic narratives and LLM-as-judge scoring as the weak link; I agree with that assessment. The strongest reason for calling this load-bearing rather than a minor methodological quibble is the closed loop: the benchmark input is generated from the target output, and the judge is in the same model family as the generator. Under those conditions, high scores can reflect prompt-formatting and summary-reconstruction ability rather than requirement quality for real end users. I therefore do not see grounds to reject the approach, but the current evidence is insufficient to accept the headline effect sizes; the appropriate action is to condition acceptance on human validation and an authentic-narrative benchmark. Since the reader already reached CONDITIONAL, my read does not change the verdict, but it sharpens the conditions under which the claim would be accepted.","tokens_in":32126,"tokens_out":6539,"duration_ms":72458,"concrete_test":"Run a single human-validated replication on a sample of 20 RGPair projects and 20 newly collected authentic narratives (written by non-experts for five new websites, without access to the sites). Have two independent human requirements engineers score blinded outputs of RequireCEG, AgileGen, and MetaGPT with the paper's INVEST and consistency rubrics, and also measure functional coverage against the original Feature files / real website function lists. Then compute human-vs-GPT-4.1-mini agreement and the human-rated effect sizes. If the human-rated RequireCEG advantage is substantially smaller than the reported 14.8% INVEST and 51.88% diversity gains, or if coverage on authentic narratives falls well below 87%, the current evidence does not support the headline claim and the paper needs revision before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that RequireCEG improves requirement quality, functional diversity, and consistency over SOTA—is load-bearing on the validity of RGPair as a proxy for end-user narratives and on LLM-as-a-judge scoring. Section 4.2.1 states that RGPair's narratives were created by prompting ChatGPT to 'Summarize the feature content as a user narrative based on the Gherkin Features,' i.e., the input is a reverse-engineered summary of the very Feature files used as ground truth. The task is therefore reconstruction of a known artifact, not elicitation from an authentic stakeholder; any method that aggressively expands the summary can look good. The metrics are not independent: INVEST and consistency scores are produced by GPT-4.1-mini (Section 5.2.2) and Fundiv entropy comes from an LLM performing FURPS classification, with no human validation or agreement statistics reported. Since RequireCEG itself is built on GPT-4o-mini, the same model family as the judge, self-preference and output-format artifacts can inflate the effect sizes. Moreover, the 51.88% diversity gain is partly built into the method: F@Num rises from 1-4 to 16.9 and the feature tree explicitly expands FURPS categories, so entropy increases almost mechanically. The field study does not close the gap: the abstract's 87% is Cover@Our (generated features that match some website function), while Cover@Real_Func—the proportion of actual website functions covered—is only about 70%, and evaluation used three graduate students, not end users, on five low-complexity websites.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes RequireCEG, a neuro-symbolic agent that uses feature trees and causal-effect graphs (CEGs) to elicit software requirements from natural language narratives and generate reviewed Gherkin scenarios. The central claim is that RequireCEG improves requirement quality (INVEST), functional diversity (FURPS entropy), and consistency over six baselines, supported by a new RGPair benchmark (40 GitHub projects, 413 Feature files), an ablation study, and a field study on five public websites reporting 87% coverage. The paper is clearly written and makes available code and data.","tokens_in":32430,"tokens_out":4695,"duration_ms":49716,"significance":"The neuro-symbolic combination of LLM reasoning with symbolic CEG review is a timely and plausible design; the feature-tree elicitation is a sensible mechanism to expand functional scope. The RGPair benchmark, if properly validated, would be a useful resource for the EUSE community. The paper's empirical claims, however, are currently grounded in a closed-loop evaluation: inputs are synthetic summaries of target Feature files (Section 4.2.1), all main metrics are LLM-as-a-judge (Section 5.2.2), and the human field study covers only five low-complexity websites with three participants. These threats need to be addressed before the quantitative claims can be accepted.","major_comments":[{"comment":"The RGPair narratives are generated by prompting ChatGPT to “Summarize the feature content as a user narrative based on the Gherkin Features” from the very Feature files that serve as ground truth. The evaluation never compares the generated Gherkin against the original Feature files, so the task is reconstruction of a known artifact rather than elicitation from an authentic stakeholder. The central claim of improving elicitation from genuine end-user narratives is therefore not supported. Please add an evaluation on authentic user narratives (Mini-RG has only 12 instances, Section 4.2.2) or, at minimum, a direct functional-overlap comparison between generated Gherkin and the original Feature files to demonstrate that the synthetic narratives preserve the target information.","section":"Section 4.2.1"},{"comment":"The INVEST scores, Fundiv classification, and consistency scores are all produced by LLM-as-a-judge using GPT-4.1-mini, with no human agreement or validation statistics reported for these metrics. Since RequireCEG itself is built on GPT-4o-mini (Section 4.3), the same model family as the judge, self-preference and output-format artifacts can inflate the reported improvements. Please provide a human-annotated evaluation on a random subset of the generated requirements (e.g., 20–30 projects) with inter-rater agreement, or use a judge from a different model family, to demonstrate that the metric scores are not artifacts of the evaluation setup.","section":"Section 5.2.2"},{"comment":"The Fundiv entropy metric is not corrected for the number of features or for the intentional spread across FURPS categories induced by the feature tree. RequireCEG outputs F@Num=16.925 while the second-best method has F@Num=4.25, and Section 3.2 explicitly designs features across satisfaction categories. The reported 51.88% diversity gain is therefore partly a structural consequence of generating more features in more categories, not a measure of intrinsic requirement quality. Please report diversity at matched feature counts or a normalized entropy (e.g., per-feature or per-category) to support the diversity claim.","section":"Section 5.2.3, Table 7"},{"comment":"The abstract's “87% coverage rate” refers to Cover@Our (proportion of generated features that match some real website function), while Table 10 shows the average Cover@Real_Func is about 70% (the proportion of actual website functions covered). The abstract and the Field Study Findings should state both metrics explicitly; as written, “coverage rate” is misleadingly presented as completeness relative to real functions. Please clarify this distinction in the abstract and the discussion.","section":"Section 6 and Abstract"},{"comment":"The human evaluation uses only five low-complexity AI-related websites and three graduate students, with no statistical significance tests reported. While Cohen's kappa > 0.82 is reported, the small scale and the homogeneity of participants limit the generalizability of the field-study claims, which are nevertheless used in the abstract's headline number. A larger or more diverse participant pool, or at least a clear statement of this limitation in the abstract, would be needed to support the current strength of the claim.","section":"Section 6.1.3"}],"minor_comments":[{"comment":"The column header “F@Sec” should be “F@Sce” for consistency with the notation used earlier in Section 5.1.2.","section":"Table 8"},{"comment":"The equation for Cover@Real_Func is incomplete: “Total number of pub-website (4)” should presumably read “Total number of pub-website functions.”","section":"Section 6.1.2"},{"comment":"The “Gherkin Example prompt template” applied to baselines is mentioned but not described; if it is identical to the template used in RequireCEG's Draft Gherkin Generator, please state this and include the prompt in an appendix to support the fairness of the comparisons.","section":"Section 4.3"},{"comment":"The Discussion repeats “About 87% of the generated features align with real website functions” without noting that this is Cover@Our and that Cover@Real_Func is about 70%; please add the clarification in the same paragraph.","section":"Section 7"},{"comment":"The MetaGPT(PRD) row reports no Acc@Syn value (“/”); please specify why or provide the value, since all other baselines report this metric.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a software engineering journal and presents a genuinely interesting neuro-symbolic architecture. However, the evaluation loop described in Sections 4.2.1 and 5.2.2 is a serious validity concern that should be addressed before publication. I recommend major revision rather than rejection: the authors have released code and data and can potentially break the loop with additional human evaluation or comparison against original Feature files."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RequireCEG is a serious piece of systems work, but the empirical evidence as it stands is circular at the core. The abstract's 87% coverage and 51.88% diversity gain are not yet credible because the benchmark narratives are ChatGPT summaries of the target Feature files, and the quality/diversity/consistency judges are in the same GPT family as the generator. That said, there is a real idea here, and the paper is honest about some of its own limitations.\n\nWhat's genuinely new: the neuro-symbolic pipeline—feature-tree scoping, LLM-built causal-effect graphs with a self-healing formal/semantic check loop, and CEG-based review of Gherkin scenarios—is a concrete contribution, not a repackaged prompt trick. The architecture is described in enough detail to reproduce, and the ablation study gives some signal that each module does what it claims: removing elicitation cuts diversity by roughly 61%, removing CEG review cuts internal consistency by about 30%. These are the right components to study.\n\nWhere it gets soft: RGPair is closed-loop. The 'user narratives' are generated by prompting ChatGPT to summarize the Gherkin features themselves, so the task is reconstruction rather than elicitation. The paper never compares against those original Feature files directly; it only compares methods against each other on the synthesized narratives. LLM-as-a-judge for INVEST and consistency, with GPT-4.1-mini judging outputs from GPT-4o-mini, compounds the problem. The diversity gain is partly mechanical: the feature tree is explicitly designed to expand FURPS categories, and F@Num jumps from 1-4 to about 17, so entropy increases almost by construction. The field study is a plus, but the abstract's 87% is Cover@Our (share of generated features that exist on the site); Cover@Real_Func averages around 70%, and the evaluators were three CS graduate students, not end users, on five low-complexity sites. The paper itself admits that initial CEGs are less than 30% correct before self-healing, and that some projects still fail the consistency check after 5 iterations—useful context, but it raises the question of how much of the quality gain depends on the self-healing loop actually succeeding.\n\nThis paper is for researchers working on LLM-based requirements elicitation or BDD automation. They will get a clear, plausible architecture and a useful set of ablation questions. The evaluation needs significant strengthening before the headline numbers can be relied on, but the method deserves a serious referee—the idea is solid and the flaws are fixable. I would want the authors to add a direct comparison to the original Feature files, human-validate a sample of the LLM judgments, and report the more honest real-function coverage in the abstract. I would not cite the effect sizes yet.","headline":"A real method with a self-referential evaluation: RequireCEG is worth engaging, but the headline numbers should not be trusted yet.","tokens_in":32990,"tokens_out":3082,"would_cite":false,"duration_ms":34327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Causal-effect graphs turn vague app requests into testable requirements.","keywords":["requirement elicitation","causal-effect graph","Gherkin","neuro-symbolic collaboration","feature tree","end-user software engineering","LLM agents","requirement review"],"falsifier":"Take a fixed set of applications, collect narratives written directly by non-technical users without any Gherkin knowledge, run RequireCEG on them, and have requirement-engineering experts who are blind to the source rank the outputs against professionally authored Gherkin for business consistency and completeness; if the expert-authored requirements match or beat the generated ones, or if the diversity and consistency scores drop sharply on these genuine narratives, the central claim of practical superiority is refuted.","tokens_in":31917,"feed_emoji":"🧩","tokens_out":11004,"duration_ms":102120,"temperature":0.7,"pith_summary":"The paper's claim is that LLM-generated software requirements fall short because nothing in the generation loop forces causal logic between preconditions and behavior actions. RequireCEG is proposed as a remedy: a neuro-symbolic agent that turns a user's vague narrative into a feature tree, derives causal-effect graphs that link atomic conditions to system responses, repairs those graphs through formal and semantic self-checks, and then uses the healed graphs to review and augment Gherkin scenarios. The paper argues that this loop makes the final requirements more complete, more diverse, and more consistent than what current Gherkin-generation methods produce, citing a 51.88% improvement in functional diversity and an 87% coverage of real functions on five public websites. If these results hold, end users could produce testable acceptance criteria from a single narrative paragraph without a requirements analyst in the loop.","feed_headline":"Causal-effect graphs pin down AI-generated app requirements","feed_subtitle":"A feature tree plus a self-healing causal graph turns one vague narrative into Gherkin with 87% function coverage.","key_machinery":"The load-bearing object is the causal-effect graph (CEG), a symbolic graph whose nodes are atomic preconditions (causes) and behavior actions (effects), linked by Boolean operators (DIR, AND, OR, NOT) and constrained among same-type nodes by EXC, INC, REQ, XOR, and MSK. It carries the argument by acting as a formal oracle the LLM cannot talk its way around: the graph is constructed from the elicited system behaviors, repaired until it passes a format check and an intervention-question semantic check, and then used to review and extend the draft Gherkin scenarios. Supporting that machinery are the feature tree, which scopes software components hierarchically and is the main source of functional diversity, and the generate-check-modify loop that makes the CEG self-healing.","core_discovery":"RequireCEG establishes that the missing ingredient in current LLM-based requirement elicitation is explicit causal reasoning about what must hold before a behavior is allowed to happen. Given only a natural-language narrative, the agent first elicits a feature tree to scope software components and describes each behavior from both the user-operation and system-response perspectives. It then decomposes those behaviors into atomic conditions (causes) and effects, builds a causal-effect graph using Boolean operators such as DIR, AND, OR, and NOT together with constraints like EXC, INC, REQ, XOR, and MSK, and runs a self-healing loop: a formal syntax check on the graph expressions, followed by a semantic check in which each condition is set false and the LLM must reason about whether the requirement still justifies the effect. The healed graph is then used to review the draft Gherkin, repairing logical mismatches between Given-When-Then clauses and adding scenarios that cover missing causal branches. The paper reports that this architecture outperforms six baseline methods on requirement quality, functional-diversity entropy, and internal and business consistency, and that in a human field study on five live websites, 87% of the generated features corresponded to real functions on the sites.","pith_inferences":["Because the narratives in RGPair were generated from Feature files rather than collected from users, a natural next experiment is to re-run the pipeline on verbatim user requests from app reviews or support tickets and measure whether the diversity and consistency gains persist on genuinely noisy input.","The same precondition-action discipline could transfer to other artifacts: API contract validation, UI acceptance criteria, or automated test-case generation, since enforcing causal consistency is a generic problem, not specific to Gherkin.","The reported 51.88% diversity gain could be more sharply attributed by applying per-category significance tests across the five FURPS classes, rather than relying on the aggregate entropy score alone.","A cheaper ablation than changing the whole pipeline would be to swap the base LLM for an open-weight model in the same agent chaining; if the gains persist, the CEG architecture rather than a particular model is the source of the improvement."],"forward_implications":["A non-technical user's single narrative paragraph can be converted, end to end and without human intervention, into a multi-feature set of executable Gherkin acceptance criteria.","Because the CEG records every precondition-action link, a failed test in the subsequently generated application can be traced back to a specific requirement node, making AI-generated software auditable.","The self-healing check means the symbolic graph, not just the LLM, is what guarantees consistency, so the approach can be pushed to larger, multi-component applications.","The feature-tree elicitation stage is what carries the diversity gains; removing it cuts functional diversity by more than 61%, so scoping and satisfaction modeling are not optional extras.","The reviewed Gherkin scenarios are ready for Behavior-Driven Development tooling, closing the loop between end-user requirements and Cucumber-style executable specifications."],"supporting_citations":[{"why":"Defines Gherkin, the domain-specific language that is both end-user-readable and executable, which the entire pipeline produces and reviews.","marker":"[36]"},{"why":"Supplies the End-User Software Engineering vision and the three criteria (structure, feasibility, implicit needs) that justify Gherkin as the shared language.","marker":"[42]"},{"why":"Introduces causal-effect graphs, the symbolic formalism at the heart of the review mechanism.","marker":"[12]"},{"why":"Is the AgileGen baseline whose limited causal logic and functional diversity motivate the work and set the state of the art to beat.","marker":"[53]"},{"why":"Provides the curated Gherkin open-source dataset that RGPair is built from after filtering and augmentation.","marker":"[9]"},{"why":"Is the closest prior approach for LLM-based Gherkin generation and the source of the syntactic-accuracy metric used for RQ1.","marker":"[22]"},{"why":"Codifies the LLM-as-a-judge protocol used for INVEST, diversity, and consistency scoring in RQ2.","marker":"[14]"},{"why":"Defines the entropy measure that quantifies functional diversity as the distribution of features across FURPS categories.","marker":"[44]"},{"why":"Supplies the Kano satisfaction categories that the feature-tree elicitor uses to widen functional scope beyond the literal narrative.","marker":"[21]"},{"why":"Gives the visual feature-tree modeling reference that guides the hierarchical scoping module.","marker":"[5]"}],"fun_headline_variants":["Causal graphs turn vague app requests into clear specs","Self-healing causal graphs make AI requirements more consistent","Neuro-symbolic agent with causal graphs boosts requirement quality","Causal-effect reasoning sharpens AI-generated software specs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation depends on treating large-language-model summaries of existing Gherkin files as if they were real end-user narratives, and on trusting an automated LLM judge's numeric ratings of quality, diversity, and consistency.","fun_headline_variants_meta":{"raw":{"variants":["Causal graphs turn vague app requests into clear specs","Self-healing causal graphs make AI requirements more consistent","Neuro-symbolic agent with causal graphs boosts requirement quality","Causal-effect reasoning sharpens AI-generated software specs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1494,"prompt_tokens":1037,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":653,"tokens_out":457,"duration_ms":5239,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:43:13.392936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of applications, collect narratives written directly by non-technical users without any Gherkin knowledge, run RequireCEG on them, and have requirement-engineering experts who are blind to the source rank the outputs against professionally authored Gherkin for business consistency and completeness; if the expert-authored requirements match or beat the generated ones, or if the diversity and consistency scores drop sharply on these genuine narratives, the central claim of practical superiority is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Gherkin, the domain-specific language that is both end-user-readable and executable, which the entire pipeline produces and reviews."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces causal-effect graphs, the symbolic formalism at the heart of the review mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the curated Gherkin open-source dataset that RGPair is built from after filtering and augmentation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the closest prior approach for LLM-based Gherkin generation and the source of the syntactic-accuracy metric used for RQ1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Kano satisfaction categories that the feature-tree elicitor uses to widen functional scope beyond the literal narrative."},{"cited_title":"Beatty and A","cited_arxiv_id":null,"evidence_quote":"Gives the visual feature-tree modeling reference that guides the hierarchical scoping module."}],"review_version":1}