{"id":"d3079e2c-f633-4c8b-aa4d-775fd2a31004","arxiv_id":"2604.23509","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SeGa detects 29 of 60 real business-logic bugs by generating tests from requirement-document scenarios, 22-25 more than four LLM-based baselines.","lead":"This paper presents SeGa, a system that turns product requirement documents into structured business rules and uses them to guide an LLM in generating unit tests that expose business-logic bugs. In four industrial Go projects with 60 known bugs, SeGa found 29, versus 4-7 for four existing LLM-based generators; in production deployment it surfaced 16 bugs developers fixed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline bug counts rest on internally inconsistent subject statistics: Section 4.1 assigns 12/9/37/2 bugs, Table 1 lists 37/9/2/12, and the paper gives no per-subject TP/FP counts to resolve the discrepancy.","rationale":"The reader identified PRD completeness and accuracy as the load-bearing assumption. That is a real external-validity caveat, consistent with the paper's own 'oracle hallucination' and requirements-drift passages in Section 6, and it appropriately conditions generalizability. However, it does not bear on the internal validity of the 29-vs-7 count: TP judgments are made by matching generated failures to developer-annotated ground-truth bugs, not by trusting PRD-derived oracles. The more immediate, concrete problem is that the paper's own subject statistics are contradictory: Section 4.1, Table 1, and the per-subject recalls in Table 2 cannot all be true as printed. The prose and Table 2 are consistent with each other and with the aggregate 29, but Table 1's Bugs column is not, and no raw TP/FP counts are provided to disambiguate. This matters because the headline advantage is a small-integer difference, so the central claim should not be treated as settled until the tables and counts are reconciled. The reader's verdict of CONDITIONAL already captures this need for correction/artifact release, so my stress-test does not move the verdict; it sharpens the reason by pointing to a reproducible arithmetic conflict rather than only to an external-input assumption.","tokens_in":19749,"tokens_out":8750,"duration_ms":99868,"concrete_test":"Publish per-subject TP/FP confusion matrices for SeGa and all four baselines, and reconcile Table 1's Bugs column with Section 4.1. Then recompute aggregate TP = 29 and precision = 0.73 from those counts. If the corrected tables reproduce the headline and precision, the data-integrity objection is resolved; if the totals shift, the 22-25-bug claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (SeGa detects 29 bugs vs 7/7/6/4, precision 0.73) depends on a benchmark whose ground-truth bug counts are stated inconsistently. Section 4.1 says Subject-1/2/3/4 contain 12/9/37/2 bugs; Table 1 lists Bugs as 37/9/2/12. These cannot both be right. Section 5.1.2 says SeGa detects 5 bugs on Subject-1 and 16 on Subject-3. With Table 2 recalls of 0.42 and 0.43, those TP counts imply approximately 5.0 and 15.9, which only matches the 12/37 assignment from Section 4.1. Under Table 1's 37/2 assignment, the same recalls would imply about 15.5 and 0.86 TPs, and the reported 29-total would not reproduce. Thus the prose and Table 2 are mutually consistent while Table 1's Bugs column appears swapped/mislabeled, but a reader cannot tell which table is authoritative. Furthermore, the paper does not report per-subject TP/FP confusion matrices, so even the aggregate precision of 0.73 cannot be recomputed from the printed tables. Since the headline advantage is a difference among small integers (22, 22, 23, 25), a single mis-assigned subject could change the per-project narrative and, if the inconsistency indicates deeper data-handling errors, the aggregate claim. No artifact or raw counts are released, so the 29-vs-7/7/6/4 result is not independently checkable from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SeGa is a semantics-driven unit-test generation technique for business logic bugs. It constructs a structured DSL knowledge base of \"functionality entries\" from product requirement documents (PRDs), uses an LLM-based semantic reasoning agent to retrieve the functionalities relevant to each focal method, refines them into business scenarios with explicit preconditions, actions, expected outcomes, and semantic constraints, and generates executable unit tests with a separate compilation-repair component. The evaluation on four industrial Go projects containing 60 real-world bugs reports that SeGa detects 29 bugs versus 7, 7, 6, and 4 for CHATTESTER, SymPrompt, HITS, and RATester, with aggregate precision 0.73 versus 0.54-0.57, plus an ablation study and a deployment on 6 production repositories that found 16 previously unknown bugs confirmed and fixed by developers. The paper also reports lessons learned about relevance, false positives, PRD completeness, oracle hallucination, and deployment in CI/CD.","tokens_in":20152,"tokens_out":9228,"duration_ms":98615,"significance":"If the quantitative claims hold, this is a valuable industrial experience paper for a practically important but under-served bug class. The design is not circular: the oracles come from external PRDs and the benchmark uses independently maintained bug annotations, and the paper includes direct comparisons, a component ablation, and a deployment study with developer confirmation. Those are real strengths. However, the headline numbers currently rest on an inconsistent subject table and on non-released proprietary data, so the central 29-vs-7/7/6/4 result is not yet independently verifiable from the manuscript. The paper's honest analysis of false-positive causes and its lessons learned are additional strengths, but the load-bearing evaluation data need to be made transparent before the result can be fully accepted.","major_comments":[{"comment":"Section 4.1 states that Subject-1..4 contain 12/9/37/2 bugs, while Table 1's Bugs column reads 37/9/2/12. These cannot both be correct. Cross-checking Table 2 with §5.1.2: SeGa has recall 0.42 on Subject-1 and is said to detect 5 bugs there, implying a denominator of 12, not 37; on Subject-3, recall 0.43 and 16 detected bugs imply a denominator of 37, not 2. Thus Table 1 appears to swap Subject-1 and Subject-4 bug counts. Please correct Table 1 and provide per-subject TP/FP/FN counts (ideally a confusion matrix) so that the aggregate 29 bugs and precision 0.73 can be recomputed. This is material because all reported numbers are small integers and the headline advantage over baselines is only 22-25 bugs.","section":"§4.1 / Table 1 / Table 2"},{"comment":"The TP/FP labels in RQ1 are assigned by \"manual inspection\" comparing failures with annotated bug ranges, but the paper does not say who performed the matching, whether they were blind to the technique, whether there was independent verification, or what inter-rater agreement was. In an industrial study with proprietary data, this creates a risk of labeling bias favoring the proposed method. Please report the labeling protocol, inter-rater reliability (or at least a second annotator's agreement), and per-subject confusion matrices. Without these, the precision comparison (0.73 versus 0.54-0.57) cannot be independently validated.","section":"§4.3.1 / §5.1.2"},{"comment":"CHATTESTER, SymPrompt, and HITS are adapted from Java/Python to Go, and RATester's local LLM calls are replaced; no adapted prompts, implementations, or artifacts are released. The statement that the authors \"strictly followed\" the original papers is not sufficient to rule out adaptation bias, especially because the weakness of the baselines is the central result. Please release the adapted code/prompts and report per-technique compilation success rates, numbers of generated and executed tests, and how the one-test-file-per-method configuration interacts with each baseline's intended workflow. Otherwise the 22-25 bug advantage may partly reflect weaker baseline configurations.","section":"§4.2 / §4.4 / §7"},{"comment":"The paper honestly documents that false positives arise from ambiguous, incomplete, or outdated PRDs and from \"oracle hallucination,\" but it does not quantify how sensitive the central result is to PRD completeness. A failure analysis of the 31 missed bugs is needed: how many were missed because the PRD lacked or obscured the relevant requirement, versus failures in retrieval, scenario derivation, test generation, or execution? Such an analysis would clarify the scope of the claimed advantage and directly inform the lessons in Section 6.","section":"§5.1.2 / §6 / §7"}],"minor_comments":[{"comment":"The claim that precision improves by 26.9%-34.3% does not match the rounded aggregate values in Table 2: relative improvements are about 28.1%, 32.7%, and 35.2% against 0.57, 0.55, and 0.54, and the absolute improvement is 18-19 percentage points. Please state whether the percentages are relative or absolute and recompute them consistently.","section":"Abstract / §1 / §9"},{"comment":"The overlap diagram is hard to read: the repeated zero labels and ambiguous intersections make it difficult to verify the 29/7/7/6/4 counts. Please replace or supplement it with a table of per-technique detected-bug IDs or a standard Venn-diagram data table.","section":"Figure 3"},{"comment":"The subject identifiers in Table 1 use inconsistent capitalization (subject-1 vs Subject-1) and the Bugs column is inconsistent with Section 4.1. Also, the \"Requirements\" column (e.g., 106 requirements for a 115,048-token PRD) seems surprisingly coarse; clarify the granularity of a \"requirement\" in this table.","section":"Table 1"},{"comment":"Typo: \"realatively\" should be \"relatively\" in the overhead discussion.","section":"§7"},{"comment":"References [18] and [42] are cited as prior work for designing the functionality DSL, but the cited titles appear to concern binary lifting and FFT semantics lifting, which seem unrelated to requirements engineering DSLs. Please verify and replace these citations.","section":"References [18] and [42]"},{"comment":"The statement that buggy code changes involve \"91.35 lines on average\" is overly precise for a small sample; report the median and range as well, or round to a meaningful precision.","section":"§5.3.2"}],"recommendation":"major_revision","confidential_remarks":"This is a strong and potentially publishable industrial experience paper, but the inconsistency between Section 4.1 and Table 1 is exactly the kind of issue that undermines confidence in the headline numbers. I would ask the authors for corrected tables, per-subject confusion matrices, and a detailed labeling protocol; if those are provided, the paper could be acceptable. The proprietary nature of the benchmark will remain a limitation, but it is not by itself disqualifying for an industrial experience report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is a genuinely interesting pipeline: SeGa extracts structured functionality entries and fine-grained business scenarios from PRDs, retrieves what is relevant to a focal method, and uses that as semantic context for LLM unit test generation. That is more than \"throw the requirements at the model,\" and the ablation shows each component earns its keep—raw PRDs, unfiltered functionalities, no scenario derivation, and no repair all hurt. Second, the headline claim (29 bugs vs 7/7/6/4, precision 0.73) is plausible but not fully checkable from the manuscript, and there is a concrete inconsistency that needs fixing before I would trust the per-subject narrative or cite the numbers.\n\nThe good parts: the motivating example is clear, the DSL design is reasonable, and the evaluation is an experience paper's strongest asset—60 real business-logic bugs across four industrial Go projects, plus a deployment that found 16 bugs developers confirmed and fixed. The gap over code-centric baselines is large and directionally makes sense: code-centric generators encode current behavior, while SeGa can encode intended behavior from requirements. The lessons learned section is honest, especially about false positives from incomplete PRDs and \"oracle hallucination.\"\n\nThe soft spots, in proportion: the stress-test note is right. Section 4.1 says Subject 1/2/3/4 have 12/9/37/2 bugs; Table 1 lists 37/9/2/12. Table 2's recalls (0.42, 0.67, 0.43, 1.00) imply TPs of roughly 5, 6, 16, 2, which only matches the 12/9/37/2 assignment. So Table 1 appears to have the Bugs column misordered, and the aggregate 29 is probably correct—but a reader cannot tell, and the paper reports no per-subject TP/FP counts, so even aggregate precision is not recomputable from the tables. The dataset is proprietary and no artifact is released, so the 29-vs-7 result rests entirely on the authors' manual matching, with no inter-rater reliability reported. That is common in industrial experience papers, but it means the claim should be treated as evidence, not as a settled benchmark. One smaller point: the RQ3 deployment says no baseline reported any issues, but does not say how baselines were applied in CI; a sentence on that would help.\n\nWho is this for? Anyone working on requirements-aware testing or LLM-based test generation. It deserves a serious referee, but I would not yet cite the headline numbers as established fact. If this lands at a venue with artifact evaluation, push for at least the per-subject counts and the corrected table.","headline":"Likely real effect, but the headline numbers sit on a table inconsistency and an unreleased dataset—worth refereeing, not yet citable as settled.","tokens_in":20613,"tokens_out":2016,"would_cite":false,"duration_ms":21813,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Testing against documented intent catches 29 real business-logic bugs that code-centric tests miss.","keywords":["business logic bugs","semantics-driven test generation","product requirement documents","LLM-based unit tests","test oracle","functionality DSL","business scenarios","Go"],"falsifier":"Take two sets of real business-logic bugs matched in complexity — one whose requirements are fully and unambiguously documented in the PRD, one whose requirements are undocumented or ambiguous. If SeGa's detection rate on the undocumented set falls to the level of the code-centric baselines, the advantage is an artifact of documentation quality, not of the semantic machinery.","tokens_in":19631,"feed_emoji":"🐛","tokens_out":3857,"duration_ms":39722,"temperature":0.7,"pith_summary":"This paper argues that business-logic bugs — cases where code is syntactically valid but violates what the software is supposed to do — stay invisible to unit test generators that read only source code. SeGa converts product requirement documents into a structured knowledge base of \"functionalities,\" retrieves the ones relevant to each method under test, and derives fine-grained \"business scenarios\" with explicit preconditions, actions, expected outcomes, and constraints. Those scenarios act as a semantic oracle that lets an LLM write tests asserting intended behavior, which can contradict the flawed implementation. In a study of 60 real bugs from four industrial Go projects, SeGa detected 29 bugs versus 4–7 for four LLM-based generators, at higher precision, and in six production repositories it surfaced 16 previously unknown bugs that developers fixed.","feed_headline":"Testing against requirement docs uncovers 29 business bugs","feed_subtitle":"Code-centric generators found 4-7 of the same 60; live deployment caught 16 more, all fixed.","key_machinery":"The carrying mechanism is the business-scenario DSL: each scenario isolates one requirement from a retrieved functionality and states it as preconditions, a triggering action, expected outcomes, and structured semantic constraints. This turns a natural-language requirement into an executable oracle that can disagree with the implementation. Around it sits the functionality DSL knowledge base (which groups requirements under a business intent), an agent that summarizes a focal method's intent to retrieve relevant functionalities, and a standalone compilation-repair component that keeps generated tests executable.","core_discovery":"SeGa's central claim is that the mismatch between implemented code and documented business semantics is discoverable at the unit level if the test generator is given the right slice of the requirements. The paper shows that raw PRDs are too noisy to use directly, but a two-level structured representation — functionality entries grouping related requirements, refined into concrete business scenarios with preconditions, triggering actions, expected outcomes, and semantic constraints — provides an actionable oracle. Using these scenarios, SeGa generated tests that exposed 29 of 60 real-world business logic bugs, compared to 4–7 for code-centric LLM generators, and during deployment uncovered 16","pith_inferences":["SeGa's effectiveness is bounded by PRD quality; the same machinery could be repurposed to flag ambiguous, incomplete, or drifting requirements as a documentation-quality check.","The retrieval-plus-scenario pattern could generalize to other specification artifacts (API docs, regulatory rules) and other languages, provided a similar structured oracle can be built.","A natural extension is a feedback loop that uses developer accept/reject decisions on reported bugs to prune or correct future scenarios, reducing the documented false positives from oracle hallucination.","Comparing SeGa on bugs whose requirements are fully documented versus undocumented would isolate how much of the advantage comes from semantics versus test construction quality."],"forward_implications":["If requirements are available, unit tests can be generated against intended behavior rather than implemented behavior, exposing a bug class that coverage-driven tools miss.","Retrieving only the requirements relevant to a focal method matters more than feeding the generator more semantic context.","A structured, scenario-level representation of requirements (preconditions, actions, outcomes, constraints) is what makes the oracle actionable for LLM-based test generation.","Standalone compilation repair improves the executability of LLM-generated tests without distracting the generation agent.","Deploying semantics-driven test generation in CI/CD can find business-logic bugs before merge, in code that already passes existing tests."],"fun_headline_variants":["Requirement-driven tests catch 29 business bugs","Semantics-based testing finds 29 real bugs","Req-aware test generation beats code-centric by 5x","From PRDs to tests: 29 bugs exposed","SeGa: requirement-guided unit tests uncover 29 bugs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the product requirement documents are a complete, current, and unambiguous description of intended behavior; if a requirement is missing, stale, or open to multiple readings, SeGa's tests will assert the wrong behavior.","fun_headline_variants_meta":{"raw":{"variants":["Requirement-driven tests catch 29 business bugs","Semantics-based testing finds 29 real bugs","Req-aware test generation beats code-centric by 5x","From PRDs to tests: 29 bugs exposed","SeGa: requirement-guided unit tests uncover 29 bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000131,"raw_usage":{"total_tokens":942,"prompt_tokens":699,"completion_tokens":243,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":167}},"tokens_in":443,"tokens_out":243,"duration_ms":3435,"temperature":1.0,"reasoning_tokens":167,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T05:21:29.994927+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two sets of real business-logic bugs matched in complexity — one whose requirements are fully and unambiguously documented in the PRD, one whose requirements are undocumented or ambiguous. If SeGa's detection rate on the undocumented set falls to the level of the code-centric baselines, the advantage is an artifact of documentation quality, not of the semantic machinery.","supporting_citations":[],"review_version":2}