{"id":"32c1db1e-e3a9-4a13-8d44-d5475658f641","arxiv_id":"2412.11731","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EvoGURI, a domain-specific extension of EvoMaster, achieved top rule coverage and near-perfect mutation scores when testing a medical rule engine with a strict OpenAPI schema.","lead":"A study at Norway's cancer registry compared five automated testing tools for its medical rule engine, including a new domain-aware tool called EvoGURI, and found that domain-specific rule targets plus a strict API schema give the best fault detection. The authors recommend EvoGURI for routine use, with mutation scores of 0.96 for validation rules and 1.0 for aggregation rules.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ5 mutation scores are single un-replicated observations per tool, so the EvoGURI-over-EvoMaster-BB recommendation rests on one unrepeated measurement.","rationale":"The paper is a genuinely useful industrial case study: RQ1-RQ4 use 10 versions, 30 repetitions, and appropriate non-parametric tests plus effect sizes, and the threats-to-validity section is unusually candid. The soft spot is the pivot to the final recommendation. RQ5 is the only place EvoGURI separates from EvoMaster-BB (0.96 vs 0.00 in Table 8), and it rests on a single suite per tool chosen by median code coverage, not by any rule-related criterion. RQ4 itself reports large repetition-to-repetition variance in rule-result frequencies, so the selected suite may not be representative of the rule targets that determine mutant killing. Section 3.8's explicit statement that no statistical tests were run on RQ5 confirms the absence of any variance estimate. The synthetic mutants are also generated by eight syntactic rule operators and described as quick to kill, so even a stable score would need validation against real rule faults before grounding an operational adoption decision. These are limitations of evidence rather than internal contradictions, and the authors disclose them honestly. The reader's CONDITIONAL verdict is therefore appropriate; I would not move the verdict, but the conditional should explicitly require replication of RQ5 across repetitions and a check against real rule evolution before EvoGURI is adopted as the sole automated testing solution.","tokens_in":27274,"tokens_out":7792,"duration_ms":76344,"concrete_test":"Re-run the RQ5 mutation experiment on all 30 repetitions of GURI v10 with the strict OAS (or on a random sample of at least 10 repetitions if the full 63-day cost is prohibitive), computing M_SV and M_SA per repetition for each tool. If EvoGURI's 0.96 and EvoMaster-BB's 0.00 are not stable, e.g., the 95% bootstrap interval for their difference includes zero, the recommendation to prefer EvoGURI over EvoMaster-BB is unsupported. As a secondary check on mutant realism, compare detection rates against the real version-to-version rule diffs from Table 1 treated as natural mutants.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recommendation (use EvoGURI at the CRN) turns on RQ5: Table 8 gives EvoGURI M_SV=0.96 while EvoMaster-BB is 0.00, and the text concludes EvoGURI should be used because of its high mutation scores and simpler setup than the white-box tools. But Section 3.7 states that RQ5 executes only one suite per tool, selected as 'the repetition with the median code coverage (from RQ1)', and Section 3.8 explicitly disclaims statistical testing because there is 'only a single MS value per tool'. The selection criterion is misaligned with the mutation metric: a suite can have median code coverage yet atypical rule-result coverage, and RQ4 Observation 4 documents substantial repetition variance in rule-execution frequencies. A different repetition could plausibly lower EvoGURI's 0.96 or raise EvoMaster-BB's 0.00, collapsing the only quantitative separation between these two tools. The paper's own observation in Section 5 that the mutants are 'relatively easy to kill' further suggests the high scores are an upper-bound sensitivity measure rather than a validated fault-detection rate. Thus the recommendation's decisive evidence is a single un-replicated observation on unvalidated synthetic mutants.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an industrial case study on automated testing of GURI, the medical rule engine of the Cancer Registry of Norway (CRN), which validates cancer messages and aggregates cancer cases using evolving medical rules. The authors compare five REST API test-generation tools: EvoMaster in black-box mode (EvoMaster-BB), three white-box variants (MIO, MOSA, WTS), and EvoGURI, a novel extension of EvoMaster-BB that adds rule-result pairs as search targets. Across ten versions of GURI, the study measures code coverage (RQ1), code errors (RQ2), domain-specific rule-result coverage (RQ3), rule-result execution frequencies compared with production data (RQ4), and mutation testing with 2,279 rule mutants produced by eight mutation operators (RQ5). The results show similar code coverage and error counts across tools; EvoGURI and EvoMaster-BB are the strongest for rule coverage; EvoMaster-WB-MIO achieves the highest Pass/Fail execution frequencies; and in mutation testing EvoGURI, EvoMaster-WB-MOSA, and EvoMaster-WB-WTS achieve near-perfect mutation scores while EvoMaster-BB scores zero. The paper recommends EvoGURI as the automated testing solution for GURI, citing its high mutation scores and simpler setup than the white-box tools.","tokens_in":27501,"tokens_out":20386,"duration_ms":167295,"significance":"The study's strengths are substantial: a real-world subject with production execution data for comparison, a careful experimental protocol for RQ1-RQ4 (30 repetitions, randomized interleaved trials, Friedman tests with Nemenyi post-hoc analysis, Benjamini-Yekutieli false-discovery-rate control, and Vargha-Delaney effect sizes), and domain-specific evaluation metrics that go beyond code coverage. The comparison of generated rule-result frequencies with production GURI (RQ4) is an uncommon and valuable contribution, and the discussion of domain-specific objectives, the oracle problem, and synthetic medical data generation is useful for the broader search-based-testing community. If the results hold, the paper provides a concrete template for adapting REST API test generation to rule-based medical systems. However, the significance of the headline recommendation depends on the mutation-testing evidence (RQ5), which is the least robust part of the study: it uses a single un-replicated test suite per tool, the mutant set is unvalidated, and this is the only metric that decisively separates EvoGURI from the much simpler EvoMaster-BB.","major_comments":[{"comment":"The recommendation to adopt EvoGURI rests on RQ5, yet RQ5 uses a single un-replicated test suite per tool: Section 3.7 states that mutation testing is executed only for version v10 and only for 'the repetition with the median code coverage (from RQ1)', and Section 3.8 explicitly declines statistical testing because there is 'only a single MS value per tool'. This RQ5 comparison is the only decisive separation between EvoGURI (M_SV=0.96) and EvoMaster-BB (M_SV=0.00); in RQ1-RQ3 the two tools are statistically equivalent (Tables 3 and 5), with only small, version-dependent differences in RQ2 and RQ4. The selection criterion is misaligned with the mutation metric: Table 3 shows near-zero variance in code coverage across repetitions (e.g., EvoGURI line coverage 20.99%±0.11), so the 'median coverage' rule does not select a suite representative of rule-execution behavior, and Section 4.4.4 (Observation 4) documents large repetition-to-repetition variance in rule-result frequencies. A different repetition could plausibly lower EvoGURI's score or raise EvoMaster-BB's score, collapsing the only quantitative separation between the two tools. I ask the authors to run the mutation analysis on several repetitions (e.g., the suites of 5-10 repetitions, possibly with mutant sampling to control cost) and report the distribution or range of mutation scores, or at minimum to verify that the EvoGURI-versus-EvoMaster-BB ordering is stable across the repetitions with the highest and lowest rule coverage.","section":"§3.7, §3.8, §4.4.4, Table 8"},{"comment":"The mutant set is not established as representative of real rule faults, and the paper's own statements undercut its use as a decision metric in the RQ5 Summary. Section 5 concedes that 'the mutations are relatively easy to kill', and Section 3.9 acknowledges that 'it is unclear whether these domain-specific metrics are correlated with \"good\" test cases' for the domain experts. The eight operators in Table 2 are purely syntactic perturbations, and no evidence is given that their difficulty distribution resembles the actual rule modifications documented in the version history (Table 1). A concrete validation is available within the manuscript's own scope: the ten versions contain real rule changes, so the authors could check whether the suites generated for version i detect the actual rule modifications between version i and version i+1, or could compare the killability of artificial mutants with the difficulty of those real diffs. Without such validation, the mutation scores in Table 8 should be reported as an upper-bound sensitivity measure rather than as a validated fault-detection rate.","section":"§3.5, §5, §3.9"},{"comment":"The comparison between EvoGURI and EvoMaster-BB in RQ5 is confounded with the archive-retention policy. EvoGURI differs from EvoMaster-BB only by adding rule-result targets (Section 3.3), and the authors themselves attribute EvoMaster-BB's M_SV=0.00 to 'the low number of retained test cases in the final test suite' (Section 5). Since RQ3 (Table 5) shows the two tools covering essentially identical rule-result sets, the case for EvoGURI over the simpler EvoMaster-BB may reflect how many tests each tool's archive keeps rather than the diagnostic power of the domain-specific targets. A control experiment is needed, e.g., running EvoMaster-BB with an archive cap comparable to EvoGURI's suite size (about 43 tests), or reporting the rule-result coverage of the archived suites themselves. Without such a control, the conclusion in Section 8 that 'EvoGURI is the most effective tool' overstates what the evidence shows.","section":"§3.3, §5, §8"}],"minor_comments":[{"comment":"There are several missing spaces and a slip in the abstract: 'testingGURI' and 'testGURI's 10 versions' (§1) lack the space before 'GURI', and 'identify artificial faults ten versions of GURI' (Abstract) is missing 'in' before 'ten'. Please proofread the manuscript.","section":"Abstract, §1"},{"comment":"The claim 'covering 0.50 percentage points (pp) more Pass and 0.60 pp more Fail results' is inconsistent with Table 5, where the strict-versus-default difference for EvoGURI validation rules is 26.95 pp (Pass) and 32.73 pp (Fail); the values 0.50 and 0.60 appear to be proportion differences (26.95/54.1 and 32.73/54.1), not percentage points, so 'vastly improved' is also difficult to reconcile with the stated 0.50 pp.","section":"§1"},{"comment":"'Medical personell' should be 'medical personnel'.","section":"§2"},{"comment":"'This is also supported by the statistical tests from Table 5' should refer to Table 6, which contains the rule-result frequency comparisons.","section":"§4.4.1"},{"comment":"'against each of the 2,279 mutation' should be 'against each of the 2,279 mutations'.","section":"§3.7"},{"comment":"'EvoGURI and EvoMaster-BB are tied for the most effective tool' should read 'tied as the most effective tools'.","section":"§4.2 (RQ2 Summary)"},{"comment":"Please specify the kill criterion for RQ5: the definition of M_S as mutants that 'lead to a failing test' (Section 3.6) does not state how a failure is detected when a test suite is executed against a mutated rule engine (e.g., any change in rule results, or any change in HTTP status code). This matters for interpreting the 0.00 scores in Table 8.","section":"§3.6, §5"}],"recommendation":"major_revision","confidential_remarks":"For the editor: This is a solid industrial case study, and the RQ1-RQ4 protocol is exemplary. My main concern is the weight the paper places on RQ5: the single-replication mutation experiment is the only decisive evidence separating EvoGURI from the much simpler EvoMaster-BB, and the authors' own explanation of the gap (number of retained tests) suggests the comparison may reflect archive policy rather than testing capability. The paper can likely be brought to a publishable state with (1) a robustness analysis of RQ5 over a handful of repetitions, (2) a control for the archive-size confound, and (3) appropriately softened conclusions, or a validation of the mutant set against the real rule diffs. The authors are well positioned to do this since they already have the HPC infrastructure and the 30 repetitions' suites. I would also draw the editor's attention to the misleading '0.50 pp / 0.60 pp' claim in Section 1, which contradicts Table 5."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing. First, this is a genuine industrial case study: ten versions of a real medical rule engine, five tool configurations, 30 repetitions, randomized ordering, proper non-parametric tests, and a rare comparison of generated rule-result frequencies against production data. Second, the paper's headline recommendation—use EvoGURI at the Cancer Registry of Norway—is largely carried by RQ5, and RQ5 is the one place where the experimental design is weakest: each tool contributes a single test suite, chosen by median code coverage, and the mutation scores are single un-replicated observations.\n\nWhat is new here is worth credit. EvoGURI extends EvoMaster-BB with explicit rule-result targets, a sensible domain-specific adaptation; the manually constrained OpenAPI schema (\"strict OAS\") is a practical, transferable intervention; and the 2,279-mutant rule-mutation study is substantial. RQ1–RQ4 are handled carefully: the statistics are appropriate, effect sizes are reported, and the production comparison in RQ4 gives the work an empirical anchor most API-testing case studies lack. The authors are also honest in the text about the limits of their metrics and the single-subject setting.\n\nThe soft spots are real, and the stress-test note lands. Section 3.7 says mutation testing ran only the repetition with median code coverage; Section 3.8 explicitly disclaims statistics for RQ5 because of a single mutation-score value per tool. The selection criterion is not tied to mutation-relevant properties, and RQ4 documents substantial repetition variance in rule-execution frequencies, so the median-coverage suite could easily be atypical. EvoGURI's 0.96 vs EvoMaster-BB's 0.00 is a striking number, but it is one observation on mutants the authors themselves describe as \"relatively easy to kill.\" If this result is meant to drive the recommendation, it needs replication across several repetitions. A smaller concern is circularity: EvoGURI is built to optimize the rule-result targets used in RQ3, so its advantage there is partly by construction. The paper is transparent about this, so it is not disqualifying, but it tempers the \"EvoGURI is best\" framing. No artifact package and a proprietary subject make independent replication difficult.\n\nThe paper is for researchers in REST API test generation, search-based testing, and empirical software engineering in regulated medical domains. It is a useful, well-described case study, not a definitive tool-selection verdict. A serious editor should send it to peer review, with the expectation that reviewers press on RQ5: replicate mutation testing across repetitions and report the spread, or soften the recommendation to match the evidence.","headline":"Solid industrial case study with a real domain-specific tool, but the headline EvoGURI recommendation leans on a single unreplicated mutation-testing measurement; the paper deserves review with RQ5 pushed hard.","tokens_in":28056,"tokens_out":5239,"would_cite":true,"duration_ms":45520,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated, domain-aware test generation can effectively test a cancer registry's medical rule engine, killing 96-100% of seeded rule faults when given a strict API schema.","keywords":["automated test generation","REST API testing","medical rule engine","mutation testing","cancer registry","OpenAPI schema","search-based software testing","EvoMaster"],"falsifier":"A concrete check: collect GURI's real rule corrections from its history, roll each back into a faulty version, run the EvoGURI strict-OAS suite against them, and compare the killed fraction with the 0.96 validation-rule mutation score; a large drop would show the artificial mutants flatter the tool. Alternatively, run the same eight mutation operators on a second registry's rule engine and see whether EvoGURI's advantage over plain EvoMaster persists outside the registry.","tokens_in":27074,"feed_emoji":"🩺","tokens_out":10892,"duration_ms":90792,"temperature":0.7,"pith_summary":"The paper reports a real-world case study on GURI, the medical rule engine of a national cancer registry, which is currently tested by hand. It asks whether automated, system-level test generation for web services (REST APIs) can do the job instead. The authors compare four configurations of the EvoMaster testing tool with a new domain-aware extension, EvoGURI, across ten versions of GURI's rules. They claim that when the web service's machine-readable API description is tightened with valid value constraints, EvoGURI's test suites execute the most rules with Pass, Fail, and Warning results and kill 96% of mutated validation rules and all mutated aggregation rules, making it suitable to replace manual testing. The result matters because registry data quality depends on hundreds of frequently evolving medical rules, and automated suites could catch regressions as the rules change.","feed_headline":"Automated tests catch 96% of seeded medical-rule faults","feed_subtitle":"Domain-aware generation plus a strict API schema lets a cancer registry replace manual rule testing.","key_machinery":"The load-bearing mechanism is EvoGURI's rule-result target archive: for each generated HTTP request, the tool parses GURI's response into (rule, result) pairs and keeps the test case only if it executes a rule with a result type — Pass, Fail, Warning, Not Applied — not seen before for that rule. This converts the medical rules themselves into coverage targets, so the search is guided by whether rules are actually exercised, not just by source lines or endpoints. The supporting machinery is the strict OpenAPI schema, which adds manual value constraints so randomly sampled requests are within the variable ranges defined by the registry's internal variable repository; without it, most rules stay Not Applied and the validation-rule mutation score collapses from 0.96 to 0.","core_discovery":"The central claim is that domain-specific targets beat generic code coverage for testing GURI. EvoGURI is EvoMaster's black-box mode extended so that each (rule, result-type) pair — for example rule V01 returning Pass — is a search target; a test case is archived whenever it covers a previously unseen pair. Combined with a manually refined \"strict\" OpenAPI schema that constrains medical variables to their valid ranges, EvoGURI produces suites that, on the latest rule version, kill 0.96 of validation-rule mutants and 1.0 of aggregation-rule mutants, tied with the two white-box EvoMaster configurations (MOSA and WTS) and far ahead of plain EvoMaster black-box. On traditional metrics the five tools look alike — about 21% line coverage and a small number of real server-side 500 errors — but on the domain-specific metrics the picture separates, and the authors recommend EvoGURI for the registry because it matches the white-box tools' fault detection with a simpler, black-box setup.","pith_inferences":["Beyond the paper, the rule-result-target idea should transfer to any service whose output is a small set of named outcomes per business rule — e.g., insurance claim validators or credit-approval engines — provided the response exposes which rule fired and with what result.","The 0.96/1.0 mutation scores are bounded by the eight mutation operators used; real rule faults (e.g., an outdated medical standard or a missing rule) may be easier or harder to kill, so the scores are an estimate rather than a guaranteed detection rate.","A natural next experiment is to seed EvoGURI's archive with historical production cancer messages, since no tool currently matches production's Pass frequency and real data could close that gap while preserving the Fail/Warning corner-case coverage.","The strict OAS was handwritten from internal documentation; if the same effect can be produced by automatically mining value ranges from production messages or existing regression tests, the approach becomes cheaper to apply to other registries."],"forward_implications":["The registry can adopt EvoGURI as an automated regression-testing solution for GURI, replacing or reducing the manual testing that is current practice.","A strict API schema with valid value constraints is necessary for the benefit: with the default schema, EvoGURI and the white-box tools kill zero validation-rule mutants.","Rule evolution across the ten studied versions barely changes the tools' relative effectiveness, so the same automated setup should remain useful as medical rules are added, deleted, and modified.","Simpler black-box generation can outperform white-box search on domain-specific rule coverage, so choosing a tool for such systems should be guided by domain objectives, not code coverage alone.","The generated suites trigger more Fail and Warning results than production traffic does, meaning automated tests are suited to corner cases but are not a substitute for production-like Pass-rate validation."],"supporting_citations":[{"why":"Supplies EvoMaster, the search-based REST API test generation tool that all four off-the-shelf configurations in the study are based on.","marker":"[6]"},{"why":"Describes EvoMaster's black-box mode, the configuration that EvoGURI extends with rule-result targets.","marker":"[7]"},{"why":"Provides the comparison baseline and the code-coverage and 500-error metrics reused in RQ1 and RQ2, including the finding that EvoMaster beat nine tools.","marker":"[29]"},{"why":"The authors' previous study showing that default-OAS EvoMaster struggles to cover GURI's rules, which motivates EvoGURI and the strict OAS.","marker":"[31]"},{"why":"Defines the eight rule mutation operators used to generate the 2,279 mutated rules for the RQ5 mutation-testing experiment.","marker":"[28]"},{"why":"Supplies the mutation-testing methodology and mutation-score definition that the paper adapts from code mutants to medical-rule mutants.","marker":"[43]"},{"why":"Supports the claim that incomplete OpenAPI schemas degrade REST test generation effectiveness, motivating the strict OAS with manual constraints.","marker":"[53]"}],"fun_headline_variants":["Domain-aware testing catches 96% of cancer registry rule faults","EvoGURI turns black-box testing into white-box-level fault detection","Targeting each rule result outperforms plain code coverage for GURI","Medical rule engine testing: domain-specific search beats generic coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 2,279 artificially generated rule mutations resemble the real faults GURI could actually suffer, and that a single test suite per tool — taken from the median-coverage repetition — is enough to measure each tool's fault-finding ability.","fun_headline_variants_meta":{"raw":{"variants":["Domain-aware testing catches 96% of cancer registry rule faults","EvoGURI turns black-box testing into white-box-level fault detection","Targeting each rule result outperforms plain code coverage for GURI","Medical rule engine testing: domain-specific search beats generic coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000921,"raw_usage":{"total_tokens":3965,"prompt_tokens":978,"completion_tokens":2987,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":2914}},"tokens_in":594,"tokens_out":2987,"duration_ms":23658,"temperature":1.0,"reasoning_tokens":2914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:37:55.830235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: collect GURI's real rule corrections from its history, roll each back into a faulty version, run the EvoGURI strict-OAS suite against them, and compare the killed fraction with the 0.96 validation-rule mutation score; a large drop would show the artificial mutants flatter the tool. Alternatively, run the same eight mutation operators on a second registry's rule engine and see whether EvoGURI's advantage over plain EvoMaster persists outside the registry.","supporting_citations":[{"cited_title":"LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine","cited_arxiv_id":"2404.03664","evidence_quote":"Defines the eight rule mutation operators used to generate the 2,279 mutated rules for the RQ5 mutation-testing experiment."}],"review_version":1}