Pith. sign in

REVIEW 3 major objections 7 minor 56 references

Testing Medical Rules Web Services in Practice

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automated, domain-aware test generation can effectively test a cancer registry's medical rule engine, killing 96-100% of seeded rule faults when given a strict API schema.

desk verdict Solid industrial case study with a real domain-specific tool, but the headline EvoGURI recommendation leans on a single unreplicated mutation-testing measurement; the paper deserves review with RQ5 pushed hard. read the letter →

arxiv 2412.11731 v1 pith:U7JSTTD5 submitted 2024-12-16 cs.SE

Christoph Laaber , Shaukat Ali , Thomas Schwitalla , Jan F. Nyg{aa}rd This is my paper · ORCID
classification cs.SE
keywords automatedtestgenerationRESTAPItestingmedicalruleenginemutationcancerregistryOpenschemasearch-basedsoftwareEvoMaster
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a real-world case study on GURI, the medical rule engine of a national cancer registry, which is currently tested by hand. It asks whether automated, system-level test generation for web services (REST APIs) can do the job instead. The authors compare four configurations of the EvoMaster testing tool with a new domain-aware extension, EvoGURI, across ten versions of GURI's rules. They claim that when the web service's machine-readable API description is tightened with valid value constraints, EvoGURI's test suites execute the most rules with Pass, Fail, and Warning results and kill 96% of mutated validation rules and all mutated aggregation rules, making it suitable to replace manual testing. The result matters because registry data quality depends on hundreds of frequently evolving medical rules, and automated suites could catch regressions as the rules change.

What carries the argument

The load-bearing mechanism is EvoGURI's rule-result target archive: for each generated HTTP request, the tool parses GURI's response into (rule, result) pairs and keeps the test case only if it executes a rule with a result type — Pass, Fail, Warning, Not Applied — not seen before for that rule. This converts the medical rules themselves into coverage targets, so the search is guided by whether rules are actually exercised, not just by source lines or endpoints. The supporting machinery is the strict OpenAPI schema, which adds manual value constraints so randomly sampled requests are within the variable ranges defined by the registry's internal variable repository; without it, most rules stay Not Applied and the validation-rule mutation score collapses from 0.96 to 0.

What would settle it

A concrete check: collect GURI's real rule corrections from its history, roll each back into a faulty version, run the EvoGURI strict-OAS suite against them, and compare the killed fraction with the 0.96 validation-rule mutation score; a large drop would show the artificial mutants flatter the tool. Alternatively, run the same eight mutation operators on a second registry's rule engine and see whether EvoGURI's advantage over plain EvoMaster persists outside the registry.

Watch

Extended reading notes

Core claim

The central claim is that domain-specific targets beat generic code coverage for testing GURI. EvoGURI is EvoMaster's black-box mode extended so that each (rule, result-type) pair — for example rule V01 returning Pass — is a search target; a test case is archived whenever it covers a previously unseen pair. Combined with a manually refined "strict" OpenAPI schema that constrains medical variables to their valid ranges, EvoGURI produces suites that, on the latest rule version, kill 0.96 of validation-rule mutants and 1.0 of aggregation-rule mutants, tied with the two white-box EvoMaster configurations (MOSA and WTS) and far ahead of plain EvoMaster black-box. On traditional metrics the five tools look alike — about 21% line coverage and a small number of real server-side 500 errors — but on the domain-specific metrics the picture separates, and the authors recommend EvoGURI for the registry because it matches the white-box tools' fault detection with a simpler, black-box setup.

Load-bearing premise

The load-bearing premise is that the 2,279 artificially generated rule mutations resemble the real faults GURI could actually suffer, and that a single test suite per tool — taken from the median-coverage repetition — is enough to measure each tool's fault-finding ability.

Editorial extensions

If this is right

  • The registry can adopt EvoGURI as an automated regression-testing solution for GURI, replacing or reducing the manual testing that is current practice.
  • A strict API schema with valid value constraints is necessary for the benefit: with the default schema, EvoGURI and the white-box tools kill zero validation-rule mutants.
  • Rule evolution across the ten studied versions barely changes the tools' relative effectiveness, so the same automated setup should remain useful as medical rules are added, deleted, and modified.
  • Simpler black-box generation can outperform white-box search on domain-specific rule coverage, so choosing a tool for such systems should be guided by domain objectives, not code coverage alone.
  • The generated suites trigger more Fail and Warning results than production traffic does, meaning automated tests are suited to corner cases but are not a substitute for production-like Pass-rate validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the rule-result-target idea should transfer to any service whose output is a small set of named outcomes per business rule — e.g., insurance claim validators or credit-approval engines — provided the response exposes which rule fired and with what result.
  • The 0.96/1.0 mutation scores are bounded by the eight mutation operators used; real rule faults (e.g., an outdated medical standard or a missing rule) may be easier or harder to kill, so the scores are an estimate rather than a guaranteed detection rate.
  • A natural next experiment is to seed EvoGURI's archive with historical production cancer messages, since no tool currently matches production's Pass frequency and real data could close that gap while preserving the Fail/Warning corner-case coverage.
  • The strict OAS was handwritten from internal documentation; if the same effect can be produced by automatically mining value ranges from production messages or existing regression tests, the approach becomes cheaper to apply to other registries.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper reports an industrial case study on automated testing of GURI, the medical rule engine of the Cancer Registry of Norway (CRN), which validates cancer messages and aggregates cancer cases using evolving medical rules. The authors compare five REST API test-generation tools: EvoMaster in black-box mode (EvoMaster-BB), three white-box variants (MIO, MOSA, WTS), and EvoGURI, a novel extension of EvoMaster-BB that adds rule-result pairs as search targets. Across ten versions of GURI, the study measures code coverage (RQ1), code errors (RQ2), domain-specific rule-result coverage (RQ3), rule-result execution frequencies compared with production data (RQ4), and mutation testing with 2,279 rule mutants produced by eight mutation operators (RQ5). The results show similar code coverage and error counts across tools; EvoGURI and EvoMaster-BB are the strongest for rule coverage; EvoMaster-WB-MIO achieves the highest Pass/Fail execution frequencies; and in mutation testing EvoGURI, EvoMaster-WB-MOSA, and EvoMaster-WB-WTS achieve near-perfect mutation scores while EvoMaster-BB scores zero. The paper recommends EvoGURI as the automated testing solution for GURI, citing its high mutation scores and simpler setup than the white-box tools.

Significance. The study's strengths are substantial: a real-world subject with production execution data for comparison, a careful experimental protocol for RQ1-RQ4 (30 repetitions, randomized interleaved trials, Friedman tests with Nemenyi post-hoc analysis, Benjamini-Yekutieli false-discovery-rate control, and Vargha-Delaney effect sizes), and domain-specific evaluation metrics that go beyond code coverage. The comparison of generated rule-result frequencies with production GURI (RQ4) is an uncommon and valuable contribution, and the discussion of domain-specific objectives, the oracle problem, and synthetic medical data generation is useful for the broader search-based-testing community. If the results hold, the paper provides a concrete template for adapting REST API test generation to rule-based medical systems. However, the significance of the headline recommendation depends on the mutation-testing evidence (RQ5), which is the least robust part of the study: it uses a single un-replicated test suite per tool, the mutant set is unvalidated, and this is the only metric that decisively separates EvoGURI from the much simpler EvoMaster-BB.

major comments (3)
  1. [§3.7, §3.8, §4.4.4, Table 8] The recommendation to adopt EvoGURI rests on RQ5, yet RQ5 uses a single un-replicated test suite per tool: Section 3.7 states that mutation testing is executed only for version v10 and only for 'the repetition with the median code coverage (from RQ1)', and Section 3.8 explicitly declines statistical testing because there is 'only a single MS value per tool'. This RQ5 comparison is the only decisive separation between EvoGURI (M_SV=0.96) and EvoMaster-BB (M_SV=0.00); in RQ1-RQ3 the two tools are statistically equivalent (Tables 3 and 5), with only small, version-dependent differences in RQ2 and RQ4. The selection criterion is misaligned with the mutation metric: Table 3 shows near-zero variance in code coverage across repetitions (e.g., EvoGURI line coverage 20.99%±0.11), so the 'median coverage' rule does not select a suite representative of rule-execution behavior, and Section 4.4.4 (Observation 4) documents large repetition-to-repetition variance in rule-result frequencies. A different repetition could plausibly lower EvoGURI's score or raise EvoMaster-BB's score, collapsing the only quantitative separation between the two tools. I ask the authors to run the mutation analysis on several repetitions (e.g., the suites of 5-10 repetitions, possibly with mutant sampling to control cost) and report the distribution or range of mutation scores, or at minimum to verify that the EvoGURI-versus-EvoMaster-BB ordering is stable across the repetitions with the highest and lowest rule coverage.
  2. [§3.5, §5, §3.9] The mutant set is not established as representative of real rule faults, and the paper's own statements undercut its use as a decision metric in the RQ5 Summary. Section 5 concedes that 'the mutations are relatively easy to kill', and Section 3.9 acknowledges that 'it is unclear whether these domain-specific metrics are correlated with "good" test cases' for the domain experts. The eight operators in Table 2 are purely syntactic perturbations, and no evidence is given that their difficulty distribution resembles the actual rule modifications documented in the version history (Table 1). A concrete validation is available within the manuscript's own scope: the ten versions contain real rule changes, so the authors could check whether the suites generated for version i detect the actual rule modifications between version i and version i+1, or could compare the killability of artificial mutants with the difficulty of those real diffs. Without such validation, the mutation scores in Table 8 should be reported as an upper-bound sensitivity measure rather than as a validated fault-detection rate.
  3. [§3.3, §5, §8] The comparison between EvoGURI and EvoMaster-BB in RQ5 is confounded with the archive-retention policy. EvoGURI differs from EvoMaster-BB only by adding rule-result targets (Section 3.3), and the authors themselves attribute EvoMaster-BB's M_SV=0.00 to 'the low number of retained test cases in the final test suite' (Section 5). Since RQ3 (Table 5) shows the two tools covering essentially identical rule-result sets, the case for EvoGURI over the simpler EvoMaster-BB may reflect how many tests each tool's archive keeps rather than the diagnostic power of the domain-specific targets. A control experiment is needed, e.g., running EvoMaster-BB with an archive cap comparable to EvoGURI's suite size (about 43 tests), or reporting the rule-result coverage of the archived suites themselves. Without such a control, the conclusion in Section 8 that 'EvoGURI is the most effective tool' overstates what the evidence shows.
minor comments (7)
  1. [Abstract, §1] There are several missing spaces and a slip in the abstract: 'testingGURI' and 'testGURI's 10 versions' (§1) lack the space before 'GURI', and 'identify artificial faults ten versions of GURI' (Abstract) is missing 'in' before 'ten'. Please proofread the manuscript.
  2. [§1] The claim 'covering 0.50 percentage points (pp) more Pass and 0.60 pp more Fail results' is inconsistent with Table 5, where the strict-versus-default difference for EvoGURI validation rules is 26.95 pp (Pass) and 32.73 pp (Fail); the values 0.50 and 0.60 appear to be proportion differences (26.95/54.1 and 32.73/54.1), not percentage points, so 'vastly improved' is also difficult to reconcile with the stated 0.50 pp.
  3. [§2] 'Medical personell' should be 'medical personnel'.
  4. [§4.4.1] 'This is also supported by the statistical tests from Table 5' should refer to Table 6, which contains the rule-result frequency comparisons.
  5. [§3.7] 'against each of the 2,279 mutation' should be 'against each of the 2,279 mutations'.
  6. [§4.2 (RQ2 Summary)] 'EvoGURI and EvoMaster-BB are tied for the most effective tool' should read 'tied as the most effective tools'.
  7. [§3.6, §5] Please specify the kill criterion for RQ5: the definition of M_S as mutants that 'lead to a failing test' (Section 3.6) does not state how a failure is detected when a test suite is executed against a mutated rule engine (e.g., any change in rule results, or any change in HTTP status code). This matters for interpreting the 0.00 scores in Table 8.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EvoGURI's evaluation is empirical against external baselines; the self-citation for mutation operators is fully specified in the paper and not load-bearing.

full rationale

The paper's central recommendation rests on an empirical comparison against external baselines (production GURI and EvoMaster configurations), not on a parameter fitted to the outcome. EvoGURI is EvoMaster-BB augmented with rule-result archive targets; RQ3 does measure rule-result coverage, which overlaps with EvoGURI's design objective, but Table 5 shows EvoGURI and EvoMaster-BB are statistically tied on that metric, so the RQ3 result does not manufacture an EvoGURI advantage. The decisive RQ5 mutation scores are obtained by executing the generated suites against 2,279 mutants derived from the eight operators in Table 2; although the operators are attributed to the authors' prior work [28], the paper fully specifies them, so the citation is provenance rather than load-bearing evidence. The single-repetition RQ5 design (Section 3.7) and the absence of statistical tests for RQ5 are internal-validity limitations, not circular steps. No fitted parameter, no self-referential uniqueness theorem, and no renaming of a known result is present. Hence no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

No free parameters are fitted in this empirical study. The central claim rests on domain assumptions about the correctness of the rule engine, the representativeness of the mutation operators, and the strict OAS constraints; these are listed as axioms. EvoGURI is an engineered artifact, so it appears in invented_entities with independent_evidence false because no external repository or artifact is provided.

assumptions (4)
  • domain assumption GURI's current medical rules are the correct specification; Pass, Fail, and Warning outcomes from the unmutated engine are treated as ground truth.
    Used throughout RQ3 and RQ4 (Section 3.6, Section 4.3); the paper itself acknowledges the oracle problem in Section 6.2.
  • domain assumption The eight rule mutation operators in Table 2 model realistic rule faults, so killing a mutant implies detecting a real fault.
    RQ5 (Sections 3.5 and 5) uses mutation score as the fault-finding metric; the paper notes the mutants are 'relatively easy to kill', so this proxy is not validated against real faults.
  • domain assumption The manually constructed strict OAS constraints correctly encode the valid values of the 51 medical variables.
    Section 3.4 states the constraints were added from a CRN-internal variable repository; if inaccurate, the measured effectiveness differences between OAS types are distorted.
  • domain assumption The 10 selected rule versions, out of 28 change points, are representative of rule evolution at CRN.
    Section 3.2 selects the dates 'where the changes are most severe'; conclusions that rule evolution does not matter depend on this selection.
invented entities (1)
  • EvoGURI
    purpose: A CRN-specific extension of EvoMaster's black-box tester that adds rule-result pairs (e.g., V01:Pass) as search targets and retains test cases covering new pairs.
    It is a concrete software artifact described in Section 3.3, but no public repository or binary is provided, so its behavior cannot be checked outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Testing Medical Rules Web Services in Practice." pith.science (2026). https://pith.science/paper/U7JSTTD5

@misc{pith2026241211731,
  author       = {Pith},
  title        = {Pith review of: Testing Medical Rules Web Services in Practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7JSTTD5}},
  note         = {Machine review of arXiv:2412.11731}
}
read the original abstract

The Cancer Registry of Norway (CRN) collects and processes cancer-related data for patients in Norway. For this, it employs a sociotechnical software system that evolves with changing requirements and medical standards. The current practice is to manually test CRN's system to prevent faults and ensure its dependability. This paper focuses on automatically testing GURI, the CRN's medical rule engine, using a system-level testing tool, EvoMaster, in both its black-box and white-box modes, and a novel CRN-specific EvoMaster-based tool, EvoGURI. We empirically evaluate the tools' effectiveness regarding code coverage, errors found, domain-specific rule coverage, and ability to identify artificial faults ten versions of GURI. Our results show that all the tools achieve similar code coverage and identified a similar number of errors. For rule coverage, EvoGURI and EvoMaster's black-box mode produce test suites that cover the highest number of rules with Pass, Fail, and Warning results. The test suites of EvoGURI and two EvoMaster white-box tools identify the most faults in a mutation testing experiment. Based on our findings, we recommend using EvoGURI in CRN's current practice. Finally, we present key takeaways and outline open research questions for the research community.

Figures

Figures reproduced from arXiv: 2412.11731 by the authors.

Figure 1
Figure 1. Number of executed rules and results per tool and version across all the repetitions. The bars are arithmetic means, the error bars are [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Rule execution frequency by result type relative to the total number of rule executions during a test generation phase (i.e., repetition) for [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 23 canonical work pages

  1. [1]

    Conducting repeatable experiments in highly variable cloud computing environments,

    A. Abedi and T. Brecht, “Conducting repeatable experiments in highly variable cloud computing environments,” in Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, ser. ICPE 2017. New York, NY, USA: ACM, 2017, pp. 287–292. [Online]. Available: http://doi.acm.org/10.1145/3030207. 3030229

    work page doi:10.1145/3030207 2017

  2. [2]

    Generating test data from OCL constraints with search techniques,

    S. Ali, M. Z. Iqbal, A. Arcuri, and L. C. Briand, “Generating test data from OCL constraints with search techniques,” IEEE Transactions on Software Engineering, vol. 39, no. 10, pp. 1376–1402, Oct. 2013. [Online]. Available: https://doi.org/10.1109/tse.2013.17

    doi:10.1109/tse.2013.17 2013

  3. [3]

    APIFuzzer – HTTP API testing framework,

    APIFuzzer, “APIFuzzer – HTTP API testing framework,” 2022, accessed 23.8.2023. [Online]. Available: https://github.com/ KissPeter/APIFuzzer

    work page 2022

  4. [4]

    EvoMaster: Evolutionary multi-context automated system test generation,

    A. Arcuri, “EvoMaster: Evolutionary multi-context automated system test generation,” in Proceedings of the 11th IEEE International Conference on Software Testing, Verification and Validation , ser. ICST 2018. IEEE, Apr. 2018, pp. 394–397. [Online]. Available: https://doi.org/10.1109/ICST.2018.00046

    arXiv 2018

  5. [5]

    Test suite generation with the Many Independent Objective (MIO) algorithm,

    ——, “Test suite generation with the Many Independent Objective (MIO) algorithm,” Information and Software Technology , vol. 104, pp. 195–206, Dec. 2018. [Online]. Available: https: //doi.org/10.1016/j.infsof.2018.05.003

    doi:10.1016/j.infsof.2018.05.003 2018

  6. [6]

    RESTful API automated test case generation with EvoMaster,

    ——, “RESTful API automated test case generation with EvoMaster,” ACM Transactions on Software Engineering and Methodology, vol. 28, no. 1, pp. 1–37, Feb. 2019. [Online]. Available: https://doi.org/10.1145/3293455

    doi:10.1145/3293455 2019

  7. [7]

    Automated black- and white-box testing of RESTful APIs with EvoMaster,

    ——, “Automated black- and white-box testing of RESTful APIs with EvoMaster,” IEEE Software, vol. 38, no. 3, pp. 72–78, May 2021. [Online]. Available: https://doi.org/10.1109/MS.2020.3013820

    arXiv 2021

  8. [8]

    A practical guide for using statistical tests to assess randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in Proceedings of the 33rd International Conference on Software Engineering, ser. ICSE 2011. ACM, 2011, pp. 1–10. [Online]. Available: https://doi.org/10.1145/1985793.1985795

    arXiv 2011

Show all 56 references
  1. [9]

    RESTler: Stateful REST API fuzzing,

    V . Atlidakis, P . Godefroid, and M. Polishchuk, “RESTler: Stateful REST API fuzzing,” in Proceedings of the 41st IEEE/ACM International Conference on Software Engineering, ser. ICSE 2019. IEEE, May 2019. [Online]. Available: https://doi.org/10.1109/icse.2019.00083

    2019

  2. [10]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P . McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, May 2015. [Online]. Available: https://doi.org/10.1109/tse.2014.2372785

    2015

  3. [11]

    The control of the false discovery rate in multiple testing under dependency,

    Y. Benjamini and D. Yekutieli, “The control of the false discovery rate in multiple testing under dependency,” The Annals of Statistics, vol. 29, no. 4, pp. 1165–1188, Aug. 2001. [Online]. Available: https://doi.org/10.1214/aos/1013699998

    2001

  4. [12]

    On the reliability of coverage-based fuzzer benchmarking,

    M. Böhme, L. Szekeres, and J. Metzman, “On the reliability of coverage-based fuzzer benchmarking,” in Proceedings of the 44th IEEE/ACM International Conference on Software Engineering , ser. ICSE 2022. ACM, May 2022, pp. 1621–1633. [Online]. Available: https://doi.org/10.1145/...

    2022

  5. [13]

    Software bug detection: Challenges and synergies (Dagstuhl seminar 23131),

    M. Böhme, M. Christakis, R. Padhye, K. Serebryany, A. Zeller, and H. F. Eniser, “Software bug detection: Challenges and synergies (Dagstuhl seminar 23131),” Dagstuhl Reports , vol. 13, no. 3, pp. 92–105, Oct. 2023. [Online]. Available: https://drops.dagstuhl.de/ entities/docum...

    2023 doi

  6. [14]

    Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries,

    F. Bray, M. Laversanne, H. Sung, J. Ferlay, R. L. Siegel, I. Soerjomataram, and A. Jemal, “Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” CA: A Cancer Journal for Clinicians , vol. 74, no. 3, pp. 229–263...

    2022 doi

  7. [15]

    Automated black-box testing of nominal and error scenarios in RESTful APIs,

    D. Corradini, A. Zampieri, M. Pasqua, E. Viglianisi, M. Dallago, and M. Ceccato, “Automated black-box testing of nominal and error scenarios in RESTful APIs,” Software Testing, Verification and Reliability , vol. 32, no. 5, Jan. 2022. [Online]. Available: https://doi.org/10.10...

    2022 doi

  8. [16]

    Fake it till you make it: Guidelines for effective synthetic data generation,

    F. K. Dankar and M. Ibrahim, “Fake it till you make it: Guidelines for effective synthetic data generation,” Applied Sciences, vol. 11, no. 5, 2021

    2021

  9. [17]

    Dredd – HTTP API testing framework,

    Dredd, “Dredd – HTTP API testing framework,” 2021, accessed 23.8.2023. [Online]. Available: https://dredd.org

    2021

  10. [18]

    Schemathesis: Property-based testing for API schemas,

    D. Dygalo, “Schemathesis: Property-based testing for API schemas,” 2023, accessed 23.8.2023. [Online]. Available: https: //schemathesis.readthedocs.io

    2023

  11. [19]

    EvoSuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ser. ESEC/FSE 2011. ACM, 2011. [Online]. Available: http...

    2011

  12. [20]

    Whole test suite generation,

    ——, “Whole test suite generation,” IEEE Transactions on Software Engineering, vol. 39, no. 2, pp. 276–291, Feb. 2013. [Online]. Available: https://doi.org/10.1109/TSE.2012.14

    2013 doi

  13. [21]

    The use of ranks to avoid the assumption of normality implicit in the analysis of variance,

    M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,” Journal of the American Statistical Association, vol. 32, no. 200, pp. 675–701, 1937. [Online]. Available: https://www.tandfonline.com/doi/abs/10. 1080/01621459.1937.10503522

    1937

  14. [22]

    Testing RESTful APIs: A survey,

    A. Golmohammadi, M. Zhang, and A. Arcuri, “Testing RESTful APIs: A survey,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 1, pp. 1–41, Nov. 2023. [Online]. Available: https://doi.org/10.1145/3617175

    2023 doi

  15. [23]

    Generation and evaluation of synthetic patient data,

    A. Goncalves, P . Ray, B. Soper, J. Stevens, L. Coyle, and A. P . Sales, “Generation and evaluation of synthetic patient data,” BMC Med Res Methodol , vol. 20, no. 1, p. 108, 2020, goncalves, Andre Ray, Priyadip Soper, Braden Stevens, Jennifer Coyle, Linda Sales, Ana Paula eng...

    2020

  16. [24]

    How can manual testing processes be optimized? Developer survey, optimization guidelines, and case studies,

    R. Haas, D. Elsner, E. Juergens, A. Pretschner, and S. Apel, “How can manual testing processes be optimized? Developer survey, optimization guidelines, and case studies,” in Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundat...

    2021

  17. [25]

    Synthetic data generation for tabular health records: A systematic review,

    M. Hernandez, G. Epelde, A. Alberdi, R. Cilla, and D. Rankin, “Synthetic data generation for tabular health records: A systematic review,” Neurocomputing, vol. 493, pp. 28–45, 2022

    2022

  18. [26]

    Robust confidence intervals for effect sizes: A comparative study of cohen’s d and cliff’s delta under non-normality and heterogeneous variances,

    M. R. Hess and J. D. Kromrey, “Robust confidence intervals for effect sizes: A comparative study of cohen’s d and cliff’s delta under non-normality and heterogeneous variances,” Annual Meeting of the American Educational Research Association, Apr. 2004

    2004

  19. [27]

    Cost reduction on testing evolving cancer registry system,

    E. Isaku, H. Sartaj, C. Laaber, S. Ali, T. Yue, T. Schwitalla, and J. F. Nygård, “Cost reduction on testing evolving cancer registry system,” in Proceedings of the 39th IEEE International Conference on Software Maintenance and Evolution , ser. ICSME

  20. [28]

    LLMs in the heart of differential testing: A case study on a medical rule engine,

    E. Isaku, C. Laaber, H. Sartaj, S. Ali, T. Schwitalla, and J. F. Nygård, “LLMs in the heart of differential testing: A case study on a medical rule engine,” 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.03664

    2024 doi arXiv

  21. [29]

    Automated test generation for REST APIs: No time to rest yet,

    M. Kim, Q. Xin, S. Sinha, and A. Orso, “Automated test generation for REST APIs: No time to rest yet,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2022. ACM, Jul. 2022, pp. 289–301. [Online]. Available: https://do...

    2022

  22. [30]

    Tcases: A model-based test case generator,

    K. Kimbrough, Juglar, and T. Kruse, “Tcases: A model-based test case generator,” 2023, accessed 23.8.2023. [Online]. Available: https://github.com/Cornutum/tcases

    2023

  23. [32]

    Challenges of testing an evolving cancer registration support system in practice,

    ——, “Challenges of testing an evolving cancer registration support system in practice,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering: Companion Proceedings , ser. ICSE- Companion 2023. IEEE, May 2023, pp. 355–359. [Online]. Available: ht...

    2023

  24. [33]

    A black box tool for robustness testing of REST services,

    N. Laranjeiro, J. Agnelo, and J. Bernardino, “A black box tool for robustness testing of REST services,” IEEE Access , vol. 9, pp. 24 738–24 754, Feb. 2021. [Online]. Available: https://doi.org/10.1109/ACCESS.2021.3056505

    2021

  25. [34]

    EvoReFuzz – evolutionary REST fuzzer,

    N. Laranjeiro, C. F. F. Santos, and J. Agnelo, “EvoReFuzz – evolutionary REST fuzzer,” 2022, accessed 23.8.2023. [Online]. Available: https://git.dei.uc.pt/cnl/bBOXRT

    2022

  26. [35]

    Can large language models reason about medical questions?

    V . Liévin, C. E. Hother, and O. Winther, “Can large language models reason about medical questions?” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2207.08143

    2023 doi arXiv

  27. [36]

    EvoCLINICAL: Evolving cyber-cyber digital twin with active transfer learning for automated cancer registry system,

    C. Lu, Q. Xu, T. Yue, S. Ali, T. Schwitalla, and J. F. Nygård, “EvoCLINICAL: Evolving cyber-cyber digital twin with active transfer learning for automated cancer registry system,” in Proceedings of the 31th ACM Joint European Software Engineering Conference and Symposium on th...

    2023

  28. [37]

    Automated refactoring of OCL constraints with search,

    H. Lu, S. Wang, T. Yue, S. Ali, and J. F. Nygård, “Automated refactoring of OCL constraints with search,” IEEE Transactions on Software Engineering, vol. 45, no. 2, pp. 148–170, Feb. 2019. [Online]. Available: https://doi.org/10.1109/tse.2017.2774829

    2019

  29. [38]

    On the faults found in REST APIs by automated test generation,

    B. Marculescu, M. Zhang, and A. Arcuri, “On the faults found in REST APIs by automated test generation,” ACM Transactions on Software Engineering and Methodology , vol. 31, no. 3, pp. 1–43, Jul

  30. [39]

    RESTest: Automated black-box testing of RESTful web APIs,

    A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, “RESTest: Automated black-box testing of RESTful web APIs,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2021. ACM, Jul. 2021, pp. 682–685. [Online]. Available: https...

    2021

  31. [40]

    Distribution-free multiple comparisons,

    P . B. Nemenyi, “Distribution-free multiple comparisons,” Doctoral Thesis, Princeton University, 1963

    1963

  32. [41]

    FuzzFactory: Domain-specific fuzzing with waypoints,

    R. Padhye, C. Lemieux, K. Sen, L. Simon, and H. Vijayakumar, “FuzzFactory: Domain-specific fuzzing with waypoints,”Proceedings of the ACM on Programming Languages, vol. 3, no. OOPSLA, pp. 1–29, Oct. 2019. [Online]. Available: https://doi.org/10.1145/3360600

    2019 doi

  33. [42]

    Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,

    A. Panichella, F. M. Kifetew, and P . Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Transactions on Software Engineering, vol. 44, no. 2, pp. 122–158, Feb. 2018. [Online]. Available: https://doi.o...

    2018

  34. [43]

    Chapter six - mutation testing advances: An analysis and survey,

    M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Chapter six - mutation testing advances: An analysis and survey,” in Advances in Computers . Elsevier, 2019, vol. 112, pp. 275–378. [Online]. Available: https: //doi.org/10.1016/bs.adcom.2018.03.015

    2019 doi

  35. [44]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P . Payne, M. Seneviratne, P . Gamble, C. Kelly, N. Scharli, A. Chowdhery, P . Mansfield, B. Aguera y Arcas, D. Webster, G. S. Corrado, Y. Matias, K. Chou, J. Gottw...

    2022 doi arXiv

  36. [45]

    The ABC of software engineering research,

    K.-J. Stol and B. Fitzgerald, “The ABC of software engineering research,” ACM Transactions on Software Engineering and Methodology, vol. 27, no. 3, pp. 1–51, Oct. 2018. [Online]. Available: https://doi.org/10.1145/3241743

    2018 doi

  37. [46]

    A critique and improvement of the

    A. Vargha and H. D. Delaney, “A critique and improvement of the "CL" common language effect size statistics of McGraw and Wong,” Journal of Educational and Behavioral Statistics , vol. 25, no. 2, pp. 101– 132, 2000. [Online]. Available: https://doi.org/10.2307/1165329

    doi

  38. [47]

    MBF4CR: A model-based framework for supporting an automated cancer registry system,

    S. Wang, H. Lu, T. Yue, S. Ali, and J. Nygård, “MBF4CR: A model-based framework for supporting an automated cancer registry system,” in Proceedings of the 12th European Conference on Modelling Foundations and Applications, ser. ECMFA 2016. Springer International Publishing, 20...

    2016 doi

  39. [48]

    RCIA: Automated change impact analysis to facilitate a practical cancer registry system,

    S. Wang, T. Schwitalla, T. Yue, S. Ali, and J. F. Nygård, “RCIA: Automated change impact analysis to facilitate a practical cancer registry system,” in Proceedings of the 33rd 17 IEEE International Conference on Software Maintenance and Evolution, ser. ICSME 2017. IEEE, Sep. 2...

    2017 doi

  40. [49]

    A large language model for electronic health records,

    X. Yang, A. Chen, N. PourNejatian, H. C. Shin, K. E. Smith, C. Parisien, C. Compas, C. Martin, A. B. Costa, M. G. Flores, Y. Zhang, T. Magoc, C. A. Harle, G. Lipori, D. A. Mitchell, W. R. Hogan, E. A. Shenkman, J. Bian, and Y. Wu, “A large language model for electronic health ...

    2022 doi

  41. [50]

    ChatDoctor: A medical chat model fine-tuned on LLaMA model using medical domain knowledge,

    L. Yunxiang, L. Zihan, Z. Kai, D. Ruilong, and Z. You, “ChatDoctor: A medical chat model fine-tuned on LLaMA model using medical domain knowledge,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.14070

    2023 doi arXiv

  42. [51]

    Enhancing resource-based test case generation for RESTful APIs with SQL handling,

    M. Zhang and A. Arcuri, “Enhancing resource-based test case generation for RESTful APIs with SQL handling,” in Proceedings of the 13th International Symposium on Search Based Software Engineering, ser. SSBSE 2021. Springer, Oct. 2021, pp. 103–117. [Online]. Available: https://...

    2021 doi

  43. [52]

    Adaptive hypermutation for search-based system test generation: A study on REST APIs with EvoMaster,

    ——, “Adaptive hypermutation for search-based system test generation: A study on REST APIs with EvoMaster,” ACM Transactions on Software Engineering and Methodology , vol. 31, no. 1, pp. 1–52, Jan. 2022. [Online]. Available: https://doi.org/10.1145/3464940

    2022 doi

  44. [53]

    Open problems in fuzzing RESTful APIs: A comparison of tools,

    ——, “Open problems in fuzzing RESTful APIs: A comparison of tools,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 6, pp. 144:1–144:45, Nov. 2023. [Online]. Available: https://doi.org/10.1145/3597205

    2023 doi

  45. [54]

    Resource and dependency based test case generation for RESTful web services,

    M. Zhang, B. Marculescu, and A. Arcuri, “Resource and dependency based test case generation for RESTful web services,” Empirical Software Engineering , vol. 26, no. 4, Jun. 2021. [Online]. Available: https://doi.org/10.1007/s10664-020-09937-1

    2021 doi

  46. [55]

    White-box fuzzing RPC-based APIs with EvoMaster: An industrial case study,

    M. Zhang, A. Arcuri, Y. Li, Y. Liu, and K. Xue, “White-box fuzzing RPC-based APIs with EvoMaster: An industrial case study,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 5, Sep. 2023. [Online]. Available: https://doi.org/10.1145/3585009

    2023 doi

  47. [2022]

    Available: https://doi.org/10.1145/3491038

    [Online]. Available: https://doi.org/10.1145/3491038

    doi

  48. [2023]

    2023, pp

    IEEE, Oct. 2023, pp. 508–518. [Online]. Available: https://doi.org/10.1109/ICSME58846.2023.00065

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.