Pith. sign in

REVIEW 3 major objections 4 minor 39 references

DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By giving an LLM static access information and closing a coverage-feedback loop over a new union-coverage metric, DiffTestGen exposes behavioral differences in 78.2% of 463 pull requests and raises union coverage to 90.7%.

desk verdict Solid change-directed testing paper with a large honest evaluation; the static reachability worry is overblown as a bias but real as a generalization limit. read the letter →

arxiv 2607.16024 v1 pith:5MWHKV63 submitted 2026-07-17 cs.SE

Huimin Hu , Cristian Cadar , Michael Pradel This is my paper · ORCID
classification cs.SE
keywords differentialtestingLLM-basedtestgenerationbehavioraldifferencespullrequestunioncoveragestaticcallgraphanalysisregressiondetectionPython
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiffTestGen aims to solve a practical problem: when a developer changes code, how can tests be generated automatically to reveal what behavior actually changed? The paper's claim is that the two reasons prior automated test generators fail are that they are not directed at the change and that changed code is often hard to reach from a public API. DiffTestGen addresses both by feeding an LLM 'access information' — import lines, signatures, docstrings, and call paths from public entry points to changed private functions — and by iterating on a new union coverage metric that tells the model which changed lines remain uncovered in old and new versions. Across 463 pull requests from four Python projects, the approach exposes behavioral differences in 78.2% of PRs and raises average union coverage to 90.7%, roughly 12 to 16 percentage points above prior methods, and 99 more PRs overall. If the results hold, the approach turns an LLM test generator into a practical tool for regression detection and code review.

What carries the argument

Access information: for each changed function, DiffTestGen classifies it as public, private, or special method and constructs a prompt with the import line, signature/docstring of the class or entry function, and — for private functions — the top-5 shortest call paths from a publicly accessible entry function, gathered by backward call-graph search. Union coverage: a metric defined as (covered changed lines in the old version plus covered changed lines in the new version) divided by the total changed executable lines in both versions. The outer loop selects a reference test closest in line distance to uncovered changed lines, annotates the old/new function bodies with # COVERED and # TO_COVE

What would settle it

Construct or select a PR whose changed private function is reachable only through dynamic dispatch (e.g., getattr) or a C extension, run DiffTestGen, and check whether any generated test covers the changed lines; if none does, the call-graph access-information premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-based differential testing of code changes is gated by two solvable problems: focusing the generator on the diff, and telling it how to reach the changed code. DiffTestGen shows that providing a small amount of statically derived access information — how to import and invoke the enclosing class or an entry function that leads to a changed private function — plus a coverage feedback loop over changed lines in both versions, raises the fraction of pull requests with exposed behavioral differences to 78.2%, with an average union coverage of 90.7%. The evaluation also shows that simply generating many more tests does not recover the same gains, isolating the

Load-bearing premise

The approach assumes that a statically computed call graph plus the project's public-API documentation identifies every viable public entry point to the changed code; changes reachable only through dynamic dispatch, reflection, or undocumented internal APIs would be invisible to the LLM.

Editorial extensions

If this is right

  • Regressions can be surfaced before merge: the generated tests feed an existing LLM-based classifier, and in the 70 PRs uniquely found by DiffTestGen, 7 were classified as regressions and 5 were manually confirmed, meaning this style of testing catches bugs prior automated testers miss.
  • More tests alone is not the answer: running a prior generator with 4.45 times the test budget still found fewer differences and lower coverage, so the gain comes from directed access information and feedback, not token budget.
  • Change-directed testing can work without a specification: behavioral differences are defined purely by comparing outputs and runtime errors on old versus new versions, requiring no oracle beyond the two versions.
  • The union coverage metric gives a single number that combines changed lines in both versions, and the paper shows it correlates with exposing behavioral differences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is applying the same access-information plus coverage-feedback recipe to languages beyond Python, provided static call graphs and API documentation exist; results may degrade for dynamically dispatched languages.
  • The dependence on static reachability suggests a testable boundary: PRs where changed code is reached only via reflection, monkey-patching, or C extensions are likely the failure cases, and future work could add dynamic tracing to recover those paths.
  • The union coverage metric could become a standard evaluation measure for change-directed testing, since it captures both sides of the diff and rewards tests that reach changed code in either version.
  • The design implies a deployment pattern for CI: run DiffTestGen on each PR, classify exposed differences as intended versus regression, and block merges only on classified regressions; the paper's 5-out-of-7 manual confirmation rate suggests the classifier step needs human review.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. DiffTestGen is a change-directed, LLM-based differential testing approach for Python pull requests. Given a PR, it extracts the changed functions, classifies them as public/private/special, and computes access information via static call-graph analysis and public-API documentation. It then prompts an LLM to generate tests, repairs them through static and runtime feedback loops, and iteratively improves a newly defined union coverage metric. The evaluation covers 463 PRs from the Testora and ChaCo datasets. The paper reports that DiffTestGen exposes behavioral differences in 78.2% of PRs with 90.7% average union coverage, outperforming Testora, Testora++, and ChaCo. An ablation study attributes gains to access information and coverage feedback, and a downstream analysis with Testora's regression classifier identifies seven regression-related PRs, five of which were manually confirmed.

Significance. If the reported results hold, DiffTestGen is a practically relevant advance: it directly targets the hard problem of reaching changed private or special functions and uses an iterative coverage-feedback loop that is simple and general. The evaluation is large, uses real PRs from multiple open-source projects, includes a Testora++ control that equalizes the number of generated tests, and provides an ablation study. The authors also promise to release code and data, which will help reproducibility. The main risks are that the central effectiveness figure depends on the completeness of static reachability information, and that all headline numbers are single-run point estimates without confidence intervals. These concerns are addressable and do not invalidate the core idea, but they need to be quantified and discussed before the claimed superiority over baselines can be fully accepted.

major comments (3)
  1. [§II-C (Algorithm 1) and §IV] The central claim of 78.2% PRs with behavioral differences depends on Algorithm 1 returning a valid public entry point for every changed private/special function. The static call graph plus public-API documentation will miss entry points reachable only through dynamic dispatch, reflection/registries, or C extensions. Section IV limits external validity to 'projects with sparse documentation,' but does not quantify how often extraction fails or yields incorrect entry points within the 463 evaluated PRs. The failure analysis of the 69 PRs groups all approaches and attributes them to non-functional changes; it does not separate DiffTestGen-specific reachability failures from cases where Testora/Testora++ succeed. Please report per-PR access-information success (entry point found, generated tests reach changed lines) and re-classify failures into reachability failures vs. genuine non-behavio
  2. [§III-C (Fig. 5, Table IV) and §IV] The headline comparisons are single-run point estimates. Section IV acknowledges LLM nondeterminism and says individual variations average out over hundreds of PRs, but no confidence intervals, variance estimates, or repeated runs are reported. A 73-PR gap between DiffTestGen (350) and Testora++ (277) could be affected by sampling randomness in the LLM, and the 70 'only DiffTestGen' PRs could vary from run to run. Please provide repeated runs on a random subset with per-run variability, or bootstrap confidence intervals over PRs, and report a statistical significance test for the main comparisons.
  3. [Abstract and §III-C] The abstract reports 78.2% of PRs and 90.7% average union coverage, but the per-dataset results are 350/439 = 79.7% at 92.7% coverage on the Testora data and 28/34 = 82.4% at 76.8% coverage on the ChaCo data, with 10 PRs overlapping between the two datasets. The combined 78.2% and 90.7% figures are not derived from the reported numbers. Please clarify how the overlapping PRs are handled and make the aggregate calculation explicit and reproducible.
minor comments (4)
  1. [§III-B5] The ChaCo comparison uses previously reported ChaCo results while DiffTestGen is re-run with GPT-4o-mini. Please state whether ChaCo used the same model and test-generation budget, and discuss the risk of comparison against reported numbers.
  2. [§II-F] When no test from a prior round covers any changed line, the reference-test selection criterion appears undefined. Please specify a fallback for this case.
  3. [§II-F, Definition 3] The union coverage denominator counts changed lines in old and new versions separately. If a function is renamed or moved, the line-level numerator and denominator may not align. A brief clarification of how such cases are handled would improve precision.
  4. [§III-E, Table VI] The per-PR cost is reported as a single average without variance. Since token usage and execution time are likely skewed, reporting medians or standard deviations would be useful.

Circularity Check

1 steps flagged · score 3.0 of 10

Union coverage is both the coverage-feedback loop's optimization objective and the headline evaluation metric, making the coverage numbers partly self-referential; the central behavioral-difference claim (78.2% of PRs, 99 more PRs) is independently measured by execution and is not circular.

  1. fitted input called prediction [§II-F (Definition 3, Fig. 2 loop condition), §III-B3b, §III-C1]
    ""Loop until: Union coverage = 100% or saturates" (Fig. 2); "as introduced in §II-F, we report the overall union coverage across all PRs" (§III-B3b); "DiffTestGen achieves the highest union coverage of 92.7%" (§III-C1)"

    The exact Cov_test_union quantity of Definition 3 is simultaneously the objective the outer loop drives, with termination condition "Union coverage = 100% or saturates," and the effectiveness metric that §III-B3b averages over PRs and §III-C1 reports as evidence (92.7% on Testora data, 90.7% overall). The reported coverage is thus the optimizer's own objective value at termination, i.e., by the paper's own equations the evaluation metric equals the quantity the method was built to maximize, so the coverage claim partly reduces to its optimization target. This is contained rather than total: the primary behavioral-difference metric (Definition 2) is computed by executing tests on old/new commits and comparing outputs and errors, independent of the coverage objective, and the baselines do no

full rationale

DiffTestGen's central effectiveness claim—behavioral differences exposed in 78.2% of 463 PRs and 99 more PRs than the baselines—is not circular: Definition 2 determines a behavioral difference by executing the generated test on the pre- and post-change commits and comparing outputs/runtime errors, with flaky tests discarded by re-execution. No parameters are fitted, and the paper re-runs Testora under the same LLM (gpt-5-mini) and controls for the difference in test count via Testora++, so the gains are not artifacts of a self-citation or of generating more tests. The one genuine self-referential element is the union-coverage metric: the same formula from Definition 3 is the object the outer loop maximizes (Fig. 2: 'Loop until: Union coverage = 100% or saturates') and the headline evaluation metric averaged in §III-B3b and reported as 92.7%/90.7% in §III-C1. This makes the coverage sub-claim partly a by-construction report of the optimizer's objective value; however, it is proportional: the primary metric (Num PR) is measured independently, and the coverage comparison against Testora/Testora++/ChaCo is still meaningful because those approaches do not optimize union coverage. Other flagged concerns do not rise to circularity: the Testora [10] self-citation is not load-bearing because Testora is re-run in this evaluation and the ablation variants show incremental independent gains; the ChaCo [12] comparison rests on an externally published dataset and reported numbers; RQ4's use of Testora's LLM classifier is anchored by manual inspection of the seven flagged PRs (five confirmed regressions); and the static-reachability limitation (dynamic dispatch, C extensions, undocumented APIs) is a validity/effectiveness threat, not an equivalence between an output and an input. No uniqueness theorems are imported, no ansatz is smuggled via citation, and no known result is merely renamed.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on design hyperparameters (k=5, five fix attempts, prompt length limits, sampling) and on domain assumptions about static analysis, LLM reliability, and the operational definition of behavioral difference. No fitted physical constants or new ontological entities are introduced; the union coverage metric is a measurement definition, not an invented entity.

free parameters (4)
  • top-k shortest call paths = k=5
    Algorithm 1 line 24 selects the top-5 shortest call paths to a private function; no sensitivity analysis is provided, so the result may depend on this hand-chosen value.
  • max error-fix attempts = 5 attempts
    Sections II-E1 and II-E2 allow up to five static and runtime error-fix attempts; this budget is not varied or justified.
  • target prompt length limit = not specified
    Table II notes that access information and function bodies are included 'only if the prompt remains within the target length limit', but the exact limit is not stated, making prompt construction partially unspecified.
  • regression classification sampling = up to 5 tests per PR
    Section III-F randomly samples up to five tests per PR for LLM-based regression classification; the reported regression counts are therefore estimates and could change with a larger sample.
assumptions (4)
  • domain assumption Static call graph and public API documentation capture the relevant ways to reach changed code
    Section II-C derives access information from static call graph analysis and public API lists; this fails for dynamic dispatch, reflection, or undocumented internal entry points.
  • domain assumption Observed output/exception differences after flaky-test filtering constitute meaningful behavioral differences
    Definition 2 and the filtering procedure in Section II-A define behavioral difference operationally; environment-dependent differences that survive a single re-execution could still be non-semantic.
  • domain assumption LLM-generated tests are executable and valid given the provided context
    The whole pipeline depends on gpt-5-mini producing tests that parse and run; validity checks correct many errors but cannot guarantee coverage of semantic space.
  • domain assumption Line coverage of changed code is a suitable proxy for behavioral-difference exposure
    The outer loop stops when union coverage saturates, but a changed line can be executed without producing an observable output difference; the paper acknowledges this gap in the threats section.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences." pith.science (2026). https://pith.science/paper/5MWHKV63

@misc{pith2026260716024,
  author       = {Pith},
  title        = {Pith review of: DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MWHKV63}},
  note         = {Machine review of arXiv:2607.16024}
}
read the original abstract

As software evolves over time, it is important to ensure that any behavioral changes occur as intended by developers. A promising approach for this goal is to generate tests that expose behavioral differences between the old and new versions of a program. However, current approaches fail to trigger behavioral differences for many code changes. This paper presents~DiffTestGen, a novel change-directed, LLM-based differential testing approach specifically designed to expose behavioral differences introduced by a code change. The approach is enabled by two key contributions: First, DiffTestGen leverages static call graph analysis and project documentation to identify valid entry points for test generation and to guide the LLM toward reaching the changed code. Second, DiffTestGen iteratively improves our newly introduced union coverage metric, which combines coverage of modified code in the old and the new version, by providing targeted coverage feedback to the LLM. We evaluate DiffTestGen on two datasets comprising a total of 463 PRs. DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%. Compared with the baselines, DiffTestGen exposes 99 more PRs overall and increases code coverage by 12.5% and 15.6% percentage points, respectively. By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.

Figures

Figures reproduced from arXiv: 2607.16024 by the authors.

Figure 1
Figure 1. Motivating example. forming differential testing of code changes, we present DiffTestGen, a change-directed, LLM-based testing approach for exposing behavioral differences between two program versions. Given a PR, DiffTestGen generates and executes tests to expose behavioral differences between the original and changed program versions. Specifically, it uses static call graph analysis to identify accessible entry po… view at source ↗
Figure 2
Figure 2. Overview of DiffTestGen. prompt (2) and sends it to an LLM to generate tests. The generated tests are then refined through an iterative error￾fixing phase (3), consisting of two inner loops: one that resolves static validity errors in the generated tests (3A), and another that addresses runtime errors identified during test execution (3B). DiffTestGen also includes an outer coverage feedback loop (with connections b… view at source ↗
Figure 3
Figure 3. Access information examples for changed public functions, changed private functions, and changed special methods. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: A reference test and its coverage information. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison of approaches in terms of PRs with behavioral differences and union coverage. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 3 canonical work pages

  1. [2]

    CoverUp: Effective High Coverage Test Generation for Python,

    J. Altmayer Pizzorno and E. D. Berger, “CoverUp: Effective High Coverage Test Generation for Python,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3729398

    doi:10.1145/3729398 2025

  2. [3]

    Test Intention Guided LLM- Based Unit Test Generation,

    Z. Nan, Z. Guo, K. Liu, and X. Xia, “Test Intention Guided LLM- Based Unit Test Generation,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 1026–1038

    2025

  3. [4]

    Advancing Code Coverage: Incorporating Program Analysis with Large Language Models,

    C. Yang, J. Chen, B. Lin, Z. Wang, and J. Zhou, “Advancing Code Coverage: Incorporating Program Analysis with Large Language Models,”ACM Trans. Softw. Eng. Methodol., vol. 35, no. 5, Apr. 2026. [Online]. Available: https://doi.org/10.1145/3748505

    doi:10.1145/3748505 2026

  4. [5]

    Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM,

    G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM,”Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643769

    doi:10.1145/3643769 2024

  5. [6]

    TestWeaver: Execution-aware, Feedback-driven Regression Testing Generation with Large Language Models,

    C. C. Le, C. D. Van, T. D. Vu, T. M. P. Vu, H. N. Phan, H. N. Phan, and T. N. Nguyen, “TestWeaver: Execution-aware, Feedback-driven Regression Testing Generation with Large Language Models,” 2026. [Online]. Available: https://arxiv.org/abs/2508.01255

    arXiv 2026

  6. [7]

    An approach and benchmark to detect behavioral changes of commits in continuous integration,

    B. Danglot, M. Monperrus, W. Rudametkin, and B. Baudry, “An approach and benchmark to detect behavioral changes of commits in continuous integration,”Empirical Software Engineering, vol. 25, no. 4, pp. 2379–2415, 2020

    2020

  7. [8]

    Nuances Are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting,

    T.-O. Li, W. Zong, Y . Wang, H. Tian, Y . Wang, S.-C. Cheung, and J. Kramer, “Nuances Are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting,” inProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’23. IEEE Press, 2024, p. 14–26. [Online]. Available: https://doi.org/10.11...

    arXiv 2024

  8. [9]

    Tracejit: Evaluating the impact of behavioral code change on just-in-time defect prediction,

    I. Morita, Y . Kashiwa, M. Kondo, J. Sohn, S. McIntosh, Y . Kamei, and N. Ubayashi, “Tracejit: Evaluating the impact of behavioral code change on just-in-time defect prediction,” in2024 IEEE International Con- ference on Software Analysis, Evolution and Reengineering (SANER), 2024, pp. 580–591

    2024

Show all 39 references
  1. [10]

    Testora: Using Natural Language Intent to Detect Behavioral Regressions,

    M. Pradel, “Testora: Using Natural Language Intent to Detect Behavioral Regressions,” 2025. [Online]. Available: https://arxiv.org/ abs/2503.18597

    2025

  2. [11]

    Generalizing Test Cases for Comprehensive Test Scenario Coverage,

    B. Qi, Y . Lin, X. Weng, C. Liu, H. Sun, G. Fraser, and J. S. Dong, “Generalizing Test Cases for Comprehensive Test Scenario Coverage,” Proc. ACM Softw. Eng., vol. 3, no. FSE, Jun. 2026. [Online]. Available: https://doi.org/10.1145/3808216

    2026 doi

  3. [12]

    Change And Cover: Last-Mile, Pull Request-Based Regression Test Augmentation,

    Z. Zhou, M. Paltenghi, M. Kim, and M. Pradel, “Change And Cover: Last-Mile, Pull Request-Based Regression Test Augmentation,” 2026. [Online]. Available: https://arxiv.org/abs/2601.10942

    2026

  4. [13]

    CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 919–931

    2023

  5. [14]

    HITS: High-coverage LLM-based Unit Test Generation via Method Slicing,

    Z. Wang, K. Liu, G. Li, and Z. Jin, “HITS: High-coverage LLM-based Unit Test Generation via Method Slicing,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY , USA: Association for Computing Machinery, 20...

    2024

  6. [15]

    eXpress: guided path exploration for efficient regression test generation,

    K. Taneja, T. Xie, N. Tillmann, and J. de Halleux, “eXpress: guided path exploration for efficient regression test generation,” in Proceedings of the 2011 International Symposium on Software Testing and Analysis, ser. ISSTA ’11. New York, NY , USA: Association for Computing Ma...

    2011

  7. [16]

    A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs,

    M. Kim, T. Stennett, S. Sinha, and A. Orso, “A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs,” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press, 2025, p. 1409–1421. [Online]. A...

    2025

  8. [17]

    CrossProbe: LLM-Empowered Cross- Project Bug Detection for Deep Learning Frameworks,

    H. Guan, G. Bai, and Y . Liu, “CrossProbe: LLM-Empowered Cross- Project Bug Detection for Deep Learning Frameworks,”Proc. ACM Softw. Eng., vol. 2, no. ISSTA, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3728984

    2025 doi

  9. [18]

    LLM-powered test case generation for detecting bugs in plausible programs,

    K. Liu, Z. Chen, Y . Liu, J. M. Zhang, M. Harman, Y . Han, Y . Ma, Y . Dong, G. Li, and G. Huang, “LLM-powered test case generation for detecting bugs in plausible programs,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    2025

  10. [19]

    Otter: Generating Tests from Issues to Validate SWE Patches,

    T. Ahmed, J. Ganhotra, R. Pan, A. Shinnar, S. Sinha, and M. Hirzel, “Otter: Generating Tests from Issues to Validate SWE Patches,” 2025. [Online]. Available: https://arxiv.org/abs/2502.05368

    2025 arXiv

  11. [20]

    A history-based test prioritization technique for regression testing in resource constrained environments,

    J.-M. Kim and A. Porter, “A history-based test prioritization technique for regression testing in resource constrained environments,” inProceedings of the 24th International Conference on Software Engineering, ser. ICSE ’02. New York, NY , USA: Association for Computing Machin...

    2002

  12. [22]

    Comparing and combining analysis-based and learning-based regression test selection,

    J. Zhang, Y . Liu, M. Gligoric, O. Legunsen, and A. Shi, “Comparing and combining analysis-based and learning-based regression test selection,” inProceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test, ser. AST ’22. New York, NY , USA: Associat...

    2022

  13. [23]

    Practical regression test selection with dynamic file dependencies,

    M. Gligoric, L. Eloussi, and D. Marinov, “Practical regression test selection with dynamic file dependencies,” inProceedings of the 2015 International Symposium on Software Testing and Analysis, ser. ISSTA 2015. New York, NY , USA: Association for Computing Machinery, 2015, p....

    2015

  14. [24]

    Fine-grained test minimization,

    A. Vahabzadeh, A. Stocco, and A. Mesbah, “Fine-grained test minimization,” inProceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 210–221. [Online]. Available: https://doi.org...

    2018

  15. [25]

    Nemo: multi- criteria test-suite minimization with integer nonlinear programming,

    J.-W. Lin, R. Jabbarvand, J. Garcia, and S. Malek, “Nemo: multi- criteria test-suite minimization with integer nonlinear programming,” inProceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY , USA: Association for Computing Machi...

    2018

  16. [26]

    Understanding myths and realities of test-suite evolution,

    L. S. Pinto, S. Sinha, and A. Orso, “Understanding myths and realities of test-suite evolution,” inProceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, ser. FSE ’12. New York, NY , USA: Association for Computing Machinery, 201...

    2012

  17. [27]

    TestEvol: A tool for analyzing test-suite evolution,

    ——, “TestEvol: A tool for analyzing test-suite evolution,” in2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 1303–1306

    2013

  18. [28]

    Automated Test Case Repair Using Language Models,

    A. Saboor Yaraghi, D. Holden, N. Kahani, and L. Briand, “Automated Test Case Repair Using Language Models,”IEEE Transactions on Software Engineering, vol. 51, no. 4, pp. 1104–1133, 2025

    2025

  19. [29]

    Patterns of Code-to-Test Co-evolution for Automated Test Suite Maintenance,

    S. Shimmi and M. Rahimi, “Patterns of Code-to-Test Co-evolution for Automated Test Suite Maintenance,” in2022 IEEE Conference on Software Testing, Verification and Validation (ICST). Los Alamitos, CA, USA: IEEE Computer Society, Apr. 2022, pp. 116–

    2022

  20. [30]

    Automated Unit Test Improvement using Large Language Models at Meta,

    N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated Unit Test Improvement using Large Language Models at Meta,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Soft...

    2024

  21. [31]

    Beyond Coverage: Automatic Test Suite Augmentation for Enhanced Effectiveness using Large Language Models,

    Z. Lu, P. Zhang, Y . Nie, Y . Yang, Y . Tang, C. Y . Chong, and Y . Zhou, “Beyond Coverage: Automatic Test Suite Augmentation for Enhanced Effectiveness using Large Language Models,”Proc. ACM Program. Lang., vol. 10, no. OOPSLA1, Apr. 2026. [Online]. Available: https://doi.org...

    2026 doi

  22. [32]

    E-Test: E’er-Improving Test Suites,

    K. Qiu, L. D. Grazia, L. Mariani, and M. Pezz `e, “E-Test: E’er-Improving Test Suites,” 2025. [Online]. Available: https://arxiv.org/abs/2510.19860

    2025

  23. [33]

    Differential testing for software,

    W. M. McKeeman, “Differential testing for software,”Digital Technical Journal, vol. 10, no. 1, pp. 100–107, 1998

    1998

  24. [34]

    Finding and understanding bugs in C compilers,

    X. Yang, Y . Chen, E. Eide, and J. Regehr, “Finding and understanding bugs in C compilers,” inProceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, ser. PLDI ’11. New York, NY , USA: Association for Computing Machinery, 2011, p. 283–...

    2011

  25. [35]

    Many-core compiler fuzzing,

    C. Lidbury, A. Lascu, N. Chong, and A. F. Donaldson, “Many-core compiler fuzzing,” inProceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, ser. PLDI ’15. New York, NY , USA: Association for Computing Machinery, 2015, p. 65–76. [Onlin...

    2015

  26. [36]

    JEST: N+1-Version Differential Testing of Both JavaScript Engines and Specification,

    J. Park, S. An, D. Youn, G. Kim, and S. Ryu, “JEST: N+1-Version Differential Testing of Both JavaScript Engines and Specification,” in 2021 IEEE/ACM 43rd International Conference on Software Engineer- ing (ICSE), 2021, pp. 13–24

    2021

  27. [37]

    A Differential Testing Approach for Evaluating Abstract Syntax Tree Mapping Algo- rithms,

    Y . Fan, X. Xia, D. Lo, A. E. Hassan, Y . Wang, and S. Li, “A Differential Testing Approach for Evaluating Abstract Syntax Tree Mapping Algo- rithms,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 2021, pp. 1174–1185

    2021

  28. [38]

    Fuzzing Auto- matic Differentiation in Deep-Learning Libraries,

    C. Yang, Y . Deng, J. Yao, Y . Tu, H. Li, and L. Zhang, “Fuzzing Auto- matic Differentiation in Deep-Learning Libraries,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 1174–1186

    2023

  29. [39]

    DLFuzz: differential fuzzing testing of deep learning systems,

    J. Guo, Y . Jiang, Y . Zhao, Q. Chen, and J. Sun, “DLFuzz: differential fuzzing testing of deep learning systems,” inProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/F...

    2018

  30. [40]

    DifFuzz: Differential Fuzzing for Side-Channel Analysis,

    S. Nilizadeh, Y . Noller, and C. S. Pasareanu, “DifFuzz: Differential Fuzzing for Side-Channel Analysis,” in2019 IEEE/ACM 41st Interna- tional Conference on Software Engineering (ICSE), 2019, pp. 176–187

    2019

  31. [127]

    Available: https://doi.ieeecomputersociety.org/10.1109/ ICST53961.2022.00023

    [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ ICST53961.2022.00023

    2022

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.