Pith. sign in

REVIEW 4 major objections 5 minor 36 references

An agentic loop that mutates harnesses in lockstep can push COBOL–Java migration tests past input-search plateaus while a deterministic parity gate accepts only matching runs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 12:37 UTC pith:5SDHJUAB

load-bearing objection Solid industrial method paper: recursive search-plus-harness-mutation with a dual-runtime parity oracle actually moves coverage past multi-algorithm plateaus; the open question is how much of the red coverage is real routing vs forced entry. the 4 major comments →

arxiv 2607.28271 v1 pith:5SDHJUAB submitted 2026-07-30 cs.SE cs.AI

Agentic Method for Deterministic Validation of Legacy Code Migration

Andras Ferenczi , Jordan Docherty , Mariya Bessonov , Matthew Findlay , Krishna Lingamneni This is my paper
classification cs.SE cs.AI
keywords legacy migrationCOBOL to Javaagentic testingdifferential testingsearch-based test generationparity oracleharness mutationbranch coverage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Enterprises moving COBOL to Java often lack enough real test data to cover corner paths, and ordinary input search stalls once remaining branches depend on state the harness cannot set. This paper proposes the Locksmith Loop: run mocked COBOL and generated Java off-mainframe, search inputs with several algorithms until coverage plateaus, then have an authoring agent propose parity-preserving mutations that open “Locked Paragraphs,” keep a mutation only if it adds coverage and both sides still agree, and resume search inside the newly opened region. On three programs from hundreds to thousands of lines, the loop beat multi-algorithm plateaus, reached near-complete coverage on two open-source cases and about 92% branch coverage on a production-like batch program, and accepted only test cases where Java matched COBOL on path, external effects, and final state. The practical claim is that agentic migration output can be judged continuously by a deterministic oracle rather than by hope or sparse production samples.

Core claim

The paper claims that alternating multi-algorithm Witness Search with parity-preserving harness mutations—applied symmetrically to a mocked COBOL runtime and its generated Java target, and retained only when coverage rises and a three-axis Parity Gate passes—systematically opens execution regions ordinary input search cannot reach, and that under this regime the migrated Java matched the COBOL reference on every accepted test case across three case studies up to 4,114 source lines.

What carries the argument

The Locksmith Loop: Witness Search (six input-space algorithms until plateau) identifies Locked Paragraphs; an Authoring Layer proposes Mutation Skills (e.g. dispatcher-arm or call-injection) applied to both COBOL mock and Java; the deterministic Parity Gate compares paragraphs hit, ordered stub log, and terminal state and is the sole acceptance oracle.

Load-bearing premise

Mutations that force external values or paragraph entry on both sides still count as fair tests of migration equivalence for real production control flow, not mainly artificial paths the live system would never take that way.

What would settle it

On a held-out COBOL–Java pair, run Locksmith to high mock coverage, then replay the accepted witnesses and any production-like inputs without the force-set mutations: a systematic rise in parity failures or large drop in true reachable coverage would refute the claim that the loop validates migration readiness.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Migration teams can treat behavioral parity under a shared harness as a continuous acceptance gate for agent-generated Java, not only a late integration check.
  • When multi-algorithm input search converges within a few branches, remaining gaps are treated as structural harness limits and become candidates for symmetric mutation rather than more sampling budget.
  • Engineers shift toward defining outcome goals and deterministic oracles while the loop authors reusable mutation skills and migrator fixes from parity divergences.
  • The same search-then-mutate pattern is offered for other 1:1 behavioral migrations where both sides can be instrumented and compared under identical witnesses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method’s value may concentrate on lift-and-shift cutover confidence; it deliberately preserves legacy bugs, so semantic correctness work still needs a separate specification oracle.
  • Production residual gaps behind multi-step internal switches suggest the next bottleneck is coordinated fixture synthesis across long state chains, not more generic fuzzers.
  • If mutation skills transfer poorly across related gates without human help, skill catalogs and transfer metrics become as important as raw branch counts for unattended runs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Locksmith Loop, an agentic test-synthesis method for validating 1:1 COBOL-to-Java migrations. A deterministic migrator and mock generator produce paired runtimes; six input-search algorithms (Witness Search) explore mocked inputs until a coverage plateau; an analyzer then identifies Locked Paragraphs and an Authoring Layer proposes parity-preserving harness mutations (e.g., dispatcher-arm and call-injection/force-set skills) applied symmetrically to COBOL mock and Java. Kept mutations must add coverage and pass a three-axis Parity Gate (paragraphs hit, stub log, terminal state). On three programs (430–4,114 SLOC), the loop reports gains past multi-algorithm plateaus: near-complete coverage on two open-source CardDemo programs and 91.90% branch / 95.1% paragraph coverage with 166/166 parity PASS on a production-like batch program. The authors position this as a novel deterministic-oracle pattern for validating agentic migration output.

Significance. If the method’s coverage gains genuinely stress migration-relevant control flow and the parity oracle is accepted as a compatibility criterion, this is a practically significant contribution to legacy modernization and to agentic SE: it combines test amplification, multi-algorithm search, differential testing, and closed-loop LLM authoring under a hard execution gate. Concrete strengths include off-mainframe mocked execution, explicit residual-gap analysis, continuous parity (not only post-hoc), and a large production-shaped run with 166 accepted cases and autonomous operation. The teacher–student split (agent proposes, deterministic student accepts/rejects) is a clear industrial pattern. These are real engineering results, not only a metaphor; the main open question is how far mock-harness coverage under forced entry should be read as cutover readiness.

major comments (4)
  1. [§III-D–E, §IV, §IV-A] §III-D–E and §IV (force-set / call-injection): The central claim that Locksmith validates migration parity beyond input-search plateaus rests on keeping parity-preserving mutations that force external returns or paragraph entry on both sides. That checks agreement of paragraph bodies and stub handling once entry is arranged; it is a weaker check that unmodified production routing (multi-step switch setup and coordinated external sequences, §IV-A) would reach those paths. On CBSTM03A, Mutation opened 8 paragraphs and 19 transitions Witness Search could not; on the production run, ΔP=+25 and ΔT=+47 after a Witness Search plateau. Please quantify, per program, what fraction of Mutation-attributed (red) coverage is reachable only under call-injection or pinned externals versus after mutations that only expand the controllable harness interface while still entering via natural dispatch. Witho
  2. [§II, §IV] Related work cites symbolic-execution COBOL→JUnit equivalence checking [32], [33] as the closest validation line, but §IV reports no head-to-head numbers (coverage, parity failures found, human effort, wall time) against that pipeline or against a pure Witness Search / fuzzing budget-matched baseline without Mutation. The claim of consistently improving beyond input-search plateaus needs an explicit ablation: Witness Search alone (longer budget), Witness Search+Mutation, and ideally the prior automated validation approach on at least the two public programs. Otherwise the incremental value of the agentic Mutation tier remains narrative.
  3. [§IV, §V-a] §V-a and experimental design: Evidence is three case studies, one internal and not releasable. That is acceptable for a systems paper only if threats are matched by stronger public reproducibility. For the two open-source programs, release (or artifact-describe) the mock harness interface, skill catalog entries that fired, plateau thresholds, and per-phase coverage tables so others can re-run Witness Search and verify Mutation deltas. As written, external validity and auditability of the production 91.90% figure are limited, which is load-bearing for the “scales to production-like” claim.
  4. [§III-F, §V-e] §III-F Parity Gate: Acceptance uses three axes (paragraph set, ordered stub log, terminal state). §V-e correctly notes silence on intermediate state, FP rounding, and JVM-only errors. For the migration-validation claim, please state whether any accepted Mutation-tier cases differ in control-flow path shape beyond the paragraph set (e.g., different PERFORM/GO TO trajectories that still hit the same paragraph set), and whether stub_log ordering is complete for SQL/CICS/MQ mocks used in the production-like program. If path multiset or data-layout fingerprints are available but unused, say why; if not, bound what class of migrator bugs the gate can miss.
minor comments (5)
  1. [Abstract, §IV] Abstract and §IV: CBSTM03A branch coverage is 74.0% (108/146) while paragraph/transition coverage is very high; the abstract’s “nearly complete coverage on the two open-source programs” should state branch vs paragraph/transition explicitly to avoid over-reading.
  2. [§IV, Fig. 3–5] Figure 3 vs Figure 4: Figure 3 reports T=37/38 and P=24/25; the prose sometimes mixes branch probes (146) with transitions (38). Add a single summary table (program × phase × P/T/branch × parity) early in §IV.
  3. [§III-C] §III-C: UCB1 exploration constant c and MAP-Elites grid/descriptor definition are not specified; even if “representative not prescribed,” give the values used in the reported runs.
  4. [§I-B, §IV-A] §I-B / Fig. 1: “student” vs “Authoring Layer” colors are described but the Migrator repair path (parity failure → migrator patch) could be shown as a distinct edge; several discrepancy classes in §IV-A are listed without counts.
  5. [§III-D, §VI] Minor prose: “EV ALUATE” spacing (§III-D); “necesarry” (§VI); arXiv-style future refs (e.g., COBOL-Coder 2026) should be checked for consistency with the venue bibliography style.

Circularity Check

1 steps flagged

No load-bearing circular derivation; mild tautology only in ‘accepted cases pass parity’ wording.

specific steps
  1. self definitional [Abstract; §III-E (mutation keep rule); §III-F / §IV parity reporting]
    "The generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases. ... A mutation must satisfy the following conditions to be kept: (1) it must add coverage, and (2) it must pass the Parity Gate. ... parity 38/38 PASS ... 166/166 PASS, 0 FAIL, 0 ERROR."

    For the subset labeled ‘accepted’/kept, parity success is required by the keep/accept filter, so ‘all accepted cases pass parity’ is true by construction of that label. This is only a wording-level tautology: the paper also describes detecting divergences and repairing the migrator/mock generator until agreement, and the coverage gains themselves are independently measured—not defined as the parity outcome.

full rationale

This is an empirical systems/methods paper, not a first-principles derivation. Witness Search plateaus, mutation-driven coverage deltas (e.g., +8 paragraphs / +19 transitions on CBSTM03A; ΔP=+25, ΔT=+47 on the production-like program), residual uncovered structure, and migrator repairs after parity divergences are all outcomes of running instrumented COBOL mocks and generated Java under a shared harness—not quantities fitted from the same targets they are said to predict. Kept mutations must add coverage and pass the three-axis Parity Gate by design of the loop, so reporting that retained mutations and accepted cases satisfy parity is partly definitional in wording; the non-tautological content is that the Authoring Layer could propose skills and migrator fixes such that the loop reached high measured coverage with 166/166 PASS after repair, which is falsifiable execution evidence. Related-work citations (RepairAgent, TestGen-LLM, EvoSuite, MAP-Elites, T-Fuzz, etc.) are external and not used as uniqueness theorems that force the result. Construct-validity concerns about whether force-set/call-injection paths match unmodified production reachability are real but are validity/threat issues (§V), not Eq.X=Eq.Y circularity. Score 1 only for the minor ‘all accepted cases pass the acceptance oracle’ phrasing.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 4 invented entities

The central claim rests on engineering assumptions about mocks, parity axes, plateau detection, and 1:1 migratability—not on fitted physical constants. Free parameters are methodological knobs (algorithm set, plateau rule, ranking policy). Axioms are domain assumptions about what equivalence means and what the harness may legally change. Invented entities are named constructs in the method (Locked Paragraph, skills, Parity Gate), which are operational definitions rather than new physical entities; they lack independent evidence outside this system’s runs.

free parameters (4)
  • Witness Search plateau rule (two consecutive sweeps, ±2–3 branches convergence) = two consecutive no-gain sweeps; algorithms within ±2–3 branches
    Empirically chosen stopping criterion that triggers mutation; different thresholds would change when Locked Paragraphs are declared.
  • Six-algorithm Witness Search portfolio and UCB1 exploration constant c = pairwise, 3-way, LHS, ART, MAP-Elites, UCB1
    Portfolio selected by experimentation and ‘not guaranteed optimal’; composition directly affects plateau height before mutation.
  • Locked Paragraph ranking policy (greedy by uncovered branch count vs bang×feasibility) = greedy in experiments
    Hand-chosen policy steers which gates are drilled first and thus path-dependent coverage.
  • Parity contract axes and normalization rules = three axes as in §III-F
    Which state is compared (paragraph set, stub log, terminal observables) and whitespace/normalization choices define pass/fail independently of full semantic equivalence.
axioms (5)
  • domain assumption 1:1 COBOL-to-Java migration can preserve input–output and control-flow behavior closely enough that dual execution is a meaningful oracle.
    Stated in scope (§I-D, contributions); required for Parity Gate to validate migration rather than an intentional redesign.
  • domain assumption Legacy external dependencies can be replaced by mocks (index files/PostgreSQL, RabbitMQ, etc.) without changing the branch structure under test in ways that invalidate coverage claims.
    Harness setup §I-B/III-A; production run uses harness-generated inputs only.
  • ad hoc to paper When multiple input-search algorithms plateau within a few branches, remaining gaps are structural (harness/state) rather than search-budget failures.
    Explicit empirical generalization in §IV used to justify switching to mutation.
  • domain assumption Behavioral fingerprint equality on paragraphs_hit, stub_log, and terminal_state is sufficient acceptance for ‘matched the COBOL reference’ on accepted tests.
    §III-F; threats §V-e admit silence on intermediate state, FP rounding, JVM-only errors.
  • ad hoc to paper Symmetric parity-preserving mutations (including force-set of external returns and call injection) are allowed validation instruments, not cheating, if both sides change identically and parity holds.
    Core of Mutation phase §III-D; distinguishes Locksmith from classical bug-injection mutation testing.
invented entities (4)
  • Locksmith Loop (Witness Search + Mutation recursion) no independent evidence
    purpose: Name the end-to-end agentic validation methodology.
    Operational pipeline; evidence is only the three case studies.
  • Locked Paragraph no independent evidence
    purpose: Identify AST regions unreachable by input search alone as mutation targets.
    Analyzer-defined construct tied to this harness; no external benchmark definition.
  • Mutation Skills / Skill Catalog (dispatcher-arm, call-injection) no independent evidence
    purpose: Reusable parity-preserving edits proposed by the Authoring Layer.
    Internal catalog mechanism; success measured only inside the loop.
  • Parity Gate (three-axis differential oracle) no independent evidence
    purpose: Deterministic accept/reject of tests and mutations without a ground-truth spec.
    Instance of differential testing specialized to this fingerprint; axes chosen by authors.

pith-pipeline@v1.2.0-daily-grok45 · 19957 in / 4178 out tokens · 98016 ms · 2026-07-31T12:37:19.849278+00:00 · methodology

0 comments
read the original abstract

Migration of legacy COBOL programs to Java requires extensive testing to ensure correct functionality. This effort is often complicated by the lack of test data and the difficulty of validating all corner cases. In this paper we propose a novel agentic test-synthesis method, the "Locksmith Loop," which is initiated by preparing two runtime environments: the COBOL source and the generated Java target are each instrumented with mocks and executed off-mainframe on commodity hardware, then an iterative agentic loop performs Witness Search over input mocks to penetrate program branches, followed by parity-preserving mutations. When routing boundaries are reached, an analyzer identifies a Locked Paragraph: a condition preventing deeper exploration. Across three COBOL-Java case studies, spanning two open-source programs and one internal production-like COBOL program and ranging from 430 to 4,114 source lines, Locksmith consistently improved coverage beyond input-search plateaus, reaching nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal production-like COBOL program. The generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases. Through these findings we demonstrate, to the best of our knowledge, a novel approach for validating agentic coding output using a deterministic oracle.

Figures

Figures reproduced from arXiv: 2607.28271 by Andras Ferenczi, Jordan Docherty, Krishna Lingamneni, Mariya Bessonov, Matthew Findlay.

Figure 1
Figure 1. Figure 1: Migration and harness setup. Colors denote ownership: student [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The Locksmith Loop. The Witness Search phase sweeps reachable paragraphs with six Witness Search algorithms, including UCB1 [34], until two consecutive rounds add no branches. On plateau, the code mutation phase opens one Locked Paragraph through a parity-preserving mutation proposed by the Authoring Layer and applied by the student to both the COBOL and its generated Java target. Post-Mutation Witness Sea… view at source ↗
Figure 3
Figure 3. Figure 3: Coverage progression for the Locksmith Loop on the test program. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Control-flow graph of the test program [ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Locksmith Loop coverage progression on the production-shape run, rendered analogously to Figure [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Control-flow graph of the production program. Same node and edge [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 4 linked inside Pith

  1. [1]

    RepairAgent: An autonomous, LLM-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “RepairAgent: An autonomous, LLM-based agent for program repair,” in2025 IEEE/ACM 47th Inter- national Conference on Software Engineering (ICSE). IEEE, 2025, pp. 2188–2200

    2025

  2. [2]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems 37, 2024, pp. 50 528–50 652

    2024

  3. [3]

    Automated unit test improvement using large language models at Meta,

    N. Alshahwan, J. Chheda, A. Finegenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at Meta,” inFSE 2024: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. ACM, 2024, pp. 185–196

    2024

  4. [4]

    A snowballing literature study on test amplification,

    B. Danglot, O. L. Vera-P ´erez, Z. Yu, A. Zaidman, M. Monperrus, and B. Baudry, “A snowballing literature study on test amplification,” Journal of Systems and Software, vol. 157, p. 110398, 2019

    2019

  5. [5]

    Automatic test improvement with DSpot: A study with ten mature open-source projects,

    B. Danglot, O. L. Vera-P ´erez, B. Baudry, and M. Monperrus, “Automatic test improvement with DSpot: A study with ten mature open-source projects,”Empirical Software Engineering, vol. 24, pp. 2603–2635, 2019

    2019

  6. [6]

    EvoSuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” inProceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ser. ESEC/FSE ’11. ACM, 2011, pp. 416–419

    2011

  7. [7]

    An empirical comparison of EvoSuite and DSpot for improving developer-written test suites with respect to mutation score,

    M. F. Roslan, J. M. Rojas, and P. McMinn, “An empirical comparison of EvoSuite and DSpot for improving developer-written test suites with respect to mutation score,” inSearch-Based Software Engineering, ser. Lecture Notes in Computer Science, vol. 13711. Springer, 2022, pp. 19–34

    2022

  8. [8]

    Testability transformation,

    M. Harman, L. Hu, R. M. Hierons, J. Wegener, H. Sthamer, A. Bare- sel, and M. Roper, “Testability transformation,”IEEE Transactions on Software Engineering, vol. 30, no. 1, pp. 3–16, 2004

    2004

  9. [9]

    A survey on software testability,

    V . Garousi, M. Felderer, and F. N. Kılıc ¸aslan, “A survey on software testability,”Information and Software Technology, vol. 108, pp. 35–64, 2019

    2019

  10. [10]

    The art, science, and engineering of fuzzing: A survey,

    V . J. M. Man `es, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo, “The art, science, and engineering of fuzzing: A survey,” IEEE Transactions on Software Engineering, vol. 47, no. 11, pp. 2312– 2331, 2021

    2021

  11. [11]

    Software fault interactions and implications for software testing,

    D. R. Kuhn, D. R. Wallace, and A. M. Gallo, “Software fault interactions and implications for software testing,”IEEE Transactions on Software Engineering, vol. 30, no. 6, pp. 418–421, 2004

    2004

  12. [12]

    Adaptive random testing: The ART of test case diversity,

    T. Y . Chen, F.-C. Kuo, R. G. Merkel, and T. H. Tse, “Adaptive random testing: The ART of test case diversity,”Journal of Systems and Software, vol. 83, no. 1, pp. 60–66, 2010

    2010

  13. [13]

    Test suite generation with the many independent objective (MIO) algorithm,

    A. Arcuri, “Test suite generation with the many independent objective (MIO) algorithm,”Information and Software Technology, vol. 104, pp. 195–206, 2018

    2018

  14. [14]

    Illuminating search spaces by mapping elites,

    J.-B. Mouret and J. Clune, “Illuminating search spaces by mapping elites,” 2015, arXiv:1504.04909. [Online]. Available: https://arxiv.org/ abs/1504.04909

    Pith/arXiv arXiv 2015

  15. [15]

    Monte carlo elites: Quality- diversity selection as a multi-armed bandit problem,

    K. Sfikas, A. Liapis, and G. N. Yannakakis, “Monte carlo elites: Quality- diversity selection as a multi-armed bandit problem,” inProceedings of the Genetic and Evolutionary Computation Conference, ser. GECCO ’21. ACM, 2021, pp. 180–188

    2021

  16. [16]

    An empirical study of OSS-Fuzz bugs,

    Z. Y . Ding and C. L. Goues, “An empirical study of OSS-Fuzz bugs,” in2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 131–142

    2021

  17. [17]

    A survey of symbolic execution techniques,

    R. Baldoni, E. Coppa, D. C. D’Elia, C. Demetrescu, and I. Finocchi, “A survey of symbolic execution techniques,”ACM Computing Surveys, vol. 51, no. 3, pp. 50:1–50:39, 2018

    2018

  18. [18]

    Driller: Augmenting fuzzing through selective symbolic execution,

    N. Stephens, J. Grosen, C. Salls, A. Dutcher, R. Wang, J. Corbetta, Y . Shoshitaishvili, C. Kruegel, and G. Vigna, “Driller: Augmenting fuzzing through selective symbolic execution,” inProceedings of the 23rd Annual Network and Distributed System Security Symposium, ser. NDSS 2016. Internet Society, 2016

    2016

  19. [19]

    T-Fuzz: Fuzzing by program transformation,

    H. Peng, Y . Shoshitaishvili, and M. Payer, “T-Fuzz: Fuzzing by program transformation,” in2018 IEEE Symposium on Security and Privacy (SP). IEEE, 2018, pp. 697–710

    2018

  20. [20]

    Mokav: Execution-driven differential testing with LLMs,

    K. Etemadi, B. Mohammadi, Z. Su, and M. Monperrus, “Mokav: Execution-driven differential testing with LLMs,”Journal of Systems and Software, vol. 230, p. 112571, 2025

    2025

  21. [21]

    DiffSpec: Differential testing with LLMs using natural language specifications and code artifacts,

    N. Rao, E. Gilbert, T. Ramananandro, N. Swamy, C. L. Goues, and S. Fakhoury, “DiffSpec: Differential testing with LLMs using natural language specifications and code artifacts,” 2024, arXiv:2410.04249. [Online]. Available: https://arxiv.org/abs/2410.04249

    Pith/arXiv arXiv 2024

  22. [22]

    Metamorphic testing: A review of challenges and opportunities,

    T. Y . Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys, vol. 51, no. 1, pp. 4:1–4:27, 2018

    2018

  23. [23]

    A survey of learning- based automated program repair,

    Q. Zhang, C. Fang, Y . Ma, W. Sun, and Z. Chen, “A survey of learning- based automated program repair,”ACM Transactions on Software Engi- neering and Methodology, vol. 33, no. 2, pp. 55:1–55:69, 2024

    2024

  24. [24]

    A survey of LLM-based automated program repair: Taxonomies, design paradigms, and applications,

    B. Yang, Z. Cai, F. Liu, B. Le, L. Zhang, T. F. Bissyand ´e, Y . Liu, and H. Tian, “A survey of LLM-based automated program repair: Taxonomies, design paradigms, and applications,”ArXiv, vol. abs/2506.23749, 2025. [Online]. Available: https://api.semanticscholar. org/CorpusID:280010745

    arXiv 2025

  25. [25]

    SWE-bench: Can language models resolve real- world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real- world GitHub issues?” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66

    2024

  26. [26]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2024

    2024

  27. [27]

    Mut4All: Fuzzing compilers via LLM-synthesized mutators learned from bug reports,

    B. Wang, P. Wang, C. Chen, Q. Sun, J. Shi, C. Yang, M. Deng, Y . Lin, Z. Yang, J. Chen, J. Sun, and D. Lo, “Mut4All: Fuzzing compilers via LLM-synthesized mutators learned from bug reports,” 2025, arXiv:2507.19275v2. [Online]. Available: https: //arxiv.org/abs/2507.19275

    arXiv 2025

  28. [28]

    XMainframe: A large language model for mainframe modernization,

    A. T. V . Dau, H. T. Dao, A. T. Nguyen, H. T. Tran, P. X. Nguyen, and N. D. Q. Bui, “XMainframe: A large language model for mainframe modernization,” 2024, arXiv:2408.04660. [Online]. Available: https://arxiv.org/abs/2408.04660

    Pith/arXiv arXiv 2024

  29. [29]

    COBOL-coder: Domain-adapted large language models for COBOL code generation and translation,

    A. T. V . Dau, S. H. Tan, J. Yang, N. D. Q. Bui, and A. T. Nguyen, “COBOL-coder: Domain-adapted large language models for COBOL code generation and translation,” 2026, arXiv:2604.03986. [Online]. Available: https://arxiv.org/abs/2604.03986

    Pith/arXiv arXiv 2026

  30. [30]

    Translation of low-resource COBOL to logically correct and readable Java leveraging high-resource Java refinement,

    S. Gandhi, M. Patwardhan, J. Khatri, L. Vig, and R. K. Medicherla, “Translation of low-resource COBOL to logically correct and readable Java leveraging high-resource Java refinement,” inProceedings of the 1st International Workshop on Large Language Models for Code, ser. LLM4Code ’24. ACM, 2024, pp. 46–53

    2024

  31. [31]

    Enterprise-scale COBOL-to-Java translation: LLMs augmented with program analysis,

    V . Chakaravarthy, A. R. Choudhury, D. Garg, V . Kanvar, S. Pan- dian, A. Raghuvanshi, Y . Sabharwal, and A. Singhee, “Enterprise-scale COBOL-to-Java translation: LLMs augmented with program analysis,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ser. ICSE-SEIP ’26, 2026, pp. 775–785

    2026

  32. [32]

    Automated testing of COBOL to Java transformation,

    S. Hans, A. Kumar, T. Yasue, K. Ono, S. Krishnan, D. Sondhi, F. Satoh, G. Mitchell, S. Kumar, and D. Saha, “Automated testing of COBOL to Java transformation,” inFSE 2025: Companion Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. ACM, 2025, pp. 227–237

    2025

  33. [33]

    Automated validation of COBOL to Java transformation,

    A. Kumar, D. Saha, T. Yasue, K. Ono, S. Krishnan, S. Hans, F. Satoh, G. Mitchell, and S. Kumar, “Automated validation of COBOL to Java transformation,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. ACM, 2024, pp. 2415–2418

    2024

  34. [34]

    Finite-time analysis of the multiarmed bandit problem,

    P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,”Machine Learning, vol. 47, no. 2–3, pp. 235–256, 2002

    2002

  35. [35]

    An exploratory study on using large language models for mutation testing,

    B. Wang, M. Chen, Y . Lin, M. Papadakis, and J. M. Zhang, “An exploratory study on using large language models for mutation testing,” 2024, arXiv:2406.09843. [Online]. Available: https://arxiv.org/abs/2406. 09843

    arXiv 2024

  36. [36]

    Carddemo - mainframe credit card management application,

    Amazon Web Services, “Carddemo - mainframe credit card management application,” GitHub repository, 2022. [Online]. Available: https: //github.com/aws-samples/aws-mainframe-modernization-carddemo

    2022