Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Quality Evaluation of COBOL to Java Code Transformation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that combining precise but partial analytic checkers with comprehensive but less precise LLM-as-a-judge evaluation yields a balanced and informative assessment of COBOL-to-Java translations.

desk verdict Well-designed industrial evaluation pipeline for COBOL-to-Java translation, but effectiveness claims lack quantitative support and the shared Class Designer ground truth leaves a real blind spot. read the letter →

arxiv 2507.23356 v1 pith:6VM443ZM submitted 2025-07-31 cs.SE cs.AI

Shmulik Froimovich , Raviv Gal , Wesam Ibraheem , Avi Ziv This is my paper · ORCID
classification cs.SEcs.AI
keywords COBOL-to-JavatransformationLLM-as-a-judgeautomatedevaluationcodetranslationqualitysemanticcheckingmainframemodernizationcontinuousintegrationbenchmarkinghallucinationdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the quality of COBOL-to-Java translation performed by large language models can be evaluated automatically and at scale by combining complementary checkers rather than relying on any single method. Proving equivalence between COBOL and Java programs is undecidable, so the system pairs precise but partial analytic checks—syntactic parsing, variable and procedure matching, middleware-call alignment, hallucination detection, and compilation—with a holistic LLM-as-a-judge that scores translations on a seven-point scale. The authors report that this hybrid approach yields a balanced and informative evaluation, supports continuous integration and large-scale benchmarking, and has substantially reduced the need for manual expert review during development of the translation component.

What carries the argument

The load-bearing mechanism is the hybrid checker suite organized around the Class Designer's variable and method mappings, which define the expected correspondence between COBOL symbols and Java variables, getters, setters, and method signatures. Syntactic checks test that the output is nonempty, non-repetitive, parsable, and contains an executable statement. Semantic checks match variable reads and writes and procedure invocations using deliberately loose matching, while middleware calls are aligned as ordered sequences with the Needleman-Wunsch algorithm under strict one-to-one correspondence, with unmatched Java calls reported as hallucinations. The LLM-as-a-judge layer adds a seven-point translation-quality scale, a prompt that requests reasoning and a hallucination count, and calibration through human expert annotations plus partial-order benchmarks in which three variants of each sample are expected to score in decreasing order.

What would settle it

Use a benchmark of COBOL paragraphs with seeded faults of each type the system claims to cover—variable misuse, wrong exception or control-flow handling, wrong middleware parameters—run the full pipeline, and compare detection against a panel of human COBOL/Java experts using the same seven-point scale; if any fault type is caught by neither the analytic checkers nor the judge in a substantial fraction of cases, the balanced-evaluation claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that no single technique—formal equivalence, dynamic testing, ground-truth comparison, human review, or LLM-as-a-judge by itself—is sufficient for assessing LLM-based COBOL-to-Java translation, and the paper therefore builds a data-centric pipeline that layers static analytic checkers, dynamic compilation and execution, and LLM-as-a-judge. The analytic checkers traverse the COBOL control-flow graph and the Java parse tree to verify that variable accesses, procedure invocations, and middleware calls (CICS, IMS, SQL) survive translation, using the Class Designer's mappings as the reference and the Needleman-Wunsch algorithm to align middleware call sequences under strict one-to-one correspondence. Dynamic testing compiles and executes translated code on z/OS, though the paper reports it currently yields the lowest return on investment. The LLM-as-a-judge scores the whole translation on a seven-point scale calibrated by human expert annotations and partial-order benchmarks. The running example with three injected faults shows analytic checkers catching two of them and the LaaJ catching a different one; none of the techniques is perfect, and the combination is what provides a good overall assessment.

Load-bearing premise

The evaluation's semantic layer assumes the Class Designer's variable and method mappings are correct, because the analytic checkers compare translations against those mappings and the LLM-as-a-judge judges a translation produced with the same mappings; if the mappings are wrong, every semantic check and score is measured against the wrong reference.

Editorial extensions

If this is right

  • The translation subsystem can be regression-tested at large scale in continuous integration, with every run stored in a database for longitudinal comparison across model versions.
  • Project managers and technical leads can compare overall quality across LLMs and benchmarks, then drill down from aggregate scores to a single translation's source, target, mappings, checker errors, and judge reasoning.
  • Coverage data tied to COBOL statements makes weak spots visible, for example CALL statements receiving a low average score while ADD statements score high, directing benchmark authors and model developers to the right targets.
  • The shared evaluation infrastructure can be reused for other LLM-based components of the assistant, such as code explanation and code generation.
  • A translation can be flagged as requiring minor versus major developer repair based on the seven-point scale even when exact equivalence cannot be proven.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: applying the same sequence-alignment method used for middleware calls to variable and procedure access sequences would catch reorderings and count mismatches that the deliberately loose matchers currently ignore, at the risk of more false positives.
  • If the Class Designer mappings are treated as a checkable artifact rather than assumed ground truth, the pipeline could rank a translation only after verifying that its variable and method map is self-consistent, since every semantic checker and the judge reference that map.
  • The partial-order benchmark design could be reused to monitor the judge itself over time, flagging judge drift when a new model version changes the alignment between judge scores and expected orderings without any change in the translator.
  • The statement-level score heatmaps suggest a closed-loop benchmark-growth policy: automatically generate new datapoints for COBOL statements with low average scores and low coverage, a direction the paper names as future work but does not formalize.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes an automated quality-evaluation system for COBOL-to-Java code translation in IBM's watsonx Code Assistant for Z (WCA4Z). The system combines static syntactic checks, semantic checkers for variable accesses, procedure invocations, and middleware calls, with an LLM-as-a-judge (LaaJ) that scores translations on a seven-point scale. Results are stored in a database and presented through Grafana-based reports for project managers and developers. The paper claims that this hybrid approach yields a balanced and informative evaluation and significantly reduces the need for manual subject-matter-expert review. The only detailed evaluation presented is a single running example with three injected faults, where the analytic checkers and the LaaJ each miss some faults and the LaaJ also raises a false positive.

Significance. If the claimed effectiveness were substantiated with data, the system would be a valuable industrial contribution, as it tackles a real and difficult problem in LLM-based code translation. The architecture is coherent, and the combination of precise but partial analytic checkers with a holistic LaaJ is a sensible design principle. The paper also introduces a partial-order benchmark methodology for validating LaaJs, which is promising. However, the central claims — reduced SME involvement, balanced and informative evaluation, and actionable insights — are not supported by any quantitative evidence. The single running example actually illustrates that each individual checker is imperfect, which supports the "no silver bullet" argument but does not validate the system's overall accuracy or its claimed benefit over manual review. The paper is best read as an architecture and experience report; as a research contribution, it lacks the empirical grounding needed to establish the effectiveness claims.

major comments (4)
  1. [Section IV.B and IV.C, Figure 2c] The variable, procedure, and middleware mappings produced by the Class Designer are used as the shared reference by the analytic checkers, the LLM prompt, and the LaaJ. If the Class Designer generates a wrong but well-typed mapping (e.g., mapping CA-CUSTOMER-NUM to a different same-typed Java field, or listing an incorrect method signature), then the LLM can translate faithfully relative to that mapping, the variable and procedure checkers will see matching accesses, and the LaaJ will compare against the same wrong reference. The paper only identifies Class Designer bugs in the context of skeleton compilation (Section IV.C), which detects syntactic and type errors, not semantically wrong but well-typed mappings. The running example injects faults only into the translated Java code, never into the mapping, so this failure mode is not exercised. The central claim that the combination of analytic checkers and the LaaJ yields a balanced and informative evaluation is therefore not established for a failure mode the authors themselves acknowledge exists.
  2. [Section I and Section V] The abstract and introduction state that the system has "significantly reduced the need for manual SME involvement" and provides "actionable insights", but no quantitative evidence is presented. There are no false-positive or false-negative rates for the analytic checkers, no agreement statistics between the LaaJ and human SMEs, no measurement of SME hours saved, and no longitudinal data from the platform's use in WCA4Z development. The only concrete evaluation is the anecdotal running example in Section IV.E, which shows a single translation with three injected faults. Without such measurements, the paper's central effectiveness claims are unsupported and cannot be assessed.
  3. [Section IV.D] The LaaJ validation via partial-order benchmarks is described only as a framework. The paper states that "a well-calibrated LaaJ will align strongly with these expected orderings" and that low alignment indicates either LaaJ deficiencies or benchmark flaws, but it does not report any alignment scores, sample sizes, or calibration results. Moreover, because the expected orderings are defined within the same project, the validation is partly self-referential. To support the claim that the LaaJ is a reliable evaluation component, the authors need to report actual alignment data and, ideally, a comparison of LaaJ scores with human SME judgments.
  4. [Section IV.C] The introduction lists "full compilation and execution of the translated code" as part of the evaluation approach, but Section IV.C states that single-method execution is not performed and full-program execution is not yet efficient, and that dynamic testing has so far yielded the lowest return on investment. This is a significant mismatch between the described architecture and its operational reality. The authors should either clarify that dynamic testing is planned but not yet operational, or temper the claims about the system's current capabilities.
minor comments (5)
  1. [Section III] The phrase "citeetl" appears to be a malformed citation or placeholder and should be removed or replaced with a proper reference.
  2. [Section IV.D] The word "complimatry" should be "complementary"; also, in Section IV.B.3 "elemetns" should be "elements", and in Section III "interactes" should be "interacts".
  3. [Section IV.C] The sentence "This mode cannot be executed" is confusing; it should clarify whether the skeleton class itself cannot be executed, or the compilation mode cannot be run.
  4. [Table II] The report type "A-V AR" is cryptic; consider using a more descriptive name such as "Variable-Access (Read)".
  5. [Section V, Figure 6] The heatmap of LaaJ scores by COBOL statement is described but the color scale and the meaning of gray cells are not explained in the text; adding a legend or an explicit description would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the evaluation system is a self-contained system description, and the shared Class Designer reference is an acknowledged assumption rather than a fitted input or predicted result.

full rationale

The paper does not derive a mathematical or statistical result from fitted parameters; it describes an evaluation pipeline and illustrates its checkers on a running example. The semantic checkers use the Class Designer's variable and procedure mappings as ground truth and provide those same mappings to the LLM, the checkers, and the LaaJ prompt (Section IV.B.1 and Figure 4). This shared reference is a genuine limitation: a mapping-compliant but semantically wrong translation could pass all evidence streams, and the paper itself notes that Class Designer problems were detected only by the skeleton compilation mode (Section IV.C). That is an acknowledged scope assumption, not a circular step, because the checkers do not claim to validate the Class Designer mapping and the central claim is only that combining accurate partial checkers with holistic LaaJ evaluation yields a balanced assessment of translations relative to the supplied design. The LaaJ prompt is calibrated against external human SME scores and partial-order benchmarks; although low alignment can be attributed to benchmark flaws, that is a validation weakness rather than a derivation whose conclusion is equivalent to its input. No fitted parameter is relabeled as a prediction, no self-citation carries the argument, and no uniqueness theorem is imported from the authors' prior work. Consequently, no circular step meets the evidence threshold, and the paper should receive a score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities. The paper relies on standard math (undecidability) and on domain assumptions about benchmark representativeness, mapping correctness, and human rater reliability. These assumptions are load-bearing for the evaluation's validity but are not empirically checked in the paper.

assumptions (4)
  • standard math Program equivalence between COBOL and Java is undecidable, so formal equivalence checking is impractical.
    Invoked in Section IV to justify relying on checkers rather than proofs; standard result cited to Sipser.
  • domain assumption Static tests with coverage of input COBOL features are sufficient to compare translation quality across versions.
    Section III states evaluation is based on static tests; benchmark creation is out of scope, so conclusions depend on benchmarks being representative.
  • domain assumption The Class Designer's variable and method mappings are correct and complete.
    Section IV-B uses these mappings as the reference for variable, procedure, and middleware matching; an error in mappings would invalidate checker results and potentially mislead the LaaJ.
  • domain assumption Human SME scores provide a reliable ground truth for LaaJ calibration.
    Section IV-D aligns the LaaJ to human annotations, but no inter-rater reliability or sample size is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quality Evaluation of COBOL to Java Code Transformation." pith.science (2026). https://pith.science/paper/6VM443ZM

@misc{pith2026250723356,
  author       = {Pith},
  title        = {Pith review of: Quality Evaluation of COBOL to Java Code Transformation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VM443ZM}},
  note         = {Machine review of arXiv:2507.23356}
}
read the original abstract

We present an automated evaluation system for assessing COBOL-to-Java code translation within IBM's watsonx Code Assistant for Z (WCA4Z). The system addresses key challenges in evaluating LLM-based translators, including model opacity and the complexity of translation quality assessment. Our approach combines analytic checkers with LLM-as-a-judge (LaaJ) techniques to deliver scalable, multi-faceted evaluations. The system supports continuous integration workflows, enables large-scale benchmarking, and reduces reliance on manual review. We describe the system architecture, evaluation strategies, and reporting mechanisms that provide actionable insights for developers and project managers, facilitating the evolution of high-quality, modernized codebases.

Figures

Figures reproduced from arXiv: 2507.23356 by the authors.

Figure 1
Figure 1. Architecture diagram of IBM watsonx Code Assistant [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. COBOL to Java translation example Coverage Data: Defines the coverage model and maps data￾points to the coverage events they address, using a many￾to-many relationship [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. illustrates the architecture and flow of the eval￾uation pipeline. The pipeline implements an ETL (Extract Transform Load) citeetl. The processing begins with a file listener that monitors the shared Git repository. When a new .jsonl file (representing an evaluation set) is detected, the listener retrieves and validates it. If valid, the evaluation points are extracted. If the file introduces a new dataset, the data… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Prompt template for COBOL-to-Java translation LaaJ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: High-level comparison between ptv23 and wca4z23 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Heatmap for COBOL statements knowledge that is not inherently available in out-of-the￾box language models. This includes addressing limitations in understanding specialized COBOL and Java constructs, legacy system behaviors, and industry-specific conventions. By integr…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging

    cs.SE 2026-07 conditional novelty 6.0 of 10

    SEDCoT combines LLM translation, symbolic-execution test generation, and delta-debugging repair to raise COBOL-to-C correctness by ≥12% over SOTA LLM baselines while preserving human-readable output.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    watsonx Code Assistant for Z,

    “watsonx Code Assistant for Z,” https://www.ibm.com/products/ watsonx-code-assistant-z, 2025, accessed: 2025-07-30

    work page 2025

  2. [2]

    Sipser, Introduction to the Theory of Computation, 3rd ed

    M. Sipser, Introduction to the Theory of Computation, 3rd ed. Cengage Learning, 2012

    work page 2012

  3. [3]

    Evaai: A multi-agent framework leveraging large language models for enhanced automated grading,

    P. Lagakis and S. Demetriadis, “Evaai: A multi-agent framework leveraging large language models for enhanced automated grading,” in Generative Intelligence and Intelligent Tutoring Systems , ser. Lecture Notes in Computer Science. Springer, 2024, vol. 14798, pp. 378–385. [Online]. Available: https://link.springer.com/chapter/10.1007/ 978-3-031-63028-6 32

    work page 2024

  4. [4]

    Stern, R

    N. Stern, R. A. Stern, and J. P. Ley, COBOL for the 21st Century . Wiley, 2013, comprehensive guide to modern COBOL programming

    work page 2013

  5. [5]

    Special Report: COBOL Survey Results Prove Perva- siveness, Value and a Bright Future,

    R. Harbeck, “Special Report: COBOL Survey Results Prove Perva- siveness, Value and a Bright Future,” https://techchannel.com/cobol/ special-report-cobol-survey-results-prove-pervasiveness-value-and-a-bright-future/, accessed: 2025-07-30

    work page 2025

  6. [6]

    IBM Application Discovery and Delivery Intelligence,

    “IBM Application Discovery and Delivery Intelligence,” https:// www.ibm.com/products/app-discovery-and-delivery-intelligence, 2025, accessed: 2025-07-30

    work page 2025

  7. [7]

    COBOL to JOBOL? A Poor Choice for Modernization,

    S. Amatam, “COBOL to JOBOL? A Poor Choice for Modernization,” https://dzone.com/articles/ cobol-to-jobol-a-poor-choice-for-modernization, 2024, accessed: 2025-07-30

    work page 2024

  8. [8]

    The general insurance application,

    “The general insurance application,” https://www.ibm.com/docs/ en/cics-ts/6.x?topic=samples-general-insurance-application, 2025, accessed: 2025-07-30

    work page 2025

Show all 18 references
  1. [9]

    IBM CICS Transaction Server for z/OS Documentation,

    IBM Corporation, “IBM CICS Transaction Server for z/OS Documentation,” https://www.ibm.com/docs/en/cics-ts/6.x?topic= available-documentation-in-pdf, 2023, accessed: 2025-07-30

    2023

  2. [10]

    No silver bullet: Essence and accidents of software engineering,

    F. P. Brooks, “No silver bullet: Essence and accidents of software engineering,” IEEE Computer, vol. 20, no. 4, pp. 10–19, 1987

    1987

  3. [11]

    Klee symbolic execution engine in 2019,

    C. Cadar and M. Nowack, “Klee symbolic execution engine in 2019,” International Journal on Software Tools for Technology Transfer, vol. 23, pp. 867–870, 2021

    2019

  4. [12]

    Cbmc – c bounded model checker,

    E. Clarke, D. Kroening, and F. Lerda, “Cbmc – c bounded model checker,” in Tools for Practical Software Verification, ser. Lecture Notes in Computer Science. Springer, 2014, vol. 7682, pp. 1–17

    2014

  5. [13]

    Forg ´acs and A

    I. Forg ´acs and A. Kov ´acs, Modern Software Testing Techniques: A Practical Guide for Developers and Testers . Springer, 2024. [Online]. Available: https://link.springer.com/book/10.1007/978-1-4842-9893-0

    2024 doi

  6. [14]

    Tree-sitter: An incremental parsing system for programming tools,

    M. Brunsfeld and contributors, “Tree-sitter: An incremental parsing system for programming tools,” https://github.com/tree-sitter/tree-sitter, 2018, accessed: 2025-07-30

    2018

  7. [15]

    A general method applicable to the search for similarities in the amino acid sequence of two proteins,

    S. B. Needleman and C. D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequence of two proteins,” Journal of Molecular Biology , vol. 48, no. 3, pp. 443–453, 1970

    1970

  8. [16]

    The dawn after the dark: An empirical study on factuality hallucination in large language models,

    J. Li, J. Chen, R. Ren, X. Cheng, X. Zhao, J.-Y . Nie, and J.-R. Wen, “The dawn after the dark: An empirical study on factuality hallucination in large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2024

  9. [17]

    Deep integration testing for z/OS powered hybrid cloud applications,

    “Deep integration testing for z/OS powered hybrid cloud applications,” https://galasa.dev/, 2025, accessed: 2025-07-30

    2025

  10. [18]

    Grafana: The open and composable observability platform,

    “Grafana: The open and composable observability platform,” https: //grafana.com/, 2025, accessed: 2025-07-30

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.