{"id":"6d083bd9-fcff-4dae-bc85-08edc81a57ca","arxiv_id":"2507.23356","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An industrial evaluation pipeline that combines analytic checkers, dynamic compilation, and LLM-as-a-judge to score and debug COBOL-to-Java translations.","lead":"IBM researchers describe an automated evaluation system for checking the quality of COBOL-to-Java translations produced by the watsonx Code Assistant for Z. The system combines syntactic and semantic checkers with an LLM acting as a judge, and presents results on dashboards for engineers and managers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Class Designer mappings are the shared ground truth for analytic checkers and the LaaJ; a wrong mapping can be faithfully translated and rated high, so the claimed balanced evaluation misses design-level errors.","rationale":"The paper is internally coherent and provides a useful architecture description; the running example does illustrate how the checkers and LaaJ behave. However, the central claim uses 'precise' analytic checkers, and their precision is relative to a reference — the Class Designer mapping — that can itself be wrong. Because the same reference is fed to the LLM, the analytic checkers, and the LaaJ, there is no independent check on it. The paper's own compilation mode only catches syntactic or type-level design problems, not semantically wrong but well-typed mappings. The LaaJ validation via partial-order benchmarks also has a self-referential element because the expected orderings are defined by the same team and human-agreement numbers are not reported, but the mapping dependence is the sharper and more load-bearing concern. The reader's verdict of CONDITIONAL is appropriate: the design is plausible, but without a mutation-style test on Class Designer mappings or SME-labeled mapping correctness data, the balanced-evaluation claim remains under-supported. No change to the reader's verdict is needed.","tokens_in":12312,"tokens_out":3495,"duration_ms":37419,"concrete_test":"Take 20–50 genapp/IMS datapoints. For each, mutate the Class Designer mapping in three ways: swap a COBOL variable to a same-typed Java getter/field, delete one procedure mapping, and change a method signature while keeping types. Produce Java translations (LLM-generated or hand-written) that faithfully follow the mutated mapping and are functionally wrong. Run the full evaluation pipeline (syntactic, variable, procedure, middleware, LaaJ) and have a COBOL/Java SME label each output. If the analytic checkers report near-zero errors and the LaaJ gives scores of 5 or higher on these mutated cases, the system cannot detect design-level errors, and the balanced-evaluation claim must be scoped to translations of a correct Class Designer output.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.B.1 states that the variable mapping is defined by the Class Designer and 'provided to both the LLM and the checker as part of the prompt'; the same mapping appears in the LaaJ prompt (Figure 4). Thus all three evidence streams share one reference. If the Class Designer produces an incorrect but type-compatible mapping — for example, maps COBOL CA-CUSTOMER-NUM to the wrong same-typed Java field, or lists a wrong method signature — the LLM can translate 'correctly' relative to that mapping, the variable/procedure matching checkers will see matching accesses, and the LaaJ will compare against the same wrong reference. The paper acknowledges Class Designer bugs only in the compilation mode (Section IV.C), which detects syntactic and type errors, not semantically wrong but well-typed mappings. The running example injects faults only into the translated Java, never into the mapping itself. Therefore the central claim — that precise analytic checks plus holistic LaaJ yield a balanced and informative evaluation — is not established for a failure mode the authors themselves admit exists.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an automated quality-evaluation system for COBOL-to-Java code translation in IBM's watsonx Code Assistant for Z (WCA4Z). The system combines static syntactic checks, semantic checkers for variable accesses, procedure invocations, and middleware calls, with an LLM-as-a-judge (LaaJ) that scores translations on a seven-point scale. Results are stored in a database and presented through Grafana-based reports for project managers and developers. The paper claims that this hybrid approach yields a balanced and informative evaluation and significantly reduces the need for manual subject-matter-expert review. The only detailed evaluation presented is a single running example with three injected faults, where the analytic checkers and the LaaJ each miss some faults and the LaaJ also raises a false positive.","tokens_in":12419,"tokens_out":4795,"duration_ms":50993,"significance":"If the claimed effectiveness were substantiated with data, the system would be a valuable industrial contribution, as it tackles a real and difficult problem in LLM-based code translation. The architecture is coherent, and the combination of precise but partial analytic checkers with a holistic LaaJ is a sensible design principle. The paper also introduces a partial-order benchmark methodology for validating LaaJs, which is promising. However, the central claims — reduced SME involvement, balanced and informative evaluation, and actionable insights — are not supported by any quantitative evidence. The single running example actually illustrates that each individual checker is imperfect, which supports the \"no silver bullet\" argument but does not validate the system's overall accuracy or its claimed benefit over manual review. The paper is best read as an architecture and experience report; as a research contribution, it lacks the empirical grounding needed to establish the effectiveness claims.","major_comments":[{"comment":"The variable, procedure, and middleware mappings produced by the Class Designer are used as the shared reference by the analytic checkers, the LLM prompt, and the LaaJ. If the Class Designer generates a wrong but well-typed mapping (e.g., mapping CA-CUSTOMER-NUM to a different same-typed Java field, or listing an incorrect method signature), then the LLM can translate faithfully relative to that mapping, the variable and procedure checkers will see matching accesses, and the LaaJ will compare against the same wrong reference. The paper only identifies Class Designer bugs in the context of skeleton compilation (Section IV.C), which detects syntactic and type errors, not semantically wrong but well-typed mappings. The running example injects faults only into the translated Java code, never into the mapping, so this failure mode is not exercised. The central claim that the combination of analytic checkers and the LaaJ yields a balanced and informative evaluation is therefore not established for a failure mode the authors themselves acknowledge exists.","section":"Section IV.B and IV.C, Figure 2c"},{"comment":"The abstract and introduction state that the system has \"significantly reduced the need for manual SME involvement\" and provides \"actionable insights\", but no quantitative evidence is presented. There are no false-positive or false-negative rates for the analytic checkers, no agreement statistics between the LaaJ and human SMEs, no measurement of SME hours saved, and no longitudinal data from the platform's use in WCA4Z development. The only concrete evaluation is the anecdotal running example in Section IV.E, which shows a single translation with three injected faults. Without such measurements, the paper's central effectiveness claims are unsupported and cannot be assessed.","section":"Section I and Section V"},{"comment":"The LaaJ validation via partial-order benchmarks is described only as a framework. The paper states that \"a well-calibrated LaaJ will align strongly with these expected orderings\" and that low alignment indicates either LaaJ deficiencies or benchmark flaws, but it does not report any alignment scores, sample sizes, or calibration results. Moreover, because the expected orderings are defined within the same project, the validation is partly self-referential. To support the claim that the LaaJ is a reliable evaluation component, the authors need to report actual alignment data and, ideally, a comparison of LaaJ scores with human SME judgments.","section":"Section IV.D"},{"comment":"The introduction lists \"full compilation and execution of the translated code\" as part of the evaluation approach, but Section IV.C states that single-method execution is not performed and full-program execution is not yet efficient, and that dynamic testing has so far yielded the lowest return on investment. This is a significant mismatch between the described architecture and its operational reality. The authors should either clarify that dynamic testing is planned but not yet operational, or temper the claims about the system's current capabilities.","section":"Section IV.C"}],"minor_comments":[{"comment":"The phrase \"citeetl\" appears to be a malformed citation or placeholder and should be removed or replaced with a proper reference.","section":"Section III"},{"comment":"The word \"complimatry\" should be \"complementary\"; also, in Section IV.B.3 \"elemetns\" should be \"elements\", and in Section III \"interactes\" should be \"interacts\".","section":"Section IV.D"},{"comment":"The sentence \"This mode cannot be executed\" is confusing; it should clarify whether the skeleton class itself cannot be executed, or the compilation mode cannot be run.","section":"Section IV.C"},{"comment":"The report type \"A-V AR\" is cryptic; consider using a more descriptive name such as \"Variable-Access (Read)\".","section":"Table II"},{"comment":"The heatmap of LaaJ scores by COBOL statement is described but the color scale and the meaning of gray cells are not explained in the text; adding a legend or an explicit description would improve readability.","section":"Section V, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"This paper reads as an industrial experience report rather than a completed research contribution. The architecture and the proposed hybrid evaluation approach are of practical interest, but the lack of any quantitative validation for the central effectiveness claims is a serious gap. If the journal has an experience-report or systems track, the paper could be accepted after repositioning; otherwise, the authors should be asked to add empirical data (e.g., agreement with SMEs, detection rates, and LaaJ calibration results) before it can be considered publishable as a research paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is an IBM Research system/experience paper on evaluating COBOL-to-Java translation in their watsonx Code Assistant for Z. The punchline: the architecture is well thought out and honestly described, but the paper makes effectiveness claims without quantitative support, and the evaluation reference (Class Designer mappings) is a shared ground truth that could let a wrong but well-typed mapping pass every check.\n\nWhat's actually new: the domain-specific integration. The variable/procedure matching checkers and the middleware call alignment via Needleman-Wunsch are tailored to COBOL-to-Java and are sensible. The partial-order benchmark framework for calibrating the LLM-as-a-judge is the most novel piece, even if it is self-referential. The paper does a good job explaining why no single method (formal equivalence, compile/execute, ground truth, human eval, LaaJ) works, and the running example with injected faults nicely shows the complementary strengths and weaknesses. The architecture is detailed enough to reproduce, and the use of Tree-sitter, ADDI CFGs, Galasa, and Grafana is pragmatic.\n\nSoft spots: the central claims — reduced SME involvement, critical role in quality improvement, actionable insights — rest on anecdote, not data. No precision/recall numbers for the checkers, no agreement stats between LaaJ and human SMEs, no before/after effort metrics. That's the main gap for a reader who wants to know if the system actually works.\n\nThe bigger conceptual issue is the shared ground truth. The Class Designer produces the variable/method mappings that go to the LLM, the analytic checkers, and the LaaJ prompt. If a mapping is wrong but type-compatible, the LLM can translate 'correctly' relative to that mapping, the checkers see matching accesses, and the LaaJ compares against the same wrong reference. The paper only mentions CD bugs being caught by compilation, which wouldn't catch a semantically wrong but well-typed mapping. The running example injects faults only into the Java, never into the mapping. So the claimed 'balanced evaluation' does not cover a failure mode the authors admit exists. Worth saying though: the paper is transparent about many limitations; this is more a scope limitation than a fatal flaw.\n\nWho it's for: practitioners building evaluation pipelines for LLM-based code translation, especially in modernization contexts. A serious referee should ask for quantitative validation, but the system description itself deserves review.","headline":"Well-designed industrial evaluation pipeline for COBOL-to-Java translation, but effectiveness claims lack quantitative support and the shared Class Designer ground truth leaves a real blind spot.","tokens_in":13009,"tokens_out":2052,"would_cite":false,"duration_ms":21172,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining precise but partial analytic checkers with comprehensive but less precise LLM-as-a-judge evaluation yields a balanced and informative assessment of COBOL-to-Java translations.","keywords":["COBOL-to-Java transformation","LLM-as-a-judge","automated evaluation","code translation quality","semantic checking","mainframe modernization","continuous integration benchmarking","hallucination detection"],"falsifier":"Use a benchmark of COBOL paragraphs with seeded faults of each type the system claims to cover—variable misuse, wrong exception or control-flow handling, wrong middleware parameters—run the full pipeline, and compare detection against a panel of human COBOL/Java experts using the same seven-point scale; if any fault type is caught by neither the analytic checkers nor the judge in a substantial fraction of cases, the balanced-evaluation claim is falsified.","tokens_in":12059,"feed_emoji":"⚖️","tokens_out":12444,"duration_ms":107924,"temperature":0.7,"pith_summary":"This paper claims that the quality of COBOL-to-Java translation performed by large language models can be evaluated automatically and at scale by combining complementary checkers rather than relying on any single method. Proving equivalence between COBOL and Java programs is undecidable, so the system pairs precise but partial analytic checks—syntactic parsing, variable and procedure matching, middleware-call alignment, hallucination detection, and compilation—with a holistic LLM-as-a-judge that scores translations on a seven-point scale. The authors report that this hybrid approach yields a balanced and informative evaluation, supports continuous integration and large-scale benchmarking, and has substantially reduced the need for manual expert review during development of the translation component.","feed_headline":"Hybrid evaluator vets COBOL-to-Java translations automatically","feed_subtitle":"Precise static checks catch specific faults; an LLM judge adds holistic scoring, reducing manual review.","key_machinery":"The load-bearing mechanism is the hybrid checker suite organized around the Class Designer's variable and method mappings, which define the expected correspondence between COBOL symbols and Java variables, getters, setters, and method signatures. Syntactic checks test that the output is nonempty, non-repetitive, parsable, and contains an executable statement. Semantic checks match variable reads and writes and procedure invocations using deliberately loose matching, while middleware calls are aligned as ordered sequences with the Needleman-Wunsch algorithm under strict one-to-one correspondence, with unmatched Java calls reported as hallucinations. The LLM-as-a-judge layer adds a seven-point translation-quality scale, a prompt that requests reasoning and a hallucination count, and calibration through human expert annotations plus partial-order benchmarks in which three variants of each sample are expected to score in decreasing order.","core_discovery":"The central claim is that no single technique—formal equivalence, dynamic testing, ground-truth comparison, human review, or LLM-as-a-judge by itself—is sufficient for assessing LLM-based COBOL-to-Java translation, and the paper therefore builds a data-centric pipeline that layers static analytic checkers, dynamic compilation and execution, and LLM-as-a-judge. The analytic checkers traverse the COBOL control-flow graph and the Java parse tree to verify that variable accesses, procedure invocations, and middleware calls (CICS, IMS, SQL) survive translation, using the Class Designer's mappings as the reference and the Needleman-Wunsch algorithm to align middleware call sequences under strict one-to-one correspondence. Dynamic testing compiles and executes translated code on z/OS, though the paper reports it currently yields the lowest return on investment. The LLM-as-a-judge scores the whole translation on a seven-point scale calibrated by human expert annotations and partial-order benchmarks. The running example with three injected faults shows analytic checkers catching two of them and the LaaJ catching a different one; none of the techniques is perfect, and the combination is what provides a good overall assessment.","pith_inferences":["A testable extension the paper leaves implicit: applying the same sequence-alignment method used for middleware calls to variable and procedure access sequences would catch reorderings and count mismatches that the deliberately loose matchers currently ignore, at the risk of more false positives.","If the Class Designer mappings are treated as a checkable artifact rather than assumed ground truth, the pipeline could rank a translation only after verifying that its variable and method map is self-consistent, since every semantic checker and the judge reference that map.","The partial-order benchmark design could be reused to monitor the judge itself over time, flagging judge drift when a new model version changes the alignment between judge scores and expected orderings without any change in the translator.","The statement-level score heatmaps suggest a closed-loop benchmark-growth policy: automatically generate new datapoints for COBOL statements with low average scores and low coverage, a direction the paper names as future work but does not formalize."],"forward_implications":["The translation subsystem can be regression-tested at large scale in continuous integration, with every run stored in a database for longitudinal comparison across model versions.","Project managers and technical leads can compare overall quality across LLMs and benchmarks, then drill down from aggregate scores to a single translation's source, target, mappings, checker errors, and judge reasoning.","Coverage data tied to COBOL statements makes weak spots visible, for example CALL statements receiving a low average score while ADD statements score high, directing benchmark authors and model developers to the right targets.","The shared evaluation infrastructure can be reused for other LLM-based components of the assistant, such as code explanation and code generation.","A translation can be flagged as requiring minor versus major developer repair based on the seven-point scale even when exact equivalence cannot be proven."],"supporting_citations":[{"why":"identifies the watsonx Code Assistant for Z translation subsystem whose COBOL-to-Java output the evaluation system assesses.","marker":"[1]"},{"why":"establishes undecidability of program equivalence, ruling out formal equivalence as the sole evaluation method.","marker":"[2]"},{"why":"supplies the LLM-as-a-judge grading technique the paper adapts for translation quality assessment.","marker":"[3]"},{"why":"provides the discovery tooling used to obtain the COBOL control-flow graph for semantic matching.","marker":"[6]"},{"why":"contributes the 'no silver bullet' argument that motivates the multi-checker evaluation design.","marker":"[10]"},{"why":"furnishes the Java parser used for syntactic checks and for building Java parse trees in semantic matching.","marker":"[14]"},{"why":"supplies the Needleman-Wunsch sequence alignment algorithm used to match middleware call sequences between COBOL and Java.","marker":"[15]"},{"why":"documents LLM hallucination behavior that motivates the hallucination checks in the semantic layer.","marker":"[16]"},{"why":"provides the scheduling framework used for compilation and execution jobs in dynamic testing on z/OS.","marker":"[17]"}],"fun_headline_variants":["Hybrid checks catch COBOL-Java translation faults","LLM judge plus static checks score code migration","Auto-evaluator blends static and LLM scoring","Scoring COBOL-to-Java shifts with LLM judge","Combined checkers and LLM vet code translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's semantic layer assumes the Class Designer's variable and method mappings are correct, because the analytic checkers compare translations against those mappings and the LLM-as-a-judge judges a translation produced with the same mappings; if the mappings are wrong, every semantic check and score is measured against the wrong reference.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid checks catch COBOL-Java translation faults","LLM judge plus static checks score code migration","Auto-evaluator blends static and LLM scoring","Scoring COBOL-to-Java shifts with LLM judge","Combined checkers and LLM vet code translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1145,"prompt_tokens":890,"completion_tokens":255,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":176}},"tokens_in":506,"tokens_out":255,"duration_ms":3363,"temperature":1.0,"reasoning_tokens":176,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:48:17.077958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a benchmark of COBOL paragraphs with seeded faults of each type the system claims to cover—variable misuse, wrong exception or control-flow handling, wrong middleware parameters—run the full pipeline, and compare detection against a panel of human COBOL/Java experts using the same seven-point scale; if any fault type is caught by neither the analytic checkers nor the judge in a substantial fraction of cases, the balanced-evaluation claim is falsified.","supporting_citations":[{"cited_title":"watsonx Code Assistant for Z,","cited_arxiv_id":null,"evidence_quote":"identifies the watsonx Code Assistant for Z translation subsystem whose COBOL-to-Java output the evaluation system assesses."},{"cited_title":"Sipser, Introduction to the Theory of Computation, 3rd ed","cited_arxiv_id":null,"evidence_quote":"establishes undecidability of program equivalence, ruling out formal equivalence as the sole evaluation method."},{"cited_title":"Evaai: A multi-agent framework leveraging large language models for enhanced automated grading,","cited_arxiv_id":null,"evidence_quote":"supplies the LLM-as-a-judge grading technique the paper adapts for translation quality assessment."},{"cited_title":"IBM Application Discovery and Delivery Intelligence,","cited_arxiv_id":null,"evidence_quote":"provides the discovery tooling used to obtain the COBOL control-flow graph for semantic matching."},{"cited_title":"No silver bullet: Essence and accidents of software engineering,","cited_arxiv_id":null,"evidence_quote":"contributes the 'no silver bullet' argument that motivates the multi-checker evaluation design."},{"cited_title":"Tree-sitter: An incremental parsing system for programming tools,","cited_arxiv_id":null,"evidence_quote":"furnishes the Java parser used for syntactic checks and for building Java parse trees in semantic matching."},{"cited_title":"A general method applicable to the search for similarities in the amino acid sequence of two proteins,","cited_arxiv_id":null,"evidence_quote":"supplies the Needleman-Wunsch sequence alignment algorithm used to match middleware call sequences between COBOL and Java."},{"cited_title":"The dawn after the dark: An empirical study on factuality hallucination in large language models,","cited_arxiv_id":null,"evidence_quote":"documents LLM hallucination behavior that motivates the hallucination checks in the semantic layer."},{"cited_title":"Deep integration testing for z/OS powered hybrid cloud applications,","cited_arxiv_id":null,"evidence_quote":"provides the scheduling framework used for compilation and execution jobs in dynamic testing on z/OS."}],"review_version":1}