Pith. sign in

REVIEW 4 major objections 5 minor 13 references

Application Modernization with LLMs: Addressing Core Challenges in Reliability, Security, and Quality

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a requirements-first, human-checked workflow, built on LLM code reasoning and code generation with progressive prompting, can make modernizing legacy Java and .NET applications reliable, secure, and affordable.

desk verdict Honest practitioner framework for LLM modernization, but the central validation gap makes the reliability claim unproven. read the letter →

arxiv 2506.10984 v1 pith:Y2M4MVEX submitted 2025-02-13 cs.SE cs.AI

Ahilan Ayyachamy Nadar Ponnusamy This is my paper · ORCID
classification cs.SEcs.AI
keywords applicationmodernizationlargelanguagemodelsprogressivepromptingcodereasoninggenerationreverseverificationhuman-in-the-looplegacysystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reliability, security, and quality problems in LLM-assisted modernization are best solved by process structure, not by a stronger model. Its framework first uses an LLM to extract functional requirements from three layers of a legacy Java or .NET application, then rebuilds the application in reverse layer order, with fixed prompts and human approval at each step. Quality is checked incrementally by humans, by feeding generated code back into the LLM to see whether it regenerates the original requirements, and optionally by comparing outputs from a second verification LLM. If this works as claimed, enterprise teams could modernize legacy systems with open, locally deployable models, without fine-tuning, prompt-engineering skill, or sending code to cloud services.

What carries the argument

The load-bearing mechanism is progressive prompting: a sequence of small generation steps in which each prompt consumes the previous artifact, such as requirements yielding a data model, the data model yielding ORM objects, those yielding APIs, and those yielding tests and user interfaces. Requirements extraction goes through the interaction layer, business-logic layer, and data layer in that order, while code generation runs the same layers in reverse so each layer builds on a stable foundation. Quality is enforced by a human-centered approach—step-by-step verification, detailed explanations, and manual integration—augmented by reverse generation verification, where the generated artifact is re-entered into the LLM to regenerate its intended input and the two are compared, and by an optional second verification LLM that replicates the generation for comparison.

What would settle it

A concrete test: run the framework on several legacy Java/.NET modules, plant known defects in the generated outputs, and measure how often reverse generation verification or the secondary verification LLM flags them while text-similarity scores remain high. If the checks miss planted defects, the central reliability claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that an opinionated pipeline, rather than a better model alone, solves the reliability, security, and quality problems of LLM-based application modernization. The pipeline extracts functional requirements from the legacy code's interaction, business-logic, and data layers, then rebuilds the target application in reverse layer order, starting with data, continuing with business logic, and finishing with the interaction layer. Fixed, embedded prompts drive each incremental generation step, and humans review every artifact before it feeds the next step. Quality assurance relies on human-centered verification plus reverse generation verification, in which generated code is fed back into the LLM to see whether the original requirements reappear. The paper demonstrates the workflow on a Java sample application, adding a new feature during modernization, and reports that the approach is practical with open models on modest local hardware.

Load-bearing premise

The quality story leans on reverse generation verification, the assumption that asking an LLM to regenerate requirements from generated code and comparing the two sets is a reliable check of correctness; the paper cites this check to the author's earlier work and does not independently validate it here.

Editorial extensions

If this is right

  • Open, locally deployed code models can handle the entire modernization workflow, so organizations can avoid fine-tuning and cloud dependency while keeping code in air-gapped environments.
  • Fixed embedded prompts reduce user-dependent variability, making generated artifacts more consistent and maintainable than outputs from general AI assistant chat.
  • Because requirements are extracted from the actual code and reviewed before development, teams can fix latent bugs and add features during modernization rather than only translating code.
  • Reverse generation verification is the more economical quality check compared with running a second verification LLM, though automating it consistently remains difficult.
  • The generated requirements may not match standard requirements templates, but they can be converted manually or automatically, so the workflow can still feed existing enterprise processes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the framework generalizes beyond the single case study, the most valuable next test is a controlled comparison: same model and target framework with and without the requirements-first pipeline, measuring defect density, security findings, and review time.
  • The paper's observations about output variation suggest that reverse generation verification could become an automated CI gate only after calibrating a reliable similarity threshold, a step the paper does not provide.
  • The embedded-prompt design predicts a measurable benefit: output consistency across users should be higher than with free-form AI chat assistants, since prompt phrasing is fixed; the paper does not report such a measurement.
  • Because the case study adds new features during modernization, the framework implicitly claims that requirements-first work is an opportunity to redesign rather than merely translate; this could be tested by tracking which legacy behaviors are preserved versus deliberately changed and whether regressions appear in changed areas.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a framework for modernizing legacy Java/.NET applications using large language models (LLMs) by combining code reasoning and code generation with human oversight. The framework generates functional requirements from the legacy code, reviews and refines them with subject-matter experts, and then generates the modernized application layer by layer (data, business logic, interaction), using progressive prompting and a reference implementation tool. A qualitative case study on the Spring PetClinic application is presented, along with a list of observations and limitations. The central claim is that this approach effectively addresses reliability, security, and quality challenges in application modernization.

Significance. If substantiated, the framework would offer a low-cost, locally deployable, standardized alternative to ad-hoc LLM-assisted modernization, with transparency-oriented model selection and built-in human review. The paper makes a concrete contribution by describing an opinionated, tool-embedded workflow and by releasing an open-source reference implementation, which lowers the barrier to experimentation. However, the current evidence is anecdotal: the case study reports no metrics, baselines, or error analysis, and the principal automated verification method is self-cited and, by the manuscript's own admission, not reliably automatable. The self-referential character of the reference implementation (generated by the framework it demonstrates) further limits the evidentiary weight. The framework is plausibly useful, but the paper does not yet establish the claimed effectiveness.

major comments (4)
  1. [Section 4.1] The foundational step of requirements generation is not validated against ground truth. The case study uses Spring PetClinic, whose actual behavior is publicly known, yet the paper never compares the LLM-generated functional requirements to the real behavior of the application. The only check is the author's review and editing, with no error-rate measurement, inter-rater agreement, or independent validation. Since every later artifact (data model, ORM, API, tests, UI) inherits the correctness of these requirements, a hallucinated or incomplete requirement propagates through the entire modernization. This directly undermines the central claim that the framework effectively addresses reliability.
  2. [Sections 3.5 and 5(d)] Reverse Generation Verification is the only automated consistency check proposed, but it is cited to the author's own prior work [12] and is not independently validated in this manuscript. Section 5(d) explicitly concedes that creating a reliably automated process is challenging and goes little beyond text-similarity scores. Using this method as evidence of output quality is therefore circular: the validity of the check is assumed rather than demonstrated. The manuscript should either provide independent evidence of the method's reliability or significantly weaken the claim that it addresses quality assurance.
  3. [Section 4 (case study overall)] The case study reports no quantitative outcomes: no success rates, defect counts, time measurements, security scan results, or comparisons against direct code translation or other modernization baselines. The abstract promises 'assessing alternative approaches where applicable,' but the case study only documents the sequence of steps and observations. Moreover, Section 5(b) admits occasional random behavior (incomplete or misaligned responses) and Section 5(e) admits random syntax errors across all models. These limitations directly contradict the abstract's implication that the framework tackles reliability challenges effectively. The paper needs a measured evaluation to support that claim.
  4. [Section 3.4] The reference implementation tool was itself generated using the framework with minimal human intervention, and is then used in the case study as evidence of the framework's capability. This is a self-referential demonstration: the tool both embodies the framework and is cited as proof of it. Without an independent implementation or external evaluation, the case study cannot distinguish whether observed successes are due to the framework's methodology, the specific LLM, the author's manual corrections, or the particular application chosen.
minor comments (5)
  1. [Section 3 heading] The heading contains a typo: 'The Framework and the Tooll' should be 'The Framework and the Tool.'
  2. [Section 3.1] The phrase 'It's worth nothing' appears to be a typo; it should read 'It's worth noting.'
  3. [Section 6] The section header '6 V. CONCLUSION' includes an extra 'V.' and should simply be '6 Conclusion.'
  4. [Figures 5-7] The figures referenced in the case study (Figures 5-7) are not described in the running text; please ensure each figure is legible, captioned, and accompanied by a brief explanation of what the reader should observe.
  5. [Section 1] The scope statement excludes COBOL and Fortran, but reference [1] is about Fortran-to-Python transformation; clarifying how that reference relates to the stated scope would avoid confusion.

Circularity Check

3 steps flagged · score 6.0 of 10

The framework's quality assurance relies on a self-referential loop: reverse-generation verification compares the LLM with itself, is attributed to the author's own prior work [12], and the reference implementation is itself a product of the same process.

  1. self definitional [Section 3.5, 'Additional Techniques for Quality Assurance', 'Reverse Generation Verification' bullet]
    "Reverse Generation Verification: This method involves feeding the generated output back into the LLM to verify its accuracy. For example, if an API is generated for a set of requirements, the API code is re-entered into the LLM with a prompt to generate the initial requirements. The two sets of requirements are then compared to ensure consistency and completeness as explained in [[12]]."

    Accuracy is operationalized as consistency between two sets of LLM-generated requirements. Because the same LLM produces both the original requirements and the re-generated requirements, the check compares the model to itself rather than to any external ground truth, such as the actual behavior of the legacy application. Section 5(d) confirms that automation 'proves challenging beyond relying on text similarity scores,' so the claimed verification reduces to self-agreement by construction.

  2. self citation load bearing [Section 3.5, 'Additional Techniques for Quality Assurance' and Section 5(d)]
    "Both reverse generation verification and the use of a verification LLM were applied selectively in the case study to assess the quality and completeness of outputs. The results showed good success in detecting and addressing potential errors. ... The Reverse Generation Verification approach[[12]] is a promising candidate for automation. However, due to the subtle variations in LLM outputs, creating a consistently reliable automation process proves challenging beyond relying on text similarity scores."

    The paper presents 'good success' in quality assurance on the basis of reverse generation verification, which is cited to the author's own prior work [12]. No external validation, independent reproduction, or benchmark is provided for that technique in the present paper. Since the only automated quality check is both self-referential and self-cited, the central claim that the framework 'effectively addresses reliability, security, and quality' rests on a citation chain that does not add independent evidence.

1 more flagged steps
  1. other [Section 3.4, 'Implementation of the Tool', first paragraph]
    "The reference tool implementation was itself generated using an LLM via progressive prompting, with minimal human intervention, demonstrating the framework’s capability in practical use."

    The tool used to demonstrate the framework was itself produced by the same kind of LLM-driven progressive prompting that the framework prescribes. This makes the demonstration self-referential: the artifact is generated by the very process it is meant to validate. Without an independent audit or comparison against a non-framework-built baseline, using the tool as evidence of the framework's capability closes the loop rather than providing external confirmation.

full rationale

The paper is a framework-and-case-study contribution, not a formal derivation, but its evidentiary chain for the central claim does contain a circular component. The framework's key automated quality check, reverse generation verification, compares an LLM's re-generated requirements with the original LLM-generated requirements; this is consistency with itself, not correctness against the legacy system. The paper itself concedes in Section 5(d) that the method reduces to text-similarity comparison, which confirms the reduction. This check is additionally attributed to the author's own prior work [12], and no independent validation is offered, making the self-citation load-bearing for the claimed 'good success' in quality assurance. The reference implementation being generated by the same process adds another self-referential layer. On the other hand, the framework also depends on human review at every step and on standard enterprise security lifecycles, so the circularity is partial rather than total. No external benchmark or ground-truth comparison is reported in the case study, so the score reflects a central quality-assurance mechanism that reduces to self-consistency, while the human-in-the-loop elements provide independent but unmeasured input.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities; the reference implementation is a software artifact, not a postulated entity. All assumptions are domain-level and none are empirically validated within the paper.

assumptions (4)
  • domain assumption Java/.NET legacy applications are well-suited for LLM modernization due to abundant training data.
    Section 1, paragraph 2: The exclusion of COBOL/Fortran and focus on contemporary languages rests on this assumption.
  • domain assumption Extracting requirements directly from application code is more reliable than existing documentation.
    Section 3.1, paragraph 3, asserts this without empirical support.
  • ad hoc to paper Reverse Generation Verification provides a valid consistency check between generated code and requirements.
    Section 3.5, cited to [12] (author's own prior work); effectiveness is asserted without data.
  • domain assumption Human review at each step is sufficient to catch errors introduced by LLMs.
    Section 3.5, human-centered protection approach assumes reviewer diligence and expertise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Application Modernization with LLMs: Addressing Core Challenges in Reliability, Security, and Quality." pith.science (2026). https://pith.science/paper/Y2M4MVEX

@misc{pith2026250610984,
  author       = {Pith},
  title        = {Pith review of: Application Modernization with LLMs: Addressing Core Challenges in Reliability, Security, and Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2M4MVEX}},
  note         = {Machine review of arXiv:2506.10984}
}
read the original abstract

AI-assisted code generation tools have revolutionized software development, offering unprecedented efficiency and scalability. However, multiple studies have consistently highlighted challenges such as security vulnerabilities, reliability issues, and inconsistencies in the generated code. Addressing these concerns is crucial to unlocking the full potential of this transformative technology. While advancements in foundational and code-specialized language models have made notable progress in mitigating some of these issues, significant gaps remain, particularly in ensuring high-quality, trustworthy outputs. This paper builds upon existing research on leveraging large language models (LLMs) for application modernization. It explores an opinionated approach that emphasizes two core capabilities of LLMs: code reasoning and code generation. The proposed framework integrates these capabilities with human expertise to tackle application modernization challenges effectively. It highlights the indispensable role of human involvement and guidance in ensuring the success of AI-assisted processes. To demonstrate the framework's utility, this paper presents a detailed case study, walking through its application in a real-world scenario. The analysis includes a step-by-step breakdown, assessing alternative approaches where applicable. This work aims to provide actionable insights and a robust foundation for future research in AI-driven application modernization. The reference implementation created for this paper is available on GitHub.

Figures

Figures reproduced from arXiv: 2506.10984 by the authors.

Figure 1
Figure 1. Interaction diagram showing the requirements generation process. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Recommended steps for Requirements Generation. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Interaction diagram showing the new application development process. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Recommended steps for Application (code) generation. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Requirements generation output • Consolidate Functional Requirements for Application/Module: – Generate functional requirements for the desired module or the entire application. – Review the generated output thoroughly and make necessary edits or additions to ensure co…
Figure 6
Figure 6. Figure 6: Consolidated Application requirements with added ratings require [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Generated Data Model and SQL Script. • Generate Additional Artifacts: – Generate ORM objects iteratively using the Data Model as the foun￾dation. – Utilize the ORM object code as input to create API code for the application. – Based on the generated API code, produce c…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

  1. [12]

    Bridging llm-generated code and requirements: Reverse generation technique and sbc metric for developer insights,

    A. A. N. Ponnusamy, “Bridging llm-generated code and requirements: Reverse generation technique and sbc metric for developer insights,” arXiv preprint arXiv:2502.07835 , 2025. [Online]. Available: https: //arxiv.org/abs/2502.07835

    arXiv 2025

  2. [1]

    Bridging eras: Transforming fortran legacies into python with the power of large language models,

    R. Pietrini, M. Paolanti, and E. Frontoni, “Bridging eras: Transforming fortran legacies into python with the power of large language models,” in 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI). IEEE, Apr. 2024

    work page 2024

  3. [2]

    Leveraging llms for legacy code modernization: Challenges and opportunities for llm-generated documentation,

    C. Diggs, M. Doyle, A. Madan, S. Scott, E. Escamilla, J. Zimmer, N. Nekoo, P. Ursino, M. Bartholf, Z. Robin, A. Patel, C. Glasz, W. Macke, P. Kirk, J. Phillips, A. Sridharan, D. Wendt, S. Rosen, N. Naik, J. F. Brunelle, and S. Thaker, “Leveraging llms for legacy code modernization: Challenges and opportunities for llm-generated documentation,” arXiv, Nov. 2024

    work page 2024

  4. [3]

    Requirements are all you need: From requirements to code with llms,

    B. Wei, “Requirements are all you need: From requirements to code with llms,” arXiv, Jun. 2024

    work page 2024

  5. [4]

    Optimizing ai-assisted code generation: En- hancing security, efficiency, and accessibility in software development,

    S. Torka and S. Albayrak, “Optimizing ai-assisted code generation: En- hancing security, efficiency, and accessibility in software development,” arXiv, Dec. 2024

    work page 2024

  6. [5]

    Ibm granite transparency report,

    I. Research, “Ibm granite transparency report,” 2024. [Online]. Available: https://research.ibm.com/blog/ibm-granite-transparency-fmti 15

    work page 2024

  7. [6]

    Ibm granite responsible use guide,

    IBM, “Ibm granite responsible use guide,” 2024. [On- line]. Available: https://www.ibm.com/granite/docs/resources/ responsible-use-guide.pdf?utm source=chatgpt.com

    work page 2024

  8. [7]

    Codet5: Identifier-aware uni- fied pre-trained encoder-decoder models for code understanding and gen- eration,

    Y. Wang, W. Wang, S. Joty, and S. Hoi, “Codet5: Identifier-aware uni- fied pre-trained encoder-decoder models for code understanding and gen- eration,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2021, pp. 8696–8708

    work page 2021

Show all 13 references
  1. [8]

    Using ai-based cod- ing assistant in practice: State of affairs, perceptions, and ways forward,

    A. Sergeyuk, Y. Golubev, T. Bryksin, and I. Ahmed, “Using ai-based cod- ing assistant in practice: State of affairs, perceptions, and ways forward,” arXiv, Nov. 2024

    2024

  2. [9]

    From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy,

    M. Gupta, C. Akiri, K. Aryal, E. Parker, and L. Praharaj, “From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy,” IEEE Access, vol. 11, pp. 80 218–80 245, 2023

    2023

  3. [10]

    Do users write more insecure code with ai assistants?

    N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . Association for Computing Machinery, 2023, pp. 2785–2799

    2023

  4. [11]

    Lost at c: A user study on the security implications of large language model code assistants,

    G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt, “Lost at c: A user study on the security implications of large language model code assistants,” in Proceedings of the 32nd USENIX Security Symposium . USENIX Association, 2023, pp. 2205–2222. [Online]. Ava...

    2023

  5. [13]

    Assessing the effectiveness and security implications of ai code generators,

    M. Taeb, H. Chi, and S. Bernadin, “Assessing the effectiveness and security implications of ai code generators,” Journal of The Colloquium for Information Systems Security Education , vol. 11, no. 6, 2024. [Online]. Available: https://cisse.info/journal/index.php/cisse/article...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.