Pith. sign in

REVIEW 3 major objections 6 minor 33 references

How is Google using AI for internal code migrations?

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a pipeline of static-analysis discovery, LLM-generated edits, automated validation, and human review reduces end-to-end code-migration time by at least 50% and makes previously stalled migrations feasible.

desk verdict An honest Google experience report with a useful agentless LLM+AST architecture; treat the headline time-saving numbers as directional, not measured. read the letter →

arxiv 2501.06972 v1 pith:YCBLEN5G submitted 2025-01-12 cs.SE

Stoyan Nikolov , Daniele Codecasa , Anna Sjovall , Maxim Tabachnyk , Satish Chandra , Siddharth Taneja , Celal Ziftci This is my paper · ORCID
classification cs.SE
keywords LLMcodemigrationexperiencereportlargelanguagemodelsabstractsyntaxtreerepository-levelchangetechnicaldebtGooglemonorepoJUnit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This experience report argues that a pipeline combining static-analysis-based change discovery, LLM-generated edits, automated build/test validation, and human review can cut the end-to-end time of large code migrations by at least half, and can restart migrations that had stalled for years. The paper's success metric is deliberately end-to-end: at least 50% savings on the whole journey, not just the code rewrite. Four case studies are presented, with the Google Ads 32-bit-to-64-bit ID migration on track to meet that target and the Joda-time-to-java.time migration estimated at about 89% savings on small clusters. A sympathetic reader takes the paper's central claim to be that bespoke, LLM-assisted migration workflows are a practical way to retire technical debt at enterprise scale.

What carries the argument

The load-bearing machinery is a migration toolkit that combines deterministic discovery with generative editing. Kythe cross-reference data locates candidate files and change sites, often expanded through breadth-first reference searches and filtered with regexes and AST parsing. A fine-tuned Gemini model, trained with the DIDACT methodology on Google's internal code and development logs, generates the actual diffs from natural-language prompt rules. Multiple prompt variations are tried in parallel, similar to a pass@k strategy, and the results are scored by automated validations such as builds and unit tests, with an optional ML-powered repair loop for failures. Human engineers review the final changelists and handle rollout, and the paper emphasizes that this hybrid LLM-plus-AST design, rather than LLM prompting alone, is what makes the migrations reliable.

What would settle it

Run the same migration task on matched codebases with and without the LLM-assisted toolkit, logging actual engineer-hours including discovery, editing, review, and rollout; if the measured difference falls below 50%, the paper's central claim would not hold. Alternatively, an independent audit of the Ads migration's tracked time against the estimated baseline would settle whether the reported savings are real.

Watch

Extended reading notes

Core claim

The paper claims that LLMs, when combined with deterministic code-analysis tools and human review, significantly reduce the time required for code migrations and lower the barrier to starting and completing them. In the Google Ads int32-to-int64 ID migration, engineers reported an estimated 50% reduction in total migration time, including review and landing. In the JUnit3-to-JUnit4 migration, 5,359 files and over 149,000 lines were migrated in three months, with about 87% of the AI-generated code committed unchanged. For small clusters in the ongoing Joda-time-to-java.time migration, the estimated time savings are about 89% compared with the prior manual approach. The paper also reports that several migrations that had been stalled for years were completed by a handful of engineers using this workflow.

Load-bearing premise

The central acceleration claim rests on the assumption that self-reported and expert-estimated time savings are accurate proxies for true end-to-end effort, since the paper's headline figures come from engineers' estimates and extrapolations rather than from controlled before-and-after measurements.

Editorial extensions

If this is right

  • The Google Ads int32-to-int64 migration is on track to meet the 50% end-to-end time-savings target, avoiding an effort estimated at hundreds of engineer-years.
  • The JUnit3-to-JUnit4 case shows that a small central team can migrate over five thousand test files in three months, with the main bottleneck being human review speed.
  • The Joda-time-to-java.time case shows that even partially correct AI-generated edits help engineers quickly find all the places that need changing, cutting effort even when the model makes mistakes.
  • Migrations that previously stalled for years become feasible with a few engineers, because one engineer can generate and coordinate changes across many teams.
  • The workflow generalizes across migration types: deterministic discovery, LLM edit generation, automated validation, and human review form a reusable pattern for other large codebases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline time-savings figures are estimates from engineers rather than controlled measurements, so if anything, the reported 50% and 89% numbers likely understate the speedup in change generation alone, because the end-to-end metric includes slower human review and rollout.
  • Because the success metric is anchored to 50% rather than to code-autonomy rate, this paper's evidence is compatible with LLMs acting as a force multiplier rather than as autonomous agents; the practical claim is about total engineer effort, not automation level.
  • The hybrid pattern of static localization plus generative editing plus validation likely transfers to other enterprise technical-debt tasks such as API deprecations, framework upgrades, and dead-code removal, where the change sites are discoverable but the edits vary too much for deterministic AST transforms.
  • The hardest unresolved piece, cleaning entangled test code where multiple flags or conditions interact, points to a concrete next research direction: automated test simplification guided by failing tests and LLM repair.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper is an experience report from Google on using LLMs for internal code migrations. It describes a pipeline that combines AST-based discovery and validation with LLM-based edit generation and human review, and it reports on four case studies: int32-to-int64 ID migration (Section IV), JUnit3-to-JUnit4 migration (Section V), Joda-time-to-java.time migration (Section VI), and experimental-flag cleanup (Section VII). The abstract claims evidence that LLM use can significantly reduce migration time and lower barriers to starting and completing migrations, with a success metric defined as at least 50% end-to-end time savings (Section III). The authors explicitly state that the paper is not a research study and does not compare against other approaches.

Significance. The paper's qualitative contribution is valuable as a practitioners' experience report. It gives a concrete, credible account of how an LLM+AST+human-review pipeline can be operationalized in a large monorepo, including localization via Kythe, fine-tuned edit generation, validation/repair loops, and deliberate throttling of review load. The discussion of when to prefer AST techniques over LLMs (Section VIII-B) is a useful practical insight, and the authors are transparent about the main evaluation limitation: the headline acceleration figures rest on expert estimates and extrapolation (Section III). If the quantitative claims are either strengthened with methodologically sound data or appropriately softened, the paper would be a solid contribution to the experience-report literature. As it stands, the quantitative claims are not established at the level the abstract implies, so the paper's central assertion currently has weaker support than its descriptive content.

major comments (3)
  1. [Section III, Section IV, Section VI] The quantitative acceleration evidence is not sufficient for the abstract's claim. Section IV reports 'The total time spent on the migration was reduced by an estimated 50% as reported by the engineers doing the migration, when compared to a similar exercise carried out without LLM assistance,' but gives no detail on how the comparison exercise was matched, when and how the estimates were collected, how many engineers or CLs were involved, or whether the estimates were prospective. Section VI reports '~89% of the time' saved in small clusters, 'calculated across multiple changes where human experts (team technical leads) have compared their experience with the AI-powered tooling to the previous purely-manual approach,' again without the number of samples, the spread, or a definition of 'small.' Section III itself concedes that for ongoing migrations 'we relied on the expert engineers estimating a time-saving for a set of changelists created by the tools and we extrapolated from that.' Since the abstract's 'We see evidence that the use of LLMs can reduce the time needed for migrations significantly' is the central claim, the paper should either provide a detailed estimation protocol (structured interviews, time logs, matched baseline CLs, raw data or distributions) or explicitly downgrade these headline numbers to anecdotal evidence in the abstract and conclusions.
  2. [Section V] The end-to-end time metric is confounded by deliberate throttling. Section V states: 'We purposefully limited the number of changes we generate every weak to avoid overwhelming reviewers.' If the number of generated changes is throttled to match review capacity, then elapsed calendar time to complete a migration is partly determined by human review availability, not by AI speed. The paper should reconcile the success metric defined in Section III ('AI saving at least 50% of the time for the end-to-end work') with this throttling, and clarify whether the reported 50% and 89% figures are estimated effort savings or measured end-to-end elapsed time. Additionally, no end-to-end time saving is reported for the JUnit migration in Section V, so that case study cannot serve as evidence for the abstract's acceleration claim.
  3. [Section IV] The reported 80% AI-authored code is a well-defined engineering metric, but the paper does not establish that it translates to the claimed 50% time savings. The paper defines the metric precisely (per-character difference between the first LLM snapshot and the committed CL), but the 50% figure is a separate, informally reported estimate. The causal link between '80% of code modifications fully AI-authored' and '50% reduction in total time' is asserted rather than demonstrated, especially given the human review and rollback costs described later in the same section. The paper should either present data linking the two or separately justify the time-savings estimate.
minor comments (6)
  1. [Section V] Typo: 'every weak' should be 'every week' in the sentence about limiting the number of changes.
  2. [Section IV] Typo: 'bread-first-search' should be 'breadth-first-search' in the description of the Kythe reference traversal.
  3. [Section VI] The term 'small clusters' is never defined. Please specify a concrete size range (e.g., number of files or CLs) so the scope of the 89% estimate is reproducible.
  4. [Section I] The claim that the effort 'saved the business hundreds of engineers worth of work' is unsupported by any calculation or definition. Please provide a basis or soften the claim.
  5. [Section VIII-E] The sentence 'We are constantly measuring ourselves at the level of business-level outcomes' is vague. Please list the specific metrics that are actually tracked for migration programs.
  6. [Figure 1] The caption says 'first 3 quarters' but does not fully specify the period or whether the counts cover only the four described migrations or the entire migration ecosystem. Please clarify and ensure the axes are legible in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an experience report whose acceleration figures rest on expert self-estimates, not on a derivation that reduces to its inputs.

full rationale

The paper explicitly disclaims being a research study and does not derive predictions from fitted parameters, so there is no derivation chain that reduces to its own inputs. The success metric in Section III ('we have defined success as AI saving at least 50% of the time for the end-to-end work') is a criterion applied to estimates, not an equation whose output is defined by its input. The reported 50% figure in Section IV and the ~89% figure in Section VI come from engineers' retrospective comparisons with manual baselines ('we relied on the expert engineers estimating a time-saving for a set of changelists created by the tools and we extrapolated from that'), which is a measurement-dependence concern, not circular reasoning. Self-citations such as DIDACT [17] and the prior migration blog post [18] provide background on model training and earlier deployment, but they are not used to infer the central acceleration numbers. There is no fitted parameter renamed as prediction, no uniqueness theorem imported from the authors, and no equation in which a claimed result is equal by construction to its input. The main weakness is that the quantitative claims rest on non-baselined, self-reported estimates; that belongs to correctness and evidence quality, not circularity, and it does not change the conclusion that the paper's argument is not self-referential in the circularity sense.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new scientific entities. Its evaluation rests on a chosen success threshold and on the assumption that self-reported expert estimates and internal proprietary tooling support the generalizable acceleration claim. No independent external benchmark is used.

free parameters (1)
  • Success threshold of 50% end-to-end acceleration = 50%
    Defined in Section III as the success metric for all migrations. It is chosen by the authors rather than derived, and the central claim is benchmarked against it.
assumptions (3)
  • domain assumption Expert engineers' retrospective time estimates validly measure end-to-end migration effort.
    The 50% and 89% savings figures rely on engineers' reports and extrapolation from expert estimates (Sections IV and VI).
  • domain assumption The internal fine-tuned Gemini model and Google-specific datasets represent what other enterprises can expect from available LLMs.
    The authors generalize beyond Google (Section I, III, VIII), but the core model is proprietary and fine-tuned on Google's monorepo, so external transferability is assumed.
  • domain assumption Human code review reliably catches errors introduced by LLM-generated edits, so code quality is maintained.
    Section III states that quality is assured through manual review same as human code; this assumption underlies the claim that acceleration does not come at a quality cost.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How is Google using AI for internal code migrations?." pith.science (2026). https://pith.science/paper/YCBLEN5G

@misc{pith2026250106972,
  author       = {Pith},
  title        = {Pith review of: How is Google using AI for internal code migrations?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YCBLEN5G}},
  note         = {Machine review of arXiv:2501.06972}
}
read the original abstract

In recent years, there has been a tremendous interest in using generative AI, and particularly large language models (LLMs) in software engineering; indeed there are now several commercially available tools, and many large companies also have created proprietary ML-based tools for their own software engineers. While the use of ML for common tasks such as code completion is available in commodity tools, there is a growing interest in application of LLMs for more bespoke purposes. One such purpose is code migration. This article is an experience report on using LLMs for code migrations at Google. It is not a research study, in the sense that we do not carry out comparisons against other approaches or evaluate research questions/hypotheses. Rather, we share our experiences in applying LLM-based code migration in an enterprise context across a range of migration cases, in the hope that other industry practitioners will find our insights useful. Many of these learnings apply to any application of ML in software engineering. We see evidence that the use of LLMs can reduce the time needed for migrations significantly, and can reduce barriers to get started and complete migration programs.

Figures

Figures reproduced from arXiv: 2501.06972 by the authors.

Figure 1
Figure 1. Landed changelists of AI-powered migrations for the first 3 quarters [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Improving AI-based features in coding tools (e.g., in the IDE) with [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. A demonstration of how a variety of AI-based features can work [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The high-level process to land an AI-authored change in the monorepo. We use LLMs extensively in code change creation, and partly in discovery [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Example execution of the multi-stage code migration process. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Prompt for int32 to int64 migration. The model makes the change [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: In the example above we prompt the model to only update the [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: The model updated also the test file with an integer that is larger than [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Excerpt from the JUnit3 to JUnit4 prompt [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Changes required for a case of JUnit3 to JUnit4 conversion. [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 12
Figure 12. Figure 12: Example of a Joda time to Java time migration [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 11
Figure 11. Figure 11: Prompt for Joda Time conversion Gemini effectively would not know, between inference invo￾cations, if the referenced file is migrated or not - it needs to assume if a direct call is needed or a type conversion. The alternative is to show the already-migrated file to t…
Figure 13
Figure 13. Figure 13: Prompt for unused flag cleanup [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 14
Figure 14. Figure 14: The model discovers and ’tags’ a test flag related to the implemen [PITH_FULL_IMAGE:figures/full_fig_p009_14.png]
Figure 15
Figure 15. Figure 15: The first large block with red background is a direct flag dependency, [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

  1. [1]

    AmazonQ. 2024. URL: https : / / aws . amazon . com / q / developer/code-transformation/

    work page 2024

  2. [2]

    CodePlan: Repository-level Coding using LLMs and Planning

    Ramakrishna Bairi et al. CodePlan: Repository-level Coding using LLMs and Planning . 2023. arXiv: 2309. 12499 [cs.SE]. URL: https : / / arxiv. org / abs / 2309 . 12499

    work page 2023

  3. [3]

    AI in software engineering at Google: Progress and the path ahead

    Satish Chandra and Maxim Tabachnyk. AI in software engineering at Google: Progress and the path ahead

    work page

  4. [4]

    Evaluating Large Language Models Trained on Code

    Mark Chen et al. Evaluating Large Language Models Trained on Code . 2021. arXiv: 2107.03374 [cs.LG]. URL: https://arxiv.org/abs/2107.03374

    arXiv 2021

  5. [5]

    URL: https://clang.llvm.org/ docs/RefactoringEngine.html

    Clang’s refactoring engine. URL: https://clang.llvm.org/ docs/RefactoringEngine.html

    work page

  6. [6]

    Devin. 2024. URL: https : / / www . cognition . ai / introducing-devin/

    work page 2024

  7. [7]

    Multi-line AI-assisted Code Author- ing

    Omer Dunay et al. Multi-line AI-assisted Code Author- ing. 2024. arXiv: 2402.04141 [cs.SE]. URL: https: //arxiv.org/abs/2402.04141

    arXiv 2024

  8. [8]

    Towards Translating Real- World Code with LLMs: A Study of Translating to Rust

    Hasan Ferit Eniser et al. Towards Translating Real- World Code with LLMs: A Study of Translating to Rust

    work page

Show all 33 references
  1. [9]

    Smart Paste for context-aware adjustments to pasted code

    Simone Forte and Marcus Revaj. Smart Paste for context-aware adjustments to pasted code . 2024. URL: https://research.google/blog/smart- paste- for- context- aware-adjustments-to-pasted-code/

    2024

  2. [10]

    URL: https://arxiv

    arXiv: 2405.11514 [cs.SE]. URL: https://arxiv. org/abs/2405.11514

    arXiv

  3. [11]

    Resolving Code Re- view Comments with Machine Learning

    Alexander Fr ¨ommgen et al. “Resolving Code Re- view Comments with Machine Learning”. In: 2024 IEEE/ACM 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE- SEIP). 2024

    2024

  4. [12]

    Resolving code review comments with ML

    Alexander Fr ¨ommgen and Lera Kharatyan. Resolving code review comments with ML . 2023. URL: https : / / research . google / blog / resolving - code - review - comments-with-ml/

    2023

  5. [13]

    Guice. 2025. URL: https://github.com/google/guice/

    2025

  6. [14]

    Ghallab, D

    M. Ghallab, D. Nau, and P. Traverso. Automated Plan- ning: Theory and Practice . The Morgan Kaufmann Series in Artificial Intelligence. Elsevier Science, 2004. ISBN : 9781558608566. URL: https://books.google.de/ books?id=eCj3cKC 3ikC

    2004

  7. [15]

    Safely repairing broken builds with ML

    Emily Johnston and Stephanie Tang. Safely repairing broken builds with ML . 2024. URL: https : / / research . google/blog/safely-repairing-broken-builds-with-ml/

    2024

  8. [16]

    Self-planning Code Generation with Large Language Models

    Xue Jiang et al. Self-planning Code Generation with Large Language Models . 2024. arXiv: 2303 . 06689 [cs.SE]. URL: https://arxiv.org/abs/2303.06689

    2024

  9. [17]

    Large sequence models for software development activities

    Petros Maniatis and Daniel Tarlow. Large sequence models for software development activities . 2023. URL: https://research.google/blog/large- sequence- models- for-software-development-activities/. 11

    2023

  10. [18]

    URL: https://kythe.io/

    Kythe. URL: https://kythe.io/

  11. [19]

    The Impact of AI on Developer Pro- ductivity: Evidence from GitHub Copilot

    Sida Peng et al. The Impact of AI on Developer Pro- ductivity: Evidence from GitHub Copilot . 2023. arXiv: 2302.06590. URL: https://arxiv.org/abs/2302.06590

    2023 arXiv

  12. [20]

    Accelerating code migrations with AI

    Stoyan Nikolov and Siddharth Taneja. Accelerating code migrations with AI . 2024. URL: https://research. google/blog/accelerating-code-migrations-with-ai/

    2024

  13. [21]

    Refactoring Programs Using Large Language Models with Few-Shot Examples

    Atsushi Shirafuji et al. “Refactoring Programs Using Large Language Models with Few-Shot Examples”. In: 2023 30th Asia-Pacific Software Engineering Con- ference (APSEC) . IEEE, Dec. 2023. DOI: 10 . 1109 / apsec60848.2023.00025. URL: http://dx.doi.org/10. 1109/APSEC60848.2023.00025

    2023

  14. [22]

    Protocol Buffers. 2025. URL: https://protobuf.dev/

    2025

  15. [23]

    ML-Enhanced Code Completion Improves Developer Productivity

    Maxim Tabachnyk and Stoyan Nikolov. ML-Enhanced Code Completion Improves Developer Productivity

  16. [24]

    SWEBench. 2024. URL: https://www.swebench.com/

    2024

  17. [25]

    AI-Assisted Assessment of Coding Practices in Modern Code Review

    Manushree Vijayvergiya et al. “AI-Assisted Assessment of Coding Practices in Modern Code Review”. In: Proceedings of the 1st ACM International Conference on AI-Powered Software . V ol. 24. AIware ’24. ACM, July 2024, pp. 85–93. DOI: 10.1145/3664646.3665664. URL: http://dx.doi....

    2024

  18. [26]

    Winters, T

    T. Winters, T. Manshreck, and H. Wright. Software Engineering at Google: Lessons Learned from Pro- gramming Over Time . O’Reilly Media, 2020. ISBN : 9781492082767. URL: https://books.google.de/books? id=V3TTDwAAQBAJ

    2020

  19. [27]

    Evaluating Human-AI Partnership for LLM-based Code Migration

    Behrooz Tehrani, Ishaani M, and Anmol Anubhai. “Evaluating Human-AI Partnership for LLM-based Code Migration”. In: May 2024, pp. 1–8. DOI: 10.1145/ 3613905.3650896

    2024

  20. [28]

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

    John Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering . 2024. arXiv: 2405.15793 [cs.SE]. URL: https://arxiv.org/abs/2405. 15793

    2024 arXiv

  21. [29]

    Planning with Large Language Models for Code Generation

    Shun Zhang et al. Planning with Large Language Models for Code Generation . 2023. arXiv: 2303.05510 [cs.LG]. URL: https://arxiv.org/abs/2303.05510

    2023 arXiv

  22. [30]

    Agentless: Demystifying LLM-based Software Engineering Agents

    Chunqiu Steven Xia et al. Agentless: Demystifying LLM-based Software Engineering Agents . 2024. arXiv: 2407.01489 [cs.SE]. URL: https://arxiv.org/abs/2407. 01489

    2024 arXiv

  23. [33]

    AutoCodeRover: Autonomous Program Improvement

    Yuntong Zhang et al. AutoCodeRover: Autonomous Program Improvement . 2024. arXiv: 2404 . 05427 [cs.SE]. URL: https://arxiv.org/abs/2404.05427. 12

    2024 arXiv

  24. [2022]

    URL: https://research.google/blog/ml-enhanced- code-completion-improves-developer-productivity/

  25. [2024]

    URL: https://research.google/blog/ai-in-software- engineering-at-google-progress-and-the-path-ahead/

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.