{"id":"e3bc3435-403a-4e77-bd3a-26f610a04ace","arxiv_id":"2501.06972","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An experience report shows Google using fine-tuned LLMs plus AST-based tooling and human review to accelerate large internal code migrations, with estimated time savings of roughly 50% to 89%.","lead":"Google engineers describe how they combined large language models with deterministic code analysis to migrate large internal codebases, for example converting 32-bit IDs to 64-bit. The report is a practical account of what worked in production, not a controlled study of whether the techniques beat alternatives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central acceleration figures rest on self-estimated, non-baselined time savings; without independent measurement the 50%/89% claims are not established.","rationale":"The reader's weakest assumption—that self-reported and expert-estimated time savings are unreliable proxies for true end-to-end effort—is the same load-bearing concern I identify. Every headline quantitative result in the paper depends on this assumption, and the paper itself explicitly states that the Joda estimate is an expert extrapolation and that the Ads estimate is engineer-reported relative to an unspecified \"similar exercise.\" Since the codebase, tooling, and data are internal and unavailable, the numbers cannot be independently audited. The paper's own disclaimer that it is not a research study lowers the expected evidential bar, but it does not eliminate the need for a baseline when an empirical acceleration claim is made. The qualitative contributions—the LLM+AST pipeline, validation loop, file-discovery heuristics, and human-review bottleneck—are valuable and credible regardless of the exact percentages. A matched-pairs repository-history check would settle whether the acceleration claim is robust or primarily an artifact of estimation method and scheduling. Because this concern supports the original CONDITIONAL verdict rather than overturning it, I recommend no change.","tokens_in":14182,"tokens_out":4125,"duration_ms":46354,"concrete_test":"Reconstruct a matched-pairs comparison from repository metadata: take the Ads int64 ID-migration CLs and a comparable pre-LLM large-scale migration (same languages, similar file counts and complexity) and compute end-to-end per-CL cycle time (first snapshot to submission) and total engineer-hours from review and approval records. If the median per-CL cycle-time reduction relative to the pre-LLM set is below 50%, the central acceleration claim fails; report the comparison both including and excluding the periods where generation was throttled to match reviewer capacity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that LLM-assisted migration delivers at least 50% end-to-end time savings (§III), with the Ads case reported at ~50% (§IV) and Joda small clusters at ~89% (§VI). The evidence for both numbers is the same mode: expert engineers' retrospective estimates, not measured effort or controlled comparison. Section III concedes: \"we relied on the expert engineers estimating a time-saving for a set of changelists created by the tools and we extrapolated from that.\" Section IV says the 50% figure is \"as reported by the engineers doing the migration, when compared to a similar exercise carried out without LLM assistance\"; neither the similarity of the baseline exercise nor the data-collection method is specified. Section VI gives no number of clusters, no spread, and no definition of \"small.\" The end-to-end metric is also confounded by deliberate throttling: Section V says they \"purposefully limited the number of changes we generate every week to avoid overwhelming reviewers,\" so the calendar time to complete a migration is partially set by reviewer capacity, not purely by AI speed. This makes the central quantitative claim vulnerable to inflated estimates and to conflation of effort savings with actual end-to-end time reduction. The qualitative pipeline details are credible and plausible, but they do not by themselves establish the acceleration headline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is an experience report from Google on using LLMs for internal code migrations. It describes a pipeline that combines AST-based discovery and validation with LLM-based edit generation and human review, and it reports on four case studies: int32-to-int64 ID migration (Section IV), JUnit3-to-JUnit4 migration (Section V), Joda-time-to-java.time migration (Section VI), and experimental-flag cleanup (Section VII). The abstract claims evidence that LLM use can significantly reduce migration time and lower barriers to starting and completing migrations, with a success metric defined as at least 50% end-to-end time savings (Section III). The authors explicitly state that the paper is not a research study and does not compare against other approaches.","tokens_in":14417,"tokens_out":5589,"duration_ms":54720,"significance":"The paper's qualitative contribution is valuable as a practitioners' experience report. It gives a concrete, credible account of how an LLM+AST+human-review pipeline can be operationalized in a large monorepo, including localization via Kythe, fine-tuned edit generation, validation/repair loops, and deliberate throttling of review load. The discussion of when to prefer AST techniques over LLMs (Section VIII-B) is a useful practical insight, and the authors are transparent about the main evaluation limitation: the headline acceleration figures rest on expert estimates and extrapolation (Section III). If the quantitative claims are either strengthened with methodologically sound data or appropriately softened, the paper would be a solid contribution to the experience-report literature. As it stands, the quantitative claims are not established at the level the abstract implies, so the paper's central assertion currently has weaker support than its descriptive content.","major_comments":[{"comment":"The quantitative acceleration evidence is not sufficient for the abstract's claim. Section IV reports 'The total time spent on the migration was reduced by an estimated 50% as reported by the engineers doing the migration, when compared to a similar exercise carried out without LLM assistance,' but gives no detail on how the comparison exercise was matched, when and how the estimates were collected, how many engineers or CLs were involved, or whether the estimates were prospective. Section VI reports '~89% of the time' saved in small clusters, 'calculated across multiple changes where human experts (team technical leads) have compared their experience with the AI-powered tooling to the previous purely-manual approach,' again without the number of samples, the spread, or a definition of 'small.' Section III itself concedes that for ongoing migrations 'we relied on the expert engineers estimating a time-saving for a set of changelists created by the tools and we extrapolated from that.' Since the abstract's 'We see evidence that the use of LLMs can reduce the time needed for migrations significantly' is the central claim, the paper should either provide a detailed estimation protocol (structured interviews, time logs, matched baseline CLs, raw data or distributions) or explicitly downgrade these headline numbers to anecdotal evidence in the abstract and conclusions.","section":"Section III, Section IV, Section VI"},{"comment":"The end-to-end time metric is confounded by deliberate throttling. Section V states: 'We purposefully limited the number of changes we generate every weak to avoid overwhelming reviewers.' If the number of generated changes is throttled to match review capacity, then elapsed calendar time to complete a migration is partly determined by human review availability, not by AI speed. The paper should reconcile the success metric defined in Section III ('AI saving at least 50% of the time for the end-to-end work') with this throttling, and clarify whether the reported 50% and 89% figures are estimated effort savings or measured end-to-end elapsed time. Additionally, no end-to-end time saving is reported for the JUnit migration in Section V, so that case study cannot serve as evidence for the abstract's acceleration claim.","section":"Section V"},{"comment":"The reported 80% AI-authored code is a well-defined engineering metric, but the paper does not establish that it translates to the claimed 50% time savings. The paper defines the metric precisely (per-character difference between the first LLM snapshot and the committed CL), but the 50% figure is a separate, informally reported estimate. The causal link between '80% of code modifications fully AI-authored' and '50% reduction in total time' is asserted rather than demonstrated, especially given the human review and rollback costs described later in the same section. The paper should either present data linking the two or separately justify the time-savings estimate.","section":"Section IV"}],"minor_comments":[{"comment":"Typo: 'every weak' should be 'every week' in the sentence about limiting the number of changes.","section":"Section V"},{"comment":"Typo: 'bread-first-search' should be 'breadth-first-search' in the description of the Kythe reference traversal.","section":"Section IV"},{"comment":"The term 'small clusters' is never defined. Please specify a concrete size range (e.g., number of files or CLs) so the scope of the 89% estimate is reproducible.","section":"Section VI"},{"comment":"The claim that the effort 'saved the business hundreds of engineers worth of work' is unsupported by any calculation or definition. Please provide a basis or soften the claim.","section":"Section I"},{"comment":"The sentence 'We are constantly measuring ourselves at the level of business-level outcomes' is vague. Please list the specific metrics that are actually tracked for migration programs.","section":"Section VIII-E"},{"comment":"The caption says 'first 3 quarters' but does not fully specify the period or whether the counts cover only the four described migrations or the entire migration ecosystem. Please clarify and ensure the axes are legible in the final version.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a practitioner-oriented experience report; it may be a better fit for a software-engineering practice venue than for a research-oriented archival journal, but if the journal publishes such reports, the main issue is the mismatch between the confident abstract and the evidence basis. The authors should either significantly strengthen the quantitative evaluation or explicitly reframe the acceleration claims as anecdotal. The paper's reliance on internal Google blog posts for its generic-tooling impact claims is acceptable in this context, but it should not be the basis for the central quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the short version: the paper is an honest experience report, not a controlled study. The real value is the architecture: an agentless LLM+AST pipeline used across four enterprise migrations, with Kythe for discovery, deterministic filters for targeting, an LLM fine-tuned internally for edit generation, and a build/test repair loop. That hybrid pattern is credible and likely useful to practitioners.\n\nWhat it does well: the case studies are concrete, with prompts and diffs, and the discussion of review bandwidth as the bottleneck matches what others see. The related work is brief but fair, and the self-citations are contextual.\n\nThe soft spots are the acceleration figures. The 50% and 89% numbers come from engineer estimates and expert extrapolation, not from a controlled baseline. The paper says this openly in Sections III and VI. There are no error bars, no raw data, and the Joda 'small clusters' are never defined. The throttling in Section V—limiting weekly changes to protect reviewers—means calendar time is partly set by human capacity, so even a well-measured end-to-end comparison would be confounded. Treat the percentages as directional.\n\nWho is this for? Practitioners at large companies will get value from the architecture and lessons. Researchers should treat it as an industrial reference point, not as evidence of specific speedups. It deserves a serious referee because it is a substantial, honest experience report. The referee should push for clearer estimation methods and label the figures as directional. I'd bring it to a reading group, but frame it as architecture, not evaluation.","headline":"An honest Google experience report with a useful agentless LLM+AST architecture; treat the headline time-saving numbers as directional, not measured.","tokens_in":786,"tokens_out":1165,"would_cite":true,"duration_ms":30138,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a pipeline of static-analysis discovery, LLM-generated edits, automated validation, and human review reduces end-to-end code-migration time by at least 50% and makes previously stalled migrations feasible.","keywords":["LLM code migration","experience report","large language models","abstract syntax tree","repository-level change","technical debt","Google monorepo","JUnit migration"],"falsifier":"Run the same migration task on matched codebases with and without the LLM-assisted toolkit, logging actual engineer-hours including discovery, editing, review, and rollout; if the measured difference falls below 50%, the paper's central claim would not hold. Alternatively, an independent audit of the Ads migration's tracked time against the estimated baseline would settle whether the reported savings are real.","tokens_in":13999,"feed_emoji":"⚙️","tokens_out":3853,"duration_ms":39735,"temperature":0.7,"pith_summary":"This experience report argues that a pipeline combining static-analysis-based change discovery, LLM-generated edits, automated build/test validation, and human review can cut the end-to-end time of large code migrations by at least half, and can restart migrations that had stalled for years. The paper's success metric is deliberately end-to-end: at least 50% savings on the whole journey, not just the code rewrite. Four case studies are presented, with the Google Ads 32-bit-to-64-bit ID migration on track to meet that target and the Joda-time-to-java.time migration estimated at about 89% savings on small clusters. A sympathetic reader takes the paper's central claim to be that bespoke, LLM-assisted migration workflows are a practical way to retire technical debt at enterprise scale.","feed_headline":"LLMs cut code-migration time by half or more","feed_subtitle":"Google's experience report shows an LLM-plus-AST pipeline finishing migrations that had stalled for years.","key_machinery":"The load-bearing machinery is a migration toolkit that combines deterministic discovery with generative editing. Kythe cross-reference data locates candidate files and change sites, often expanded through breadth-first reference searches and filtered with regexes and AST parsing. A fine-tuned Gemini model, trained with the DIDACT methodology on Google's internal code and development logs, generates the actual diffs from natural-language prompt rules. Multiple prompt variations are tried in parallel, similar to a pass@k strategy, and the results are scored by automated validations such as builds and unit tests, with an optional ML-powered repair loop for failures. Human engineers review the final changelists and handle rollout, and the paper emphasizes that this hybrid LLM-plus-AST design, rather than LLM prompting alone, is what makes the migrations reliable.","core_discovery":"The paper claims that LLMs, when combined with deterministic code-analysis tools and human review, significantly reduce the time required for code migrations and lower the barrier to starting and completing them. In the Google Ads int32-to-int64 ID migration, engineers reported an estimated 50% reduction in total migration time, including review and landing. In the JUnit3-to-JUnit4 migration, 5,359 files and over 149,000 lines were migrated in three months, with about 87% of the AI-generated code committed unchanged. For small clusters in the ongoing Joda-time-to-java.time migration, the estimated time savings are about 89% compared with the prior manual approach. The paper also reports that several migrations that had been stalled for years were completed by a handful of engineers using this workflow.","pith_inferences":["The headline time-savings figures are estimates from engineers rather than controlled measurements, so if anything, the reported 50% and 89% numbers likely understate the speedup in change generation alone, because the end-to-end metric includes slower human review and rollout.","Because the success metric is anchored to 50% rather than to code-autonomy rate, this paper's evidence is compatible with LLMs acting as a force multiplier rather than as autonomous agents; the practical claim is about total engineer effort, not automation level.","The hybrid pattern of static localization plus generative editing plus validation likely transfers to other enterprise technical-debt tasks such as API deprecations, framework upgrades, and dead-code removal, where the change sites are discoverable but the edits vary too much for deterministic AST transforms.","The hardest unresolved piece, cleaning entangled test code where multiple flags or conditions interact, points to a concrete next research direction: automated test simplification guided by failing tests and LLM repair."],"forward_implications":["The Google Ads int32-to-int64 migration is on track to meet the 50% end-to-end time-savings target, avoiding an effort estimated at hundreds of engineer-years.","The JUnit3-to-JUnit4 case shows that a small central team can migrate over five thousand test files in three months, with the main bottleneck being human review speed.","The Joda-time-to-java.time case shows that even partially correct AI-generated edits help engineers quickly find all the places that need changing, cutting effort even when the model makes mistakes.","Migrations that previously stalled for years become feasible with a few engineers, because one engineer can generate and coordinate changes across many teams.","The workflow generalizes across migration types: deterministic discovery, LLM edit generation, automated validation, and human review form a reusable pattern for other large codebases."],"supporting_citations":[{"why":"Supplies the earlier blog report on the Ads int32-to-int64 migration that this case study extends.","marker":"[18]"},{"why":"Describes the DIDACT methodology used to train the edit-generation model on Google's internal code and development logs.","marker":"[17]"},{"why":"Provides the Kythe cross-reference information used to locate and cluster change sites in the migrations.","marker":"[16]"},{"why":"Describes the large-scale change infrastructure that splits and routes changelists to code owners for review.","marker":"[26]"},{"why":"Introduces the pass@k evaluation strategy that inspired the parallel prompt-variation approach for generating candidate edits.","marker":"[4]"}],"fun_headline_variants":["Google: LLMs cut code-migration time by 50–89%","LLMs plus AST finish Google migrations stalled for years","Google's AI pipeline halves effort for code migrations","How Google's LLM workflow unblocks stalled migrations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central acceleration claim rests on the assumption that self-reported and expert-estimated time savings are accurate proxies for true end-to-end effort, since the paper's headline figures come from engineers' estimates and extrapolations rather than from controlled before-and-after measurements.","fun_headline_variants_meta":{"raw":{"variants":["Google: LLMs cut code-migration time by 50–89%","LLMs plus AST finish Google migrations stalled for years","Google's AI pipeline halves effort for code migrations","How Google's LLM workflow unblocks stalled migrations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1526,"prompt_tokens":890,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":569}},"tokens_in":506,"tokens_out":636,"duration_ms":6916,"temperature":1.0,"reasoning_tokens":569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:48:47.965328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same migration task on matched codebases with and without the LLM-assisted toolkit, logging actual engineer-hours including discovery, editing, review, and rollout; if the measured difference falls below 50%, the paper's central claim would not hold. Alternatively, an independent audit of the Ads migration's tracked time against the estimated baseline would settle whether the reported savings are real.","supporting_citations":[{"cited_title":"URL: https://kythe.io/","cited_arxiv_id":null,"evidence_quote":"Supplies the earlier blog report on the Ads int32-to-int64 migration that this case study extends."},{"cited_title":"Large sequence models for software development activities","cited_arxiv_id":null,"evidence_quote":"Describes the DIDACT methodology used to train the edit-generation model on Google's internal code and development logs."},{"cited_title":"Winters, T","cited_arxiv_id":null,"evidence_quote":"Describes the large-scale change infrastructure that splits and routes changelists to code owners for review."}],"review_version":1}