{"id":"45f07e2e-3fd1-4dbe-9be8-82ad3ff30784","arxiv_id":"2507.13499","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuned Llama models generate exact-match patches for 68% of internal code review comments, and a safety trial shows AI suggestions slow reviewers unless hidden from them.","lead":"Engineers at Meta built and deployed an AI that turns code review comments into suggested fixes, and measured how it changed the review process. The most notable finding: AI patches shown to reviewers slowed reviews by over 5%, so the tool now shows suggestions only to authors, a design lesson for AI code assistants.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 9.22pp production gain over GPT-4o is identified across non-overlapping periods with no concurrent control; a period confound, not model quality, could explain it. A same-period randomized A/B test or pre-registered covariate analysis would settle it.","rationale":"The reader's weakest assumption correctly identifies the non-overlapping measurement periods as the load-bearing vulnerability. I agree with that diagnosis: the strongest claim is the 9.22pp ActionableToApplied improvement over GPT-4o, and every route to that number is a comparison of Expt. 4 (Feb-Mar 2025) against a prior period (Oct-Dec 2024), with no concurrent control. The authors are transparent about this choice in Section 7.2 and concede confounding in Section 8.2, so the paper is not internally deceptive; but transparency does not convert a monitored rollout into a valid causal estimate. The randomized safety trials are well-designed and support the secondary behavioral claim that showing AI patches to reviewers increased TimeInReview by 5.5% and that collapsing suggestions removed the regression; that part of the paper deserves credit. The concern does not warrant rejection: the deployment story is plausible, the offline results are directionally consistent, and the authors clearly label the production experiments as full rollouts. It does mean the headline magnitude should be presented as a rollout observation, not as a causal model comparison. A same-period randomized A/B test with the same funnel would settle the question directly. I also note the 206-point offline benchmark appears to have no confidence intervals; a paired bootstrap or McNemar test would be a cheap way to confirm the 8.74pp EM gap is not noise, but that is secondary to the production-period confound.","tokens_in":19983,"tokens_out":5974,"duration_ms":65519,"concrete_test":"Run a same-period randomized production experiment: for 3-4 weeks, randomly assign diffs (or review comments) to LargeLSFT versus the exact GPT-4o pipeline used in Expt. 2, keeping the actionability classifier, validity gate, and author-only UX identical. Compare ActionableToApplied and ShownToApplied with confidence intervals. If the concurrent GPT-4o rate is statistically consistent with the historical ~10.5% baseline, the published 9.22pp difference is supported; if the concurrent GPT-4o rate drifts toward 15-19%, the headline is a period artifact. As a fallback, re-analyze Expt. 4 with a pre-registered difference-in-differences or covariate-adjusted model (diff size, language, team, week), but only a concurrent randomized carve-out settles exchangeability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central production claim (Table 5, Expt. 4: ActionableToApplied 19.75%, p < .001; claimed 9.22pp over GPT-4o) is identified from non-overlapping calendar windows. GPT-4o was measured in Expts. 1-2 (Oct 17 - Dec 20, 2024), SmallLSFT in Expt. 3 (Dec 18 - Jan 14), and LargeLSFT in Expt. 4 (Feb 1 - Mar 2, 2025). Section 7.2 explicitly declines a controlled trial, and Section 8.2 concedes unmeasured confounding. A Fisher test on aggregate counts from different periods cannot establish exchangeability: changes in diff mix, seasonality, codebase/tooling, or actionability-classifier behavior over those months are not controlled for. The safety-trial finding (Expts. 1-2) is randomized and credible, but it does not support the model-vs-model adoption comparisons. Thus the 9.22pp headline should be read as a monitored rollout difference, not a causal estimate of LargeLSFT's superiority. The offline EM comparison (Table 3) likewise rests on 206 internal points without confidence intervals, but the production number is the load-bearing evidence for 'at scale' impact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes MetaMateCR, a production system at Meta that generates AI patch suggestions for code review comments. The authors fine-tune Llama-3-70B on internal review-comment/patch pairs (SmallLSFT on 2.9K human-labeled points, LargeLSFT on 64K points that mix human and classifier-labeled data), benchmark these models against GPT-4o on a 206-point internal test set, and then run two randomized safety trials plus two full-population production rollouts. The main reported results are: LargeLSFT achieves 67.96% Exact Match versus 59.22% for GPT-4o (Table 3); the first safety trial shows a statistically significant regression in review time (TimeInReview +5.5%, p=0.029; TimeSpent +6.7%, p<0.001) that disappears in the second trial when AI patches are collapsed for reviewers (Section 7.1, Table 4); and in the final production rollout, ActionableToApplied is 19.75% for LargeLSFT, reported as a 9.22pp improvement over GPT-4o (Table 5). The paper frames the production rollouts as monitored deployments rather than controlled experiments, and Section 8.2 concedes unmeasured confounding.","tokens_in":20198,"tokens_out":6432,"duration_ms":69551,"significance":"If the causal production claims were supported, the paper would be a substantial industrial-scale demonstration of LLM-based review-comment resolution. The randomized safety trials are the strongest part of the work: they provide credible, falsifiable evidence that showing AI patches to reviewers increases reviewer burden, and that collapsing the suggestions removes the regression while preserving author acceptance. This is a practically important and methodologically sound contribution. The offline comparisons are suggestive but not conclusive at n=206, and the production model-to-model differences are identified across non-overlapping calendar windows with no concurrent control, so the headline 9.22pp gain over GPT-4o cannot be assigned to model quality alone. The paper is transparent about its inability to release data and about confounding, which is commendable, but the central production claim needs substantial reframing or additional analysis before the paper can be accepted.","major_comments":[{"comment":"The production comparison of LargeLSFT with GPT-4o is not a controlled comparison: GPT-4o was measured in Expts. 1-2 (Oct 17-Dec 20, 2024), SmallLSFT in Expt. 3 (Dec 18-Jan 14, 2025), and LargeLSFT in Expt. 4 (Feb 1-Mar 2, 2025). Section 7.2 states that no controlled trial was run, and Section 8.2 concedes unmeasured confounding. A Fisher test on aggregate counts from different calendar windows cannot establish that the 9.22pp ActionableToApplied difference is due to model quality rather than to changes in diff mix, seasonality, or tooling. Please either (a) report a same-period randomized or matched comparison, (b) provide an explicit before/during/after analysis with covariate control, or (c) reframe the headline as a monitored rollout difference rather than an improvement causal to the model.","section":"Section 7.2 / Table 5 / Table 1"},{"comment":"The offline benchmark has 206 high-quality points, yet Table 3 reports EM percentages without confidence intervals or significance tests. For the 8.74pp GPT-4o-to-LargeLSFT gap, a two-proportion test on n=206 yields a p-value around 0.007, which is suggestive but should be reported explicitly, together with a bootstrap interval. More importantly, the paper does not state whether the 206 benchmark points are disjoint from the 18K human-labeled and 46K classifier-labeled SFT data points described in Sections 3.3-3.4. Because both sets are mined from the same Meta review corpus using similar pruning, contamination would directly inflate the fine-tuned models' EM. Please state the disjointness explicitly and, if overlap exists, remove it.","section":"Section 4.1 / Table 3"},{"comment":"The production metrics depend on the actionability classifier (a GPT-4o few-shot classifier with F1=0.85 on 240 points) and on the shown gating rule (validity checks or reviewer approval). The paper does not state that the classifier, its prompt, or the gating thresholds were held constant across the four experiments. Since the denominator of ActionableToApplied is 'Actionable,' any drift in classifier behavior between Oct 2024 and Mar 2025 is absorbed into the headline rates. Please state the stability of these components or model/adjust for their changes.","section":"Section 4.2.1 / Equations (1)-(2)"}],"minor_comments":[{"comment":"There are numeric inconsistencies in the claimed percentage-point improvements: Section 1.2 gives the SmallLSFT production improvement over GPT-4o as 1.25pp while Section 7.2 reports 5.6pp, and Section 1.3 gives the LargeLSFT improvement over GPT-4o as 6.6pp while Section 7.2 reports 9.22pp. Please reconcile these numbers.","section":"Sections 1.2-1.3 / Section 7.2"},{"comment":"The text says 'Figure 4 shows the results of the safety trials,' but Figure 4 shows a diff with an accepted code suggestion; the safety-trial results are in Table 4. Additionally, the text reports TimeSpent as 5.5%, p=0.029, but Table 4 lists TimeSpent as 6.7%, p<0.001; please correct the text or the table.","section":"Section 7.1 / Table 4"},{"comment":"The sentence 'We described our experimental methodology in Section 4.2 and outlined the experiments in Figure 1' refers to Figure 1, but Figure 1 shows model patch examples rather than the experiment timeline; the timeline appears in Table 1.","section":"Section 4.2 / Table 1"},{"comment":"The column layout of Table 5 makes it difficult to identify which pairwise comparison each p-value refers to, especially for ShownToApplied (where p<0.001 appears for Expt. 3 and p=0.054 for Expt. 4). Please format the table so that each p-value is explicitly associated with the two experiments being compared.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a credible industrial experience paper, but the headline production claim currently exceeds what the evidence supports. The randomized safety trials are the most robust contribution and should be highlighted. The production model-comparison sections need to be reframed as monitored rollout observations, or supplemented with a concurrent control or covariate analysis; otherwise the 9.22pp claim will not hold up to scrutiny from statistically sophisticated readers. The offline benchmark's overlap with the training set also needs an explicit statement. With those changes, the paper would be a solid TSE contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The randomized safety trial is the real contribution: showing AI patches to reviewers slowed reviews by 5.5% (p=.029), and collapsing them for reviewers removed the regression. That is a non-obvious, actionable result. The headline production numbers — LargeLSFT's 19.75% ActionableToApplied, 9.22pp over GPT-4o — are monitored rollout differences across non-overlapping periods, not causal estimates. The paper says as much in Section 8.2, but the abstract and intro state the comparison without that caveat.\n\nWhat's new: the safety-trial evidence, the author-only UX remedy, and the LLM-classifier-scaled 64k dataset construction. The offline EM gains (68% vs 59%) are plausible and consistent with the production funnel. The paper is also candid that the benchmark is internal and not released, and it doesn't oversell the benchmark as a contribution.\n\nSoft spots. The main one is the production comparison: GPT-4o was measured Oct-Dec 2024, SmallLSFT Dec-Jan, LargeLSFT Feb-Mar 2025. No concurrent control, no covariate adjustment, and the authors explicitly declined a controlled trial. That's a defensible product decision, but it means the 9.22pp should be described as a monitored difference, not a model effect. The 206-point offline benchmark has no confidence intervals; the 9pp gap is suggestive but not tight. Minor: Table 4 and the text disagree on the TimeSpent regression (6.7% vs 5.5%). And ShownToApplied for LargeLSFT vs SmallLSFT is p=.054, which the paper reports as 'no statistically significant change' — fine, but worth being explicit that the improvement there is not established.\n\nOn balance the paper is honest and the core safety finding is solid. The production story is plausible and consistent with the offline results. It deserves a serious referee. I'd suggest the review ask for (a) a concurrent or quasi-experimental comparison for the production claim, or at minimum a caveat in the abstract, and (b) confidence intervals on the offline benchmark.","headline":"A solid industrial case study whose real contribution is the randomized safety finding; the headline production gains are rollout comparisons that shouldn't be read as causal.","tokens_in":20850,"tokens_out":2884,"would_cite":true,"duration_ms":28658,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a Llama model on 64,000 internal review-comment/patch pairs makes it convert 19.75% of actionable code-review comments into accepted patches, beating GPT-4o by 9.22 points, while a user-experience change removes the reviewer…","keywords":["code review","AI-assisted patches","supervised fine-tuning","large language models","production deployment","randomized controlled trial","Llama","GPT-4o"],"falsifier":"Run both models on the same pool of diffs in an interleaved or randomized design for several weeks; if the ActionableToApplied gap between LargeLSFT and GPT-4o falls to zero or reverses, the reported production advantage is an artifact of the non-overlapping measurement periods rather than model quality.","tokens_in":19708,"feed_emoji":"🤖","tokens_out":10539,"duration_ms":100058,"temperature":0.7,"pith_summary":"The paper reports the design and production rollout of MetaMateCR, a system that generates code patches from code-review comments at scale. Its central claim is that a Llama-3-70B model fine-tuned on 64,000 internal review-comment/patch pairs (LargeLSFT) produces better fixes than the general-purpose GPT-4o: 67.96\\% exact-match patches on an internal benchmark versus 59.22\\%, and in production an ActionableToApplied rate of 19.75\\% versus 10.53\\% (a 9.22-point gap). The paper also claims that showing AI patches to reviewers made reviews more than 5\\% slower, and that collapsing the patches for reviewers removed the regression while keeping author acceptance essentially unchanged. A reader should care because the paper traces a complete path from offline fine-tuning to a deployed assistant, and because the safety-trial result identifies a concrete cost of putting AI suggestions in front of reviewers.","feed_headline":"AI code-review fixes hit 19.7% acceptance, beating GPT-4o by 9 points","feed_subtitle":"Llama tuned on 64k review pairs hits 68% exact matches; UI changes remove reviewer slowdowns.","key_machinery":"The central machinery is supervised fine-tuning on \\langle review comment, patch\\rangle pairs mined from the company's own review history. SmallLSFT is built by continual pre-training Llama-3-70B on internal code and then fine-tuning on 2.9K human-annotated pairs; LargeLSFT adds 46K pairs labeled by an 8B Llama classifier trained on 7.5K human labels, for 64K total, and uses a 128K context window. The production funnel is also load-bearing: an in-context-learning actionability classifier decides which comments to attempt (F1 of 0.85), validity checks (build, lint, tests) gate which patches are shown, and the UX hides patches from reviewers while letting authors accept them. Offline exact match is measured against the file content humans actually shipped, so it is a strict lower bound on logical correctness, and the production metrics ActionableToApplied and ShownToApplied track the funnel from comment to applied fix.","core_discovery":"The paper's core discovery is that scaling supervised fine-tuning data from 2.9K human-labeled pairs to 64K pairs, using an 8B Llama classifier to label the additional 46K pairs, moves exact-match patch generation from 63.11\\% (SmallLSFT) to 67.96\\% (LargeLSFT) while GPT-4o sits at 59.22\\% in the same offline setting. In production, the large fine-tuned model converts 19.75\\% of actionable review comments into patches that authors apply, compared with 16.13\\% for SmallLSFT and 10.53\\% for GPT-4o. The paper also establishes that the extra burden created by showing AI patches to reviewers is real and measurable: reviewers spent 5.5\\% longer in review and 6.7\\% more active time when patches were visible, and both regressions disappeared when the patches were collapsed and shown only to authors. The authors read these results as evidence that fine-tuning on internal review practice carries the improvement, and that a safe deployment needs a UX that puts the burden of applying fixes on authors rather than reviewers.","pith_inferences":["The paper leaves open whether the 9.22-point production gain over GPT-4o is causal, because the models ran in non-overlapping calendar periods; an interleaved assignment would test whether diff mix or seasonality explains part of the gap.","Because exact match is a strict lower bound and acceptance is detected partly by heuristics that can miss reworked or manually pasted patches, the true applied rate is probably higher than the reported 19.75 percent.","The collapse-for-reviewers UX lesson likely transfers to other AI coding assistants that surface generated code inside review tools; the measured slowdown suggests reviewer attention is a scarce resource that generated suggestions can tax.","The classifier-labeled data approach could be tested against a purely human-labeled dataset of the same size; the paper does not isolate label quality from quantity, so part of LargeLSFT's gain may come from the 18K human pairs rather than the 46K model-labeled ones."],"forward_implications":["If LargeLSFT's offline lead holds, then scaling SFT data with an LLM classifier is a viable substitute for expensive human labeling once a small high-quality seed exists.","The safety-trial results imply that AI patch suggestions should be shown to authors but collapsed for reviewers in similar review tools; otherwise reviewers spend 5 to 7 percent longer per review.","The monotonic improvements from GPT-4o to SmallLSFT to LargeLSFT in both offline exact match and online ActionableToApplied imply that internal-review fine-tuning, not general model strength alone, drives acceptance.","With the UX fixed, the system runs at scale across four programming languages without measured review-time regressions, so the acceptance gains are not bought by slowing down the review process.","The lack of a significant ShownToApplied gain for LargeLSFT suggests the next lever is increasing the number of patches shown rather than improving patch quality further."],"supporting_citations":[{"why":"Google's production system for resolving code review comments with machine learning; provides the closest deployed comparison against which the paper positions its acceptance and shown-to-applied rates.","marker":"[6]"},{"why":"The Llama 3 herd of models; supplies the 70B base model and the 8B classifier architecture used throughout the fine-tuning pipeline.","marker":"[7]"},{"why":"Jadad and Enkin's randomized controlled trials primer; is the methodological basis for the two safety-trial designs.","marker":"[8]"},{"why":"Shan et al.'s study of code-review nudges at the company; supplies the production experimentation and monitoring infrastructure and the review-time norms the safety metrics build on.","marker":"[9]"},{"why":"The GPT-4o system card; documents the public model used as the zero-shot offline baseline and as the actionability classifier.","marker":"[11]"},{"why":"SWE-bench; is one of the coding benchmarks cited to justify choosing GPT-4o as the state-of-the-art public baseline.","marker":"[14]"},{"why":"Continual pre-training of language models; motivates the internal-code pre-training step that creates iCodeLlama from the Llama base.","marker":"[15]"},{"why":"Language models are few-shot learners; supports the claim that large SFT datasets tolerate some label noise, justifying the use of an LLM classifier to create the additional 46K training pairs.","marker":"[19]"},{"why":"CodeReviewer; gives the earlier pre-trained approach to code-review comment resolution whose roughly 30 percent exact-match result the paper compares with its 67.96 percent.","marker":"[35]"}],"fun_headline_variants":["Meta AI patches beat GPT-4o by 9pp, hit 19.7% acceptance","Llama with 64K review pairs nails 68% exact-match rate","AI patches slowed reviewers; UX fix removed regression","19.7% of AI code-review patches applied at Meta, vs GPT-4o's 10.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline production comparison assumes that the October to December 2024 period used for GPT-4o and the February to March 2025 period used for LargeLSFT are exchangeable, so changes in diff mix, seasonality, codebase, or other tooling do not explain the 9.22-point improvement.","fun_headline_variants_meta":{"raw":{"variants":["Meta AI patches beat GPT-4o by 9pp, hit 19.7% acceptance","Llama with 64K review pairs nails 68% exact-match rate","AI patches slowed reviewers; UX fix removed regression","19.7% of AI code-review patches applied at Meta, vs GPT-4o's 10.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000956,"raw_usage":{"total_tokens":4152,"prompt_tokens":1096,"completion_tokens":3056,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":2965}},"tokens_in":712,"tokens_out":3056,"duration_ms":23309,"temperature":1.0,"reasoning_tokens":2965,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:24:17.516979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run both models on the same pool of diffs in an interleaved or randomized design for several weeks; if the ActionableToApplied gap between LargeLSFT and GPT-4o falls to zero or reverses, the reported production advantage is an artifact of the non-overlapping measurement periods rather than model quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Jadad and Enkin's randomized controlled trials primer; is the methodological basis for the two safety-trial designs."}],"review_version":1}