Pith. sign in

REVIEW 3 major objections 4 minor 40 references

AI-Assisted Fixes to Code Review Comments at Scale

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a Llama model on 64,000 internal review-comment/patch pairs makes it convert 19.75% of actionable code-review comments into accepted patches, beating GPT-4o by 9.22 points, while a user-experience change removes the reviewer…

desk verdict A solid industrial case study whose real contribution is the randomized safety finding; the headline production gains are rollout comparisons that shouldn't be read as causal. read the letter →

arxiv 2507.13499 v1 pith:PGHFHBII submitted 2025-07-17 cs.SE cs.AIcs.PL

Chandra Maddila , Negar Ghorbani , James Saindon , Parth Thakkar , Vijayaraghavan Murali , Rui Abreu , Jingyue Shen , Brian Zhou
show 2 more authors
Nachiappan Nagappan Peter C. Rigby
This is my paper
classification cs.SEcs.AIcs.PL
keywords codereviewAI-assistedpatchessupervisedfine-tuninglargelanguagemodelsproductiondeploymentrandomizedcontrolledtrialLlamaGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports the design and production rollout of MetaMateCR, a system that generates code patches from code-review comments at scale. Its central claim is that a Llama-3-70B model fine-tuned on 64,000 internal review-comment/patch pairs (LargeLSFT) produces better fixes than the general-purpose GPT-4o: 67.96\% exact-match patches on an internal benchmark versus 59.22\%, and in production an ActionableToApplied rate of 19.75\% versus 10.53\% (a 9.22-point gap). The paper also claims that showing AI patches to reviewers made reviews more than 5\% slower, and that collapsing the patches for reviewers removed the regression while keeping author acceptance essentially unchanged. A reader should care because the paper traces a complete path from offline fine-tuning to a deployed assistant, and because the safety-trial result identifies a concrete cost of putting AI suggestions in front of reviewers.

What carries the argument

The central machinery is supervised fine-tuning on \langle review comment, patch\rangle pairs mined from the company's own review history. SmallLSFT is built by continual pre-training Llama-3-70B on internal code and then fine-tuning on 2.9K human-annotated pairs; LargeLSFT adds 46K pairs labeled by an 8B Llama classifier trained on 7.5K human labels, for 64K total, and uses a 128K context window. The production funnel is also load-bearing: an in-context-learning actionability classifier decides which comments to attempt (F1 of 0.85), validity checks (build, lint, tests) gate which patches are shown, and the UX hides patches from reviewers while letting authors accept them. Offline exact match is measured against the file content humans actually shipped, so it is a strict lower bound on logical correctness, and the production metrics ActionableToApplied and ShownToApplied track the funnel from comment to applied fix.

What would settle it

Run both models on the same pool of diffs in an interleaved or randomized design for several weeks; if the ActionableToApplied gap between LargeLSFT and GPT-4o falls to zero or reverses, the reported production advantage is an artifact of the non-overlapping measurement periods rather than model quality.

Watch

Extended reading notes

Core claim

The paper's core discovery is that scaling supervised fine-tuning data from 2.9K human-labeled pairs to 64K pairs, using an 8B Llama classifier to label the additional 46K pairs, moves exact-match patch generation from 63.11\% (SmallLSFT) to 67.96\% (LargeLSFT) while GPT-4o sits at 59.22\% in the same offline setting. In production, the large fine-tuned model converts 19.75\% of actionable review comments into patches that authors apply, compared with 16.13\% for SmallLSFT and 10.53\% for GPT-4o. The paper also establishes that the extra burden created by showing AI patches to reviewers is real and measurable: reviewers spent 5.5\% longer in review and 6.7\% more active time when patches were visible, and both regressions disappeared when the patches were collapsed and shown only to authors. The authors read these results as evidence that fine-tuning on internal review practice carries the improvement, and that a safe deployment needs a UX that puts the burden of applying fixes on authors rather than reviewers.

Load-bearing premise

The headline production comparison assumes that the October to December 2024 period used for GPT-4o and the February to March 2025 period used for LargeLSFT are exchangeable, so changes in diff mix, seasonality, codebase, or other tooling do not explain the 9.22-point improvement.

Editorial extensions

If this is right

  • If LargeLSFT's offline lead holds, then scaling SFT data with an LLM classifier is a viable substitute for expensive human labeling once a small high-quality seed exists.
  • The safety-trial results imply that AI patch suggestions should be shown to authors but collapsed for reviewers in similar review tools; otherwise reviewers spend 5 to 7 percent longer per review.
  • The monotonic improvements from GPT-4o to SmallLSFT to LargeLSFT in both offline exact match and online ActionableToApplied imply that internal-review fine-tuning, not general model strength alone, drives acceptance.
  • With the UX fixed, the system runs at scale across four programming languages without measured review-time regressions, so the acceptance gains are not bought by slowing down the review process.
  • The lack of a significant ShownToApplied gain for LargeLSFT suggests the next lever is increasing the number of patches shown rather than improving patch quality further.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the 9.22-point production gain over GPT-4o is causal, because the models ran in non-overlapping calendar periods; an interleaved assignment would test whether diff mix or seasonality explains part of the gap.
  • Because exact match is a strict lower bound and acceptance is detected partly by heuristics that can miss reworked or manually pasted patches, the true applied rate is probably higher than the reported 19.75 percent.
  • The collapse-for-reviewers UX lesson likely transfers to other AI coding assistants that surface generated code inside review tools; the measured slowdown suggests reviewer attention is a scarce resource that generated suggestions can tax.
  • The classifier-labeled data approach could be tested against a purely human-labeled dataset of the same size; the paper does not isolate label quality from quantity, so part of LargeLSFT's gain may come from the 18K human pairs rather than the 46K model-labeled ones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes MetaMateCR, a production system at Meta that generates AI patch suggestions for code review comments. The authors fine-tune Llama-3-70B on internal review-comment/patch pairs (SmallLSFT on 2.9K human-labeled points, LargeLSFT on 64K points that mix human and classifier-labeled data), benchmark these models against GPT-4o on a 206-point internal test set, and then run two randomized safety trials plus two full-population production rollouts. The main reported results are: LargeLSFT achieves 67.96% Exact Match versus 59.22% for GPT-4o (Table 3); the first safety trial shows a statistically significant regression in review time (TimeInReview +5.5%, p=0.029; TimeSpent +6.7%, p<0.001) that disappears in the second trial when AI patches are collapsed for reviewers (Section 7.1, Table 4); and in the final production rollout, ActionableToApplied is 19.75% for LargeLSFT, reported as a 9.22pp improvement over GPT-4o (Table 5). The paper frames the production rollouts as monitored deployments rather than controlled experiments, and Section 8.2 concedes unmeasured confounding.

Significance. If the causal production claims were supported, the paper would be a substantial industrial-scale demonstration of LLM-based review-comment resolution. The randomized safety trials are the strongest part of the work: they provide credible, falsifiable evidence that showing AI patches to reviewers increases reviewer burden, and that collapsing the suggestions removes the regression while preserving author acceptance. This is a practically important and methodologically sound contribution. The offline comparisons are suggestive but not conclusive at n=206, and the production model-to-model differences are identified across non-overlapping calendar windows with no concurrent control, so the headline 9.22pp gain over GPT-4o cannot be assigned to model quality alone. The paper is transparent about its inability to release data and about confounding, which is commendable, but the central production claim needs substantial reframing or additional analysis before the paper can be accepted.

major comments (3)
  1. [Section 7.2 / Table 5 / Table 1] The production comparison of LargeLSFT with GPT-4o is not a controlled comparison: GPT-4o was measured in Expts. 1-2 (Oct 17-Dec 20, 2024), SmallLSFT in Expt. 3 (Dec 18-Jan 14, 2025), and LargeLSFT in Expt. 4 (Feb 1-Mar 2, 2025). Section 7.2 states that no controlled trial was run, and Section 8.2 concedes unmeasured confounding. A Fisher test on aggregate counts from different calendar windows cannot establish that the 9.22pp ActionableToApplied difference is due to model quality rather than to changes in diff mix, seasonality, or tooling. Please either (a) report a same-period randomized or matched comparison, (b) provide an explicit before/during/after analysis with covariate control, or (c) reframe the headline as a monitored rollout difference rather than an improvement causal to the model.
  2. [Section 4.1 / Table 3] The offline benchmark has 206 high-quality points, yet Table 3 reports EM percentages without confidence intervals or significance tests. For the 8.74pp GPT-4o-to-LargeLSFT gap, a two-proportion test on n=206 yields a p-value around 0.007, which is suggestive but should be reported explicitly, together with a bootstrap interval. More importantly, the paper does not state whether the 206 benchmark points are disjoint from the 18K human-labeled and 46K classifier-labeled SFT data points described in Sections 3.3-3.4. Because both sets are mined from the same Meta review corpus using similar pruning, contamination would directly inflate the fine-tuned models' EM. Please state the disjointness explicitly and, if overlap exists, remove it.
  3. [Section 4.2.1 / Equations (1)-(2)] The production metrics depend on the actionability classifier (a GPT-4o few-shot classifier with F1=0.85 on 240 points) and on the shown gating rule (validity checks or reviewer approval). The paper does not state that the classifier, its prompt, or the gating thresholds were held constant across the four experiments. Since the denominator of ActionableToApplied is 'Actionable,' any drift in classifier behavior between Oct 2024 and Mar 2025 is absorbed into the headline rates. Please state the stability of these components or model/adjust for their changes.
minor comments (4)
  1. [Sections 1.2-1.3 / Section 7.2] There are numeric inconsistencies in the claimed percentage-point improvements: Section 1.2 gives the SmallLSFT production improvement over GPT-4o as 1.25pp while Section 7.2 reports 5.6pp, and Section 1.3 gives the LargeLSFT improvement over GPT-4o as 6.6pp while Section 7.2 reports 9.22pp. Please reconcile these numbers.
  2. [Section 7.1 / Table 4] The text says 'Figure 4 shows the results of the safety trials,' but Figure 4 shows a diff with an accepted code suggestion; the safety-trial results are in Table 4. Additionally, the text reports TimeSpent as 5.5%, p=0.029, but Table 4 lists TimeSpent as 6.7%, p<0.001; please correct the text or the table.
  3. [Section 4.2 / Table 1] The sentence 'We described our experimental methodology in Section 4.2 and outlined the experiments in Figure 1' refers to Figure 1, but Figure 1 shows model patch examples rather than the experiment timeline; the timeline appears in Table 1.
  4. [Table 5] The column layout of Table 5 makes it difficult to identify which pairwise comparison each p-value refers to, especially for ShownToApplied (where p<0.001 appears for Expt. 3 and p=0.054 for Expt. 4). Please format the table so that each p-value is explicitly associated with the two experiments being compared.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the offline and production evaluations are anchored to external human decisions, and the only self-referential elements are background citations.

full rationale

The paper's derivation chain is not circular. The offline EM benchmark is anchored to the final patches that human engineers actually pushed in response to review comments, and the SPG metric is tied to whether the generated patch can be applied and passes validity checks. The production goal metrics, ActionableToApplied and ShownToApplied, are grounded in human author acceptance behavior, not in the model's own outputs. The safety findings come from randomized controlled trials (Expts 1 and 2) that compare against a no-AI control condition, providing an external anchor. The LargeLSFT model is trained on human-labeled and classifier-labeled review-comment/patch pairs, but its evaluation uses a separate 206-point human-annotated benchmark and its production numbers are measured from live acceptance events rather than derived from the training objective. The GPT-4o actionability classifier defines the denominator for all models, so it does not by construction favor any particular patch model; it is a shared pre-filter rather than a fitted target. The main methodological weakness, comparing models across non-overlapping production periods (Expts 3 and 4 vs Expts 1 and 2), is a confounding and generalizability threat, not a circularity, and the paper explicitly concedes unmeasured confounding in Section 8.2. Self-citations to prior internal-code continual pre-training work [17], [18] and to nudging work [9] provide background and motivation but are not load-bearing for the paper's central claims, which rest on the new experiments and measurements reported here. No step reduces an output to its input by definition.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims are empirical measurements of a deployed system, not derivations; this ledger records the hand-chosen gates and unvalidated assumptions the headline rates rest on. The two most consequential choices are the GPT-4o actionability classifier that fixes the denominator of ActionableToApplied, and the acceptance of 46k LLM-classified training points without human re-check. Neither is fitted to the evaluation benchmark, which keeps circularity burden low.

free parameters (2)
  • Shown gating rule (validity checks or reviewer approval) = binary gate: build/lint/test pass OR reviewer approve
    Hand-chosen rule in Section 4.2.1 defining the Shown denominator; directly shapes ShownToApplied and the Expt 4 comparison (p=.054).
  • Actionability classifier (GPT-4o few-shot) = precision 0.86, recall 0.85 on 240 labeled points
    Defines the Actionable denominator of ActionableToApplied (Section 4.2.1); errors or drift rescale the headline 19.75% rate, though the same classifier is applied across models.
assumptions (5)
  • domain assumption The internal 206-point human-curated benchmark is representative of production code review comments at Meta.
    Used for all offline EM/SPG numbers (Section 4.1). The paper argues OSS benchmarks miss internal patterns but provides no evidence that 206 points span the production comment distribution.
  • domain assumption The human-final patch used as Exact Match ground truth is the correct standard; functionally equivalent patches count as misses.
    Section 4.1 calls EM 'extremely restrictive' and a lower bound, so this assumption understates model accuracy but is an acknowledged design choice.
  • ad hoc to paper The 46k LLM-classified data points are accurate enough that training on the combined 64k set improves the model.
    Section 3.4: the 8B classifier shows precision 0.88/recall 0.75 on 700 held-out points, but the 46k 'good' labels are never human-verified; the paper asserts large dataset size compensates for label noise, citing [19].
  • domain assumption The randomized assignment in Expts 1 and 2 isolates the effect of showing AI patches on review-time metrics.
    Standard RCT assumption invoked in Section 4.2 and Section 7.1; plausible for the two safety trials, but Expts 3 and 4 do not include a control condition.
  • domain assumption The safety metrics TimeInReview, TimeSpent, and WallClock capture the relevant side effects of AI patch suggestions.
    Defined in Section 4.2.2; Section 8.2 concedes unmeasured confounders such as code quality and team dynamics could affect these metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI-Assisted Fixes to Code Review Comments at Scale." pith.science (2026). https://pith.science/paper/PGHFHBII

@misc{pith2026250713499,
  author       = {Pith},
  title        = {Pith review of: AI-Assisted Fixes to Code Review Comments at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGHFHBII}},
  note         = {Machine review of arXiv:2507.13499}
}
read the original abstract

Aim. There are 10s of thousands of code review comments each week at Meta. We developed Metamate for Code Review (MetaMateCR) that provides AI-assisted fixes for reviewer comments in production at scale. Method. We developed an internal benchmark of 64k <review comment, patch> data points to fine-tune Llama models. Once our models achieve reasonable offline results, we roll them into production. To ensure that our AI-assisted fixes do not negatively impact the time it takes to do code reviews, we conduct randomized controlled safety trials as well as full production experiments. Offline Results. As a baseline, we compare GPT-4o to our small and large Llama models. In offline results, our LargeLSFT model creates an exact match patch 68% of the time outperforming GPT-4o by 9 percentage points (pp). The internal models also use more modern Hack functions when compared to the PHP functions suggested by GPT-4o. Safety Trial. When we roll MetaMateCR into production in a safety trial that compares no AI patches with AI patch suggestions, we see a large regression with reviewers taking over 5% longer to conduct reviews. After investigation, we modify the UX to only show authors the AI patches, and see no regressions in the time for reviews. Production. When we roll LargeLSFT into production, we see an ActionableToApplied rate of 19.7%, which is a 9.2pp improvement over GPT-4o. Our results illustrate the importance of safety trials in ensuring that AI does not inadvertently slow down engineers, and a successful review comment to AI patch product running at scale.

Figures

Figures reproduced from arXiv: 2507.13499 by the authors.

Figure 2
Figure 2. The funnel for MetaMateCR. The universe of code review comments at meta is filtered through an actionability classifier. If it is actionable, then validity checks are run, i.e. does the AI patch build and pass linters and tests. If yes, then it is shown in the UX and it can be accepted, i.e. assisted, otherwise it is discarded. set of 240 data points that are manually labeled by human annotators as either actionable… view at source ↗
Figure 3
Figure 3. The MetaMateCR system architecture 6 THE MetaMateCR SYSTEM We provide an overview of the product experience, various components involved in making the MetaMateCR system a reality, and the AI-assisted comment resolution workflow. This helps in developing a deeper understanding of the user experience, workflows, and contextualizing different problems we need to solve to operationalize such a system at scale. 6.1 Syste… view at source ↗
Figure 4
Figure 4. Diff with an accepted code suggestion [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: The AI suggestion from the code review comment [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Dialog to review & apply AI code suggestions [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Reviewer approval of an AI suggestion TABLE 5: We compare the ActionableToApplied and ShownToApplied rates for each experiment and report statistical comparisons between the progressive models. Experiment Expt. 1 Expt. 2 Expt. 3 Expt 4. Model GPT-4o —vs— GPT-4o —vs— Sm…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 16 canonical work pages

  1. [1]

    Design and Code Inspections to Reduce Errors in Program Development,

    M. E. Fagan, “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems Journal, vol. 15, no. 3, pp. 182– 211, 1976

    work page 1976

  2. [2]

    Understanding the sources of variation in software inspections,

    A. Porter, H. Siy, A. Mockus, and L. Votta, “Understanding the sources of variation in software inspections,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 7, no. 1, pp. 41– 79, 1998

    work page 1998

  3. [3]

    Convergent contemporary software peer review practices,

    P . C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013, pp. 202–212

    work page 2013

  4. [4]

    Automated code review in practice,

    U. Cihan, V . Haratian, A. ˙Ic ¸¨oz, M. K. G ¨ul, ¨Omercan Devran, E. F. Bayendur, B. M. U c ¸ar, and E. T ¨uz ¨un, “Automated code review in practice,” 2024. [Online]. Available: https://arxiv.org/abs/2412.18531

    arXiv 2024

  5. [5]

    Au- totransform: Automated code transformation to support modern code review process,

    P . Thongtanunam, C. Pornprasit, and C. Tantithamthavorn, “Au- totransform: Automated code transformation to support modern code review process,” in 2022 IEEE/ACM 44th International Confer- ence on Software Engineering (ICSE), 2022, pp. 237–248

    work page 2022

  6. [6]

    Resolving code review comments with machine learning,

    A. Froemmgen, J. Austin, P . Choy, N. Ghelani, L. Kharatyan, G. Surita, E. Khrapko, P . Lamblin, P .-A. Manzagol, M. Revaj, M. Tabachnyk, D. Tarlow, K. Villela, D. Zheng, S. Chandra, and P . Maniatis, “Resolving code review comments with machine learning,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in...

    arXiv 2024

  7. [7]

    The llama 3 herd of models,

    G. etal, “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    arXiv 2024

  8. [8]

    A. R. Jadad and M. W. Enkin, Randomized controlled trials: questions, answers and musings. John Wiley & Sons, 2007

    work page 2007

Show all 40 references
  1. [10]

    Understanding why we cannot model how long a code review will take: An industrial case study,

    L. Chen, P . C. Rigby, and N. Nagappan, “Understanding why we cannot model how long a code review will take: An industrial case study,” ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3540250.3558945

    2022

  2. [11]

    Gpt-4o system card,

    H. et al, “Gpt-4o system card,” 2024. [Online]. Available: https://arxiv.org/abs/2410.21276

    2024 arXiv

  3. [12]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    2021 arXiv

  4. [13]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P . de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P . Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter...

    2021 arXiv

  5. [14]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” 2024. [Online]. Available: https://arxiv.org/abs/2310.06770 TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. Y , JUL Y 2025 14

    2024 arXiv

  6. [15]

    Continual pre-training of language models,

    Z. Ke, Y. Shao, H. Lin, T. Konishi, G. Kim, and B. Liu, “Continual pre-training of language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.03241

    2023 arXiv

  7. [16]

    The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities,

    V . B. Parthasarathy, A. Zafar, A. Khan, and A. Shahid, “The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities,” 2024. [Online]. Available: https://arxi...

    2024 arXiv

  8. [17]

    Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation,

    V . Murali, C. Maddila, I. Ahmad, M. Bolin, D. Cheng, N. Ghorbani, R. Fernandez, N. Nagappan, and P . C. Rigby, “Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation,” Proc. ACM Softw. Eng. , vol. 1, no. FSE, Jul. 2024. [Online]. Available:...

    2024 doi

  9. [18]

    Multi-line ai-assisted code authoring,

    O. Dunay, D. Cheng, A. Tait, P . Thakkar, P . C. Rigby, A. Chiu, I. Ahmad, A. Ganesan, C. Maddila, V . Murali, A. Tayyebi, and N. Nagappan, “Multi-line ai-assisted code authoring,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software...

    2024

  10. [19]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B...

    2020 arXiv

  11. [20]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” 2024. [Online]. Available: https://arxiv.org/abs/2403.07974

    2024 arXiv

  12. [21]

    Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents,

    M. S. Rashid, C. Bock, Y. Zhuang, A. Buchholz, T. Esler, S. Valentin, L. Franceschi, M. Wistuba, P . T. Sivaprasad, W. J. Kim, A. Deoras, G. Zappella, and L. Callot, “Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents,” 2025. [Online]. A...

    2025 arXiv

  13. [22]

    Repairbench: Leaderboard of frontier models for program repair,

    A. Silva and M. Monperrus, “Repairbench: Leaderboard of frontier models for program repair,” 2024. [Online]. Available: https://arxiv.org/abs/2409.18952

    2024

  14. [23]

    Octopack: Instruction tuning code large language models,

    N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang, L. von Werra, and S. Longpre, “Octopack: Instruction tuning code large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2308.07124

    2024 arXiv

  15. [24]

    A survey on in-context learning,

    Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu, B. Chang, X. Sun, L. Li, and Z. Sui, “A survey on in-context learning,” 2024. [Online]. Available: https://arxiv.org/abs/2301.00234

    2024 arXiv

  16. [25]

    A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,

    Y. Song, T. Wang, S. K. Mondal, and J. P . Sahoo, “A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,” 2022. [Online]. Available: https://arxiv.org/abs/2205.06743

    2022 arXiv

  17. [26]

    An improved inspection technique,

    J. C. Knight and E. A. Myers, “An improved inspection technique,” Communications of the ACM, vol. 36, no. 11, pp. 50–61, 1993

    1993

  18. [27]

    Does every inspection need a meeting?

    L. G. Votta, “Does every inspection need a meeting?” SIGSOFT Softw. Eng. Notes , vol. 18, no. 5, p. 107–114, Dec. 1993. [Online]. Available: https://doi.org/10.1145/167049.167070

    1993

  19. [28]

    Reducing inspection interval in large-scale software development,

    D. Perry, A. Porter, M. Wade, L. Votta, and J. Perpich, “Reducing inspection interval in large-scale software development,” IEEE Transactions on Software Engineering, vol. 28, no. 7, pp. 695–705, 2002

    2002

  20. [29]

    Open source software peer review practices,

    P . Rigby, D. German, and M.-A. Storey, “Open source software peer review practices,” in 2008 ACM/IEEE 30th International Conference on Software Engineering, 2008, pp. 541–550

    2008

  21. [30]

    Peer review on open-source software projects: Parameters, statistical models, and theory,

    P . C. Rigby, D. M. German, L. Cowen, and M.-A. Storey, “Peer review on open-source software projects: Parameters, statistical models, and theory,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 23, no. 4, p. 35, 2014

    2014

  22. [31]

    Nudge: Accelerating overdue pull requests toward completion,

    C. Maddila, S. S. Upadrasta, C. Bansal, N. Nagappan, G. Gousios, and A. van Deursen, “Nudge: Accelerating overdue pull requests toward completion,” ACM Trans. Softw. Eng. Methodol., vol. 32, no. 2, Mar. 2023. [Online]. Available: https://doi.org/10.1145/3544791

    2023 doi

  23. [32]

    Whodo: automating reviewer suggestions at scale,

    S. Asthana, R. Kumar, R. Bhagwan, C. Bird, C. Bansal, C. Maddila, S. Mehta, and B. Ashok, “Whodo: automating reviewer suggestions at scale,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softwa...

    2019

  24. [33]

    Factoring exper- tise, workload, and turnover into code review recommendation,

    F. Hajari, S. Malmir, E. Mirsaeedi, and P . C. Rigby, “Factoring exper- tise, workload, and turnover into code review recommendation,” IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 884–899, 2024

    2024

  25. [34]

    Commentfinder: a simpler, faster, more accurate code review comments recommendation,

    Y. Hong, C. Tantithamthavorn, P . Thongtanunam, and A. Aleti, “Commentfinder: a simpler, faster, more accurate code review comments recommendation,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engin...

    2022

  26. [35]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan, “Automating code review activities by large-scale pre-training,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on ...

    2022

  27. [36]

    Ai- assisted sql authoring at industry scale,

    C. Maddila, N. Ghorbani, K. Jabre, V . Murali, E. Kim, P . Thakkar, N. P . Laptev, O. Harman, D. Hsu, R. Abreu, and P . C. Rigby, “Ai- assisted sql authoring at industry scale,” 2024. [Online]. Available: https://arxiv.org/abs/2407.13280

    2024 arXiv

  28. [37]

    Moving faster and reducing risk: Using llms in release deployment,

    R. Abreu, V . Murali, P . C. Rigby, C. Maddila, W. Sun, J. Ge, K. Chinniah, A. Mockus, M. Mehta, and N. Nagappan, “Moving faster and reducing risk: Using llms in release deployment,” 2024. [Online]. Available: https://arxiv.org/abs/2410.06351

    2024 arXiv

  29. [38]

    The impact of ai on developer productivity: Evidence from github copilot,

    S. Peng, E. Kalliamvakou, P . Cihon, and M. Demirer, “The impact of ai on developer productivity: Evidence from github copilot,” in Arxiv, 2023. [Online]. Available: https://arxiv.org/abs/2302.06590

    2023 arXiv

  30. [39]

    The effects of generative ai on high skilled work: Evidence from three field experiments with software developers,

    Z. K. Cui, M. Demirer, S. Jaffe, L. Musolff, S. Peng, and T. Salz, “The effects of generative ai on high skilled work: Evidence from three field experiments with software developers,” Available at SSRN 4945566, 2024

    2024

  31. [40]

    How much does ai impact development speed? an enterprise- based randomized controlled trial,

    E. Paradis, K. Grey, Q. Madison, D. Nam, A. Macvean, V . Meimand, N. Zhang, B. Ferrari-Church, and S. Chandra, “How much does ai impact development speed? an enterprise- based randomized controlled trial,” 2024. [Online]. Available: https://arxiv.org/abs/2410.12944

    2024 arXiv

  32. [41]

    If it’s what i wanted that’s great but if it’s not i just wasted time

    A. Murillo, A. Macvean, I. Chu, Q. Madison, and S. D’Angelo, “If it’s what i wanted that’s great but if it’s not i just wasted time”: Unpacking the perceived costs/benefits of ml enhanced developer tooling,” in Proc. Deep Learning 4 Code Workshop, 2023, pp. 1–9

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.