Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

BitsAI-CR: Automated Code Review via LLM in Practice

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM code-review pipeline attains 75% precision in production by adding a second validating model and a data-driven rule-pruning loop.

desk verdict A real industrial deployment with a genuinely useful new metric, but the quantitative evidence is too self-referential and internally inconsistent to support the convergence claims as written. read the letter →

arxiv 2501.15134 v1 pith:QNXIXUW7 submitted 2025-01-25 cs.SE

Tao Sun , Jian Xu , Yuanpeng Li , Zhao Yan , Ge Zhang , Lintao Xie , Lu Geng , Zheng Wang
show 4 more authors
Yueyan Chen Qin Lin Wenbo Duan Kaixin Sui
This is my paper · ORCID
classification cs.SE
keywords codereviewlargelanguagemodelsautomateddataflywheeloutdatedratetaxonomyRuleCheckerFilter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BitsAI-CR argues that LLM code review fails in practice not because models cannot spot issues but because too many comments are wrong or superfluous, so developers stop reading them. The paper's solution is a two-stage pipeline in which a fine-tuned LLM (RuleChecker) proposes issues from a 219-rule taxonomy and a second fine-tuned LLM (ReviewFilter) validates each candidate before it is shown. Around this pipeline, a data flywheel measures precision and an 'Outdated Rate'—the fraction of flagged code ranges that are later modified—and drops rules that are precise but rarely acted on. The reported result is 75% precision at peak and a 26.7% Outdated Rate for Go, with over 12,000 weekly active users, which the authors present as evidence that automated review can be both accurate enough and useful enough for daily industrial use.

What carries the argument

The load-bearing mechanism is the two-stage pipeline: RuleChecker, a fine-tuned LLM (using Low-Rank Adaptation, a parameter-efficient fine-tuning method) that proposes candidate issues against the 219-rule taxonomy, followed by ReviewFilter, a second fine-tuned LLM that returns a binary keep/reject verdict on each candidate. The paper's design choice for ReviewFilter is the 'Conclusion-First' reasoning pattern, where the verdict token precedes the explanation, keeping inference at 1.7 seconds per sample while reaching 77.09% precision in the reasoning-pattern ablation. The companion mechanism is the Outdated Rate, defined as the percentage of comments whose flagged code range is modified in subsequent commits; combined with a precision target near 65% and an Outdated Rate target near 25%, it drives weekly decisions to retain or decommission rules and feeds the retraining loop.

What would settle it

A controlled study in which comments are randomly withheld from half of the merge requests could settle the question: if flagged code ranges in the withheld half change just as often as in the reviewed half, the Outdated Rate is measuring background edit activity rather than comment adoption, and the flywheel's pruning signal disappears.

Watch

Extended reading notes

Core claim

The paper's central claim is that a generate-then-validate design, anchored in a taxonomy of 219 review rules and steered by an automated acceptance metric, closes the gap between technically correct and practically useful automated code review. In the offline evaluation, the RuleChecker alone reaches 57.03% precision once trained on the taxonomy, while the same model trained on unclassified human review data reaches only 16.83%, and strong open models sit near 10%. Adding ReviewFilter lifts the taxonomy-guided model to 65.59% offline, and in production the combined pipeline peaks at 75.0% precision while sacrificing roughly a quarter of recall. The Outdated Rate is the other half of the claim: it counts a comment as outdated if any line in its flagged range is modified in a later commit, and because it is automated it makes weekly flywheel decisions possible. Monitoring that rate and manual precision annotations together led the operators to remove low-adoption rules, and over 18 weeks the Go Outdated Rate rose from about 15% to 26.7%, moving toward the 35–46% range measured for human reviewers.

Load-bearing premise

The load-bearing premise is that the Outdated Rate—any line in a comment's flagged range being modified in a later commit—measures whether the comment caused a change; the paper itself concedes this is not proven, and both its rule-pruning decisions and its claim of converging toward human-level impact rely on that proxy.

Editorial extensions

If this is right

  • A deployment that reports 75% precision and a 26.7% Go Outdated Rate provides a working counterexample to the assumption that LLM review comments are too noisy for daily industrial use.
  • The taxonomy of 219 rules turns an otherwise unstructured fine-tuning target into a measurable set of categories, which is what makes both targeted data collection and per-rule decommissioning possible.
  • Because the Outdated Rate is automated, the same flywheel can be run weekly without the manual annotation bottleneck that stops most precision evaluations from scaling.
  • The ReviewFilter ablation attributes a large precision gain (57.03% to 65.59% offline; to about 75% online) to the second stage, implying that a validator model can rescue a generator that continues to hallucinate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same generate-then-validate recipe should transfer to other domains where false positives are costly, such as security linting or automated documentation checks, though the paper does not test this.
  • The Outdated Rate conflates 'addressed the comment' with 'the flagged line changed for any reason'; an extension would measure time-to-change and whether the follow-up commit references the comment, making the proxy harder to game.
  • Because the taxonomy is built from function-level and within-diff signals and the paper lists cross-file review as future work, the flywheel may systematically under-invest in rules whose payoff appears only across files.
  • The human Outdated Rate baseline (35–46% in Go) is itself measured in a workflow where automated comments already exist; if adoption of the tool changes how developers edit, the baseline may drift and should be re-measured periodically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. BitsAI-CR is an industrial LLM-based code review system deployed at ByteDance. The paper proposes a two-stage architecture—RuleChecker for issue detection and ReviewFilter for precision verification—built on a taxonomy of 219 review rules, plus a 'data flywheel' that uses user feedback, manual precision annotation, and a new Outdated Rate metric to continuously add or remove rules. The abstract claims 75.0% precision in comment generation, a Go-language Outdated Rate of 26.7%, and deployment to over 12,000 weekly active users. The evaluation combines offline comparisons against open baselines, an ablation of ReviewFilter, weekly online precision and Outdated Rate trends over 18 weeks, a user survey (N=137), expert interviews (N=12), and retention data.

Significance. If the empirical claims are sound, this is a valuable large-scale industry case study: the taxonomy-driven data flywheel, the two-stage filtering architecture, and the 12k-WAU deployment are distinct from prior academic code-review systems. The paper also honestly acknowledges in Section 3.4 that Outdated Rate does not prove that comments caused code changes. However, the evaluation has several load-bearing weaknesses: inconsistent headline numbers, a test set drawn from the same taxonomy used for training, an LLM judge from the same model family as the system, and a data flywheel that removes low-scoring rules on the same metric later used to claim improvement. These issues prevent the current evidence from establishing the paper's central quantitative claims, though the deployment and qualitative feedback are genuine strengths.

major comments (5)
  1. [Abstract, §4.2, Fig. 6] The abstract claims '75.0% precision in review comment generation', but the offline evaluation in Table 2 reports 65.59% overall precision for BitsAI-CR with ReviewFilter, and Figure 6 shows 75.0% only as a week-17 peak in the online trend. The paper never reconciles these numbers, so the headline precision figure is ambiguous and the offline evaluation does not support it directly.
  2. [§4.2, Table 1] Section 4.2 states that the offline evaluation set consists of 1397 cases 'drawn from the taxonomy of review rules, as categorized in Table 1'. Since the taxonomy-guided model is trained on data constructed according to the same taxonomy, the comparison between BitsAI-CR and BitsAI-CR w/o Taxonomy partly measures how well the test set matches the training distribution rather than general review quality. An independent test set, or a leakage/overlap analysis, is required to support the claim that the taxonomy is the cause of the precision gain.
  3. [§4.1, §4.2] The LLM-as-a-judge evaluation uses Doubao-Pro-32K-0828 (Section 4.2), which is the same model family as the base model fine-tuned for both RuleChecker and ReviewFilter (Section 4.1). This creates a same-family bias risk: the judge may share systematic blind spots or preferences with the system under test. The paper should report agreement between this judge and human annotators on a sample, or use a judge from a different model family, to establish that the reported precision is not inflated.
  4. [§3.4, §3.5.3, Fig. 7] The data flywheel removes review rules with consistently low Outdated Rate and precision (Section 3.5.3), and then Figure 7 shows the aggregate Outdated Rate rising to 26.7%, which is interpreted as convergence toward the human rate of 35–46%. Removing low-scoring rules mechanically raises the aggregate rate even if no individual rule improves. Additionally, Eq. (2) counts a comment as 'outdated' whenever any line in its flagged range is later modified, which, as Section 3.4 concedes, does not prove that the modification was caused by the comment. Because the removal decisions are optimized on this same proxy, the flywheel improvement and convergence claims require an evaluation that is not optimized on the same metric, such as a held-out set of rules or human-labeled acceptance of comments.
  5. [§4.2, §5] The ablation text in Section 4.2 says ReviewFilter increases overall precision 'from 54.50% to 67.12%', but Table 2 reports 57.03% (Only RuleChecker) to 65.59% (With ReviewFilter) for the same comparison. Section 5 additionally states precision increased 'from 60% to 75%' in a different context, and earlier says 'from 30.92% to 65.59% with ReviewFilter in the Go language'. These numbers are not mutually consistent and must be reconciled with the exact dataset, metric, and language being described.
minor comments (6)
  1. [§3.4, Tables 2–3] Recall is reported in Tables 2 and 3 but never formally defined; the paper should define recall (presumably the fraction of ground-truth issues correctly commented) and describe how ground truth was constructed for the offline set.
  2. [§3.5.3] The text contains a typo: 'RuleFiler' should be 'ReviewFilter'.
  3. [§1] The introduction has ungrammatical phrases ('metring the Outdated Rate' and 'inadequate or lack their practical impact comprehensive evaluation') that should be corrected.
  4. [Fig. 7] The 'Human Outdated Rate' baseline is described only as 'how often code flagged by human reviewers gets modified'; the definition, measurement window, and whether it uses the same Eq. (2) should be stated explicitly.
  5. [§4.2] The offline dataset description says 767 samples 'violate' and 630 'follow' best practices, but the notion of violation and who labeled the samples is not defined; specify the annotation process and inter-annotator agreement.
  6. [§7] The conclusion heading reads 'Feature Work'; it should be 'Future Work'.

Circularity Check

2 steps flagged · score 6.0 of 10

The flywheel's headline improvement is partly self-measured: it is optimized on Outdated Rate and then reports the rise in that same metric as success, while precision is judged by the same model family and taxonomy used to build the system.

  1. fitted input called prediction [Section 3.4 (Eq. 2), Section 3.5.3, Section 4.3 (Figure 7)]
    "Outdated Rate = |{c∈C_seen ∧ isOutdated(c)}| / |C_seen| × 100% ... Starting from week 14, we begin optimizing BitsAI-CR by removing underperforming review rules based on Outdated Rate and precision metrics, which leads to a gradual increase in the Outdated Rate, ultimately reaching a peak of 26.7% by week 18."

    The data flywheel is optimized on the same metric it later reports: Section 3.5.3 decommissions rules with consistently low Outdated Rate, and removing low-rate rules mathematically raises the aggregate rate defined in Eq. (2), even if no individual rule's comments are accepted more often. Section 4.3 then interprets this mechanically increased number as 'gradual convergence ... toward human-level performance' and as evidence that the flywheel works. The paper also concedes that Outdated Rate 'doesn't definitively prove that changes were made in direct response to BitsAI-CR's comments,' so the metric is too noisy and too entangled with the optimization to bear the central improvement claim.

  2. other [Section 4.1 and Section 4.2]
    "We utilize Doubao-Pro-32K-0828, ByteDance’s developed LLM, which ensures compliance with our security policies while maintaining high-performance standards. ... We employ a fine-tuning approach using the Low-Rank Adaptation (LoRA) on the Doubao-Pro-32K-0828 for both RuleChecker and ReviewFilter. ... Following the LLM-as-a-judge methodology, we employ Doubao-Pro-32K-0828 to evaluate automatically our business code dataset while preserving data confidentiality. The review comment is deemed correct only if the model determines it aligns with the ground truth."

    The headline precision figures are measured by Doubao-Pro-32K-0828, the same base model from which RuleChecker and ReviewFilter are fine-tuned, on offline cases that the paper says are 'drawn from the taxonomy of review rules' used to construct the training data. The judge therefore shares the system's priors and categories, making the precision score a self-consistency check rather than an independent external benchmark. This does not nullify the comparison against Qwen and DeepSeek baselines, but it weakens the 75.0% claim as an objective validation.

full rationale

The deployment evidence (12,000 WAU, retention, N=137 survey, expert interviews) is independent and not circular, and the human Outdated Rate baseline plus open-model comparisons provide some external anchoring. However, the two quantitative success claims are not self-contained: precision is scored by a judge from the same model family on taxonomy-derived cases, and the Outdated Rate convergence is partly a selection artifact of removing low-rate rules on the very metric being reported. These are central claims, so the partial circularity warrants a score of 6 rather than a lower score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central effectiveness claims depend on measures (Outdated Rate, LLM-judge precision) and rule-selection thresholds that are defined internally by the authors, plus the assumption that the taxonomy and datasets are unbiased. No external benchmark or released artifact anchors the numbers.

free parameters (3)
  • Rule retention thresholds = Outdated Rate ~25% (5% band), precision ~65% (5% band)
    Used in Section 3.5.3 to decide which review rules to keep; they determine which rules remain active and therefore shape the reported Outdated Rate and precision in Figure 7.
  • Context expansion bounds = 4x diff size for function extension, otherwise 3x
    Context Preparation parameters in Section 3.3; chosen by hand and affect model input, but less directly tied to the headline metrics.
  • Sampling cap for daily manual annotation = 10%
    Section 3.5.3 states manual sampling 'generally does not exceed 10%'; the specific rate is a practical choice, not fitted, but could affect precision measurement representativeness.
assumptions (5)
  • domain assumption The LLM-as-a-judge (Doubao-Pro-32K-0828) accurately determines whether a review comment is correct relative to ground truth.
    Used to compute all offline precision values in Table 2. No human agreement or calibration against the judge is reported, and the judge is from the same model family used to build the system.
  • domain assumption The offline test dataset (1397 cases) is representative of production code review and unbiased.
    The cases are sampled from the same taxonomy of review rules used to structure the system's training data, which can inflate measured precision.
  • domain assumption Subsequent modification of a flagged line is a valid proxy for developer acceptance of a comment (Outdated Rate).
    The paper itself states Outdated Rate 'doesn't definitively prove that changes were made in direct response' (Section 3.4). Yet this metric is used to guide rule removal and to claim improvement.
  • domain assumption The 'Human Outdated Rate' at ByteDance is computed in the same way as BitsAI-CR's Outdated Rate, making the comparison valid.
    No details are given for how human reviewer comments are selected or how outdated status is determined for them; baseline values are stated without methodology.
  • standard math Supervised fine-tuning with LoRA produces a model that generalizes to new code diffs in production.
    General ML assumption; standard, but not verified independently here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BitsAI-CR: Automated Code Review via LLM in Practice." pith.science (2026). https://pith.science/paper/QNXIXUW7

@misc{pith2026250115134,
  author       = {Pith},
  title        = {Pith review of: BitsAI-CR: Automated Code Review via LLM in Practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNXIXUW7}},
  note         = {Machine review of arXiv:2501.15134}
}
read the original abstract

Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.

Figures

Figures reproduced from arXiv: 2501.15134 by the authors.

Figure 1
Figure 1. The overview of BitsAI-CR framework for enhancing code review. practices, data collection mechanisms, and continuous op￾timization through a data-driven feedback loop. • A two-stage approach combining RuleChecker for issue detection and ReviewFilter for verification, demonstrating substantial improvements in review effectiveness. • The introduction of an Outdated Rate metric that addresses two key limitations of tra… view at source ↗
Figure 2
Figure 2. An Example of Input and Output in BitsAI-CR a Data Flywheel mechanism for continuous improvement. The Re￾view Comment Generation pipeline processes code reviews through four steps: (1) Context Preparation that structures the input for anal￾ysis, (2) RuleChecker that identifies potential issues using an LLM, (3) ReviewFilter that validates the detected issues, and (4) Com￾ment Aggregation that consolidates similar fe… view at source ↗
Figure 3
Figure 3. A Correct but Superfluous Comment [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: The MR interface shows the “outdated” status of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Weekly Progression of BitsAI-CR Precision reasoning patterns reveals that the Reasoning-First pattern attains a superior recall of 81.80%, albeit at the cost of a noticeably pro￾longed inference time of 31s/sample, rendering it less feasible for deployment in productio…
Figure 8
Figure 8. Figure 8: Weekly User Retention Rate of BitsAI-CR from these interviews, categorized by usage patterns, quality as￾sessment, performance considerations, and feature requests. 4.4 Large-Scale Industrial Deployment Before full-scale implementation, we conduct canary testing to val…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation

    cs.SE 2025-09 conditional novelty 6.0 of 10

    SWR-Bench is a PR-centric code review benchmark with objective LLM scoring; current ACR tools reach at best 19.4% F1, and multi-review aggregation yields relative F1 gains up to 43.7%.

  2. BitsAI-Fix: LLM-Driven Approach for Automated Lint Error Resolution in Practice

    cs.SE 2025-08 conditional novelty 6.0 of 10

    A 32B LLM trained with progressive reinforcement learning on cold-start and user-feedback data fixes Go lint errors at 84.68% accuracy with 1.72% redundancy on a private ByteDance test set.

  3. P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark

    cs.CL 2025-05 conditional novelty 6.0 of 10

    P2P is a multi-agent framework that automatically generates HTML-rendered academic posters from papers, backed by a 30k instruction dataset and a 121-pair evaluation benchmark.

Reference graph

Works this paper leans on

56 extracted references · 18 canonical work pages · cited by 3 Pith papers

  1. [1]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel

  2. [2]

    Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and chal- lenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 712–721

    work page 2013

  3. [3]

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. McEval: Massively Multi- lingual Code Evaluation. arXiv preprint arXiv:2406.07436 (2024)

    arXiv 2024

  4. [4]

    Qiuyuan Chen, Dezhen Kong, Lingfeng Bao, Chenxing Sun, Xin Xia, and Shan- ping Li. 2022. Code Reviewer Recommendation in Tencent: Practice, Challenge, and Direction*. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (Pittsburgh, PA, USA, 2022-05). IEEE, 115–124. doi:10.1109/ICSE-SEIP55303...

    arXiv 2022

  5. [5]

    A GPT-based Code Review System for Programming Language Learning

    Lee Dong-Kyu. 2024.A GPT-based Code Review System for Programming Language Learning. arXiv.org. https://arxiv.org/abs/2407.04722v1

    work page Pith review arXiv 2024

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

  7. [7]

    Sigrid Eldh. 2024. Code Review Evolution. IEEE Software 41, 5 (2024), 4–8

    work page 2024

  8. [8]

    Pär Emanuelsson and Ulf Nilsson. 2008. A comparative study of industrial static analysis tools. Electronic notes in theoretical computer science 217 (2008), 5–21

    work page 2008

Show all 56 references
  1. [9]

    Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, and Shanping Li. 2024. Exploring the Capabilities of LLMs for Code Change Related Tasks . arXiv.org. https://arxiv.org/abs/2407.02824v1

    2024 arXiv

  2. [10]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP ...

    2020 doi

  3. [11]

    Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2024. Exploring the Potential of ChatGPT in Automated Code Re- finement: An Empirical Study. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (New York,...

    2024

  4. [12]

    Md Asif Haider, Ayesha Binte Mostofa, Sk Sabit Bin Mosaddek, Anindya Iqbal, and Toufique Ahmed. 2024. Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation. arXiv preprint arXiv:2411.10129 (2024)

    2024 arXiv

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  6. [14]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al . 2024. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905 (2024)

    2024 arXiv

  7. [15]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-Coder Technical Report. arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  8. [16]

    Tobias Jetzen, Xavier Devroey, Nicolas Matton, and Benoît Vanderose. 2024. Towards Debiasing Code Review Support . doi:10.48550/arXiv.2407.01407 arXiv:2407.01407 [cs]

    2024 doi arXiv

  9. [17]

    Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2014. The promises and perils of mining github. In Proceedings of the 11th working conference on mining software repositories . 92–101

    2014

  10. [18]

    Charles Koutcheme, Nicola Dainese, Arto Hellas, Sami Sarsa, Juho Leinonen, Syed Ashraf, and Paul Denny. 2024. Evaluating Language Models for Generating and Judging Programming Feedback. arXiv.org. https://arxiv.org/abs/2407.04873v1

    2024 arXiv

  11. [19]

    Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, and Chun Zuo. 2022. AUGER: Automatically Generating Review Comments with Pre-training Models. doi:10.48550/arXiv.2208.08014 arXiv:2208.08014 [cs]

    2022 doi arXiv

  12. [20]

    Yichen Li, Yintong Huo, Zhihan Jiang, Renyi Zhong, Pinjia He, Yuxin Su, Lionel C Briand, and Michael R Lyu. 2024. Exploring the Effectiveness of LLMs in Auto- mated Logging Statement Generation: An Empirical Study. IEEE Transactions on Software Engineering (2024)

    2024

  13. [21]

    Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan

  14. [22]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.02977 (2024)

    2024 arXiv

  15. [23]

    Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. 2024. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310 (2024)

    2024 arXiv

  16. [24]

    Siyao Liu, He Zhu, Jerry Liu, Shulin Xin, Aoyan Li, Rui Long, Li Chen, Jack Yang, Jinxiang Xia, ZY Peng, et al . 2024. FullStack Bench: Evaluating LLMs as Full Stack Coder. arXiv preprint arXiv:2412.00535 (2024). BitsAI-CR: Automated Code Review via LLM in Practice

    2024 arXiv

  17. [25]

    Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. LLaMA-Reviewer: Ad- vancing Code Review Automation with Large Language Models through Parameter- Efficient Fine-Tuning. arXiv.org. https://arxiv.org/abs/2308.11148v2

    2023 arXiv

  18. [26]

    Srijoni Majumdar, Ayush Bansal, Partha Pratim Das, Paul D Clough, Kausik Datta, and Soumya Kanti Ghosh. 2022. Automated evaluation of comments to aid software maintenance. Journal of Software: Evolution and Process 34, 7 (2022), e2463

    2022

  19. [27]

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishin- skaya, Maja Trebacz, and Jan Leike. 2024. LLM Critics Help Catch LLM Bugs . arXiv.org. https://arxiv.org/abs/2407.00215v1

    2024 arXiv

  20. [28]

    Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E Hassan. 2016. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering 21 (2016), 2146–2189

    2016

  21. [29]

    Atharva Naik, Marcus Alenius, Daniel Fried, and Carolyn Rose. 2024. CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells. doi:10.48550/arXiv.2409.19801 arXiv:2409.19801 [cs]

    2024 doi arXiv

  22. [30]

    OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023). https://arxiv.org/abs/2303.08774

    2023 arXiv

  23. [31]

    Chanathip Pornprasit and Chakkrit Tantithamthavorn. 2024. Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation . arXiv.org. https://arxiv.org/abs/2402.00905v4

    2024 arXiv

  24. [32]

    Rachel Potvin and Josh Levenberg. 2016. Why Google stores billions of lines of code in a single repository. Commun. ACM 59, 7 (2016), 78–87

    2016

  25. [33]

    Zeeshan Rasheed, Malik Abdul Sami, Muhammad Waseem, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen, Kari Systä, and Pekka Abrahamsson. 2024. AI- powered Code Review with LLMs: Early Results . arXiv.org. https://arxiv.org/abs/ 2404.18496v1

    2024

  26. [34]

    Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern code review: a case study at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice. 181–190

    2018

  27. [35]

    Devarshi Singh, Varun Ramachandra Sekar, Kathryn T Stolee, and Brittany Johnson. 2017. Evaluating how static analysis tools can reduce code review effort. In 2017 IEEE symposium on visual languages and human-centric computing (VL/HCC). IEEE, 101–105

    2017

  28. [36]

    Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. 2024. Unicoder: Scaling code large language model via universal code. arXiv preprint arXiv:2406.16441 (2024)

    2024 arXiv

  29. [37]

    Tao Sun, Yang Yang, Xianfu Cheng, Jian Yang, Yintong Huo, Zhuoren Ye, Rubing Yang, Xiangyuan Guan, Wei Zhang, Hangyuan Ji, et al. [n. d.]. RepoFixEval: A Repository-Level Program Repair Benchmark From Issue Discovering to Bug Fixing. ([n. d.])

  30. [38]

    Bissyande

    Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawende F. Bissyande. 2024. CodeAgent: Autonomous Communicative Agents for Code Review . arXiv.org. https://arxiv.org/abs/2402. 02172v5

    2024

  31. [39]

    Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. 2022. Using Pre-Trained Models to Boost Code Review Automation. doi:10.48550/arXiv.2201.06850 arXiv:2201.06850 [cs]

    2022 doi arXiv

  32. [40]

    Manushree Vijayvergiya, Małgorzata Salawa, Ivan Budiselić, Dan Zheng, Pas- cal Lamblin, Marko Ivanković, Juanjo Carin, Mateusz Lewko, Jovan Andonov, Goran Petrović, Daniel Tarlow, Petros Maniatis, and René Just. 2024. AI- Assisted Assessment of Coding Practices in Modern Code ...

    2024

  33. [41]

    Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and Michael R Lyu. 2024. Automatically generating UI code from screenshot: A divide-and-conquer-based approach. arXiv preprint arXiv:2406.16386 (2024)

    2024 arXiv

  34. [42]

    Chaozheng Wang, Shuzheng Gao, Cuiyun Gao, Wenxuan Wang, Chun Yong Chong, Shan Gao, and Michael R Lyu. 2024. A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software ...

    2024

  35. [43]

    Chaozheng Wang, Zongjie Li, Cuiyun Gao, Wenxuan Wang, Ting Peng, Hail- iang Huang, Yuetang Deng, Shuai Wang, and Michael R Lyu. 2024. Exploring Multi-Lingual Bias of Large Code Models in Code Generation. arXiv preprint arXiv:2404.19368 (2024)

    2024 arXiv

  36. [44]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  37. [45]

    Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zhiyao Xu, and Michael R Lyu. 2024. Interaction2Code: How Far Are We From Automatic Interactive Webpage Gen- eration? arXiv preprint arXiv:2411.03292 (2024)

    2024

  38. [46]

    Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, and Junyang Lin. 2024. Evaluating and Aligning CodeLLMs on Human Preference. arXiv preprint arXiv:2412.05210 (2024)

    2024 arXiv

  39. [47]

    Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. 2023. EvaCRC: Evaluating Code Review Comments. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco CA USA, 202...

    2023

  40. [48]

    Zezhou Yang, Cuiyun Gao, Zhaoqiang Guo, Zhenhao Li, Kui Liu, Xin Xia, and Yuming Zhou. 2024. A Survey on Modern Code Review: Progresses, Challenges and Opportunities. arXiv.org. https://arxiv.org/abs/2405.18216v1

    2024

  41. [49]

    Ying Yin, Yuhai Zhao, Yiming Sun, and Chen Chen. 2023. Automatic Code Review by Learning the Structure Information of Code Graph. 23, 5 (2023), 2551. Issue 5. doi:10.3390/s23052551

    2023 doi

  42. [50]

    Yongda Yu, Guoping Rong, Haifeng Shen, He Zhang, Dong Shao, Min Wang, Zhao Wei, Yong Xu, and Juhong Wang. 2024. Fine-Tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review. (2024), 3695993. doi:10.1145/3695993

    2024 doi

  43. [51]

    Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023. Unifying the perspectives of nlp and software engi- neering: A survey on language models for code. arXiv preprint arXiv:2311.07989 (2023)

    2023 arXiv

  44. [52]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  45. [53]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658 (2024)

    2024 arXiv

  46. [54]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al . 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence.arXiv preprint arXiv:2406.11931 (2024)

    2024 arXiv

  47. [2022]

    Automating Code Review Activities by Large-Scale Pre-training . doi:10. 48550/arXiv.2203.09095 arXiv:2203.09095 [cs]

    doi arXiv

  48. [2024]

    https://arxiv.org/abs/2408.05534v1

    Can LLMs Replace Manual Annotation of Software Engineering Artifacts? arXiv.org. https://arxiv.org/abs/2408.05534v1

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.