Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Combining Large Language Models with Static Analyzers for Code Review Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Feeding static analyzer results into LLM prompts makes generated code reviews more accurate and more complete.

desk verdict Useful empirical comparison of three ways to bolt static analysis onto LLM code review generation, but the coverage ranking is contaminated by the same LLM that filtered the training data, so only the RAG accuracy result carries real weight. read the letter →

arxiv 2502.06633 v1 pith:7OMB3JIT submitted 2025-02-10 cs.SE cs.AI

Imen Jaoua , Oussama Ben Sghaier , Houari Sahraoui This is my paper · ORCID
classification cs.SEcs.AI
keywords codereviewgenerationlargelanguagemodelsstaticanalysisretrieval-augmenteddataaugmentationLLM-as-a-judgehybridsystemsJava
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that automated code review gets better when rule-based static analyzers and large language models are combined rather than used alone. It tests three ways to combine them: retraining the model on data augmented with static-analysis outputs, injecting static-analysis results into the prompt at generation time, and simply concatenating the two outputs. In an evaluation on Java code changes, the prompt-injection method improved both accuracy and coverage of generated review comments, while data-augmented training improved coverage at some cost to precision. If the result holds, teams using LLM-based review tools can get closer to human-quality feedback by feeding structured analyzer findings into the model without waiting for a better base model.

What carries the argument

The load-bearing machinery is the choice of where static-analysis knowledge enters the LLM pipeline. Data-Augmented Training (DAT) builds an augmented fine-tuning set in which static-analyzer reports and LLM-generated reviews are rated by a larger LLM, filtered, and balanced, then retrains the model on it. Retrieval-Augmented Generation (RAG) retrieves the static analyzer's findings for the code under review and includes them directly in the prompt, so the model generates with rule-based evidence in context. Naive Concatenation of Outputs (NCO) appends the static analyzer's report to the LLM's comment after generation. The argument rests on RAG, because it is the only strategy that improves accuracy and coverage simultaneously.

What would settle it

Ask human reviewers to independently list every genuine issue in a random sample of the test code changes, then check whether RAG and DAT reviews cover more of those human-listed issues than the judge LLM's own list; if the coverage advantage disappears under human enumeration, the central claim would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that knowledge from static analyzers can be fused into an LLM code-review pipeline at three stages, and that the stage matters. Retrieval-augmented generation (RAG), which inserts analyzer findings into the prompt, was the most effective configuration: it raised the accuracy of generated comments relative to the fine-tuned model alone and gave strong issue coverage. Data-augmented training (DAT) achieved the broadest coverage, often ranking first, but with a bimodal pattern in which the model either covered most issues or missed most. Naive concatenation of outputs (NCO) gave only moderate gains because it inherits the LLM's weaknesses. The paper concludes that combining rule-based precision with learning-based adaptability is beneficial, with RAG as the recommended integration point.

Load-bearing premise

The coverage results assume the larger LLM used as a judge has already listed every issue worth finding in each code change, but that model was only validated for classifying reviews as accurate, partially accurate, or not accurate, not for exhaustively enumerating issues.

Editorial extensions

If this is right

  • RAG-style prompting can raise the accuracy of LLM-generated code reviews without retraining, as long as static analyzer findings are available at inference time.
  • DAT can expand the range of issues a small fine-tuned model detects, but the bimodal ranking pattern warns that coverage gains may come with unstable precision.
  • NCO is the cheapest integration but the least reliable, since its quality is capped by the LLM's own review quality.
  • The hybrid approach can be applied to any LLM and any source-code static analyzer, not just the specific model and tools used in the experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The coverage comparison uses the larger judge LLM as the reference list of issues, so the reported coverage advantage may partly reflect how closely each approach mimics the judge's own comment style; a human-exhaustive issue enumeration would be a stronger test.
  • Combining DAT and RAG is the natural next step: the paper evaluates them separately, but their complementary strengths (DAT for breadth, RAG for accuracy) suggest an additive benefit.
  • Because only Java and two style-oriented analyzers were tested, the relative ordering of the three strategies could shift for languages with stronger semantic analyzers or for bug-oriented tools.
  • The DAT bimodality implies the augmented dataset's balance between analyzer and LLM reviews is a sensitive hyperparameter; rebalancing toward one source may trade coverage for consistency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes three hybrid strategies to combine static analyzers (KBS) with a fine-tuned large language model (LBS) for automated code review generation: Data-Augmented Training (DAT), Retrieval-Augmented Generation (RAG), and Naive Concatenation of Outputs (NCO). Using CodeLlama-7b as the LBS and PMD/Checkstyle as the KBS on a Java dataset, the authors evaluate the strategies with a 10% human sample and an LLM-as-a-judge (Llama3-70B) on a filtered set of 1,245 common code differences. The results show that RAG improves accuracy over the standalone LLM (RQ1/RQ3), while DAT and NCO do not; RQ4's coverage ranking places DAT and RAG at the top. The paper concludes that hybrid strategies enhance accuracy and coverage, with RAG being the most effective.

Significance. If the RAG result is robust, the paper offers a practical, low-cost way to improve LLM-generated code reviews by injecting static-analysis context into prompts, and it ships a replication package and dataset. The human-evaluation sanity check (Cohen's kappa 0.72) provides some independent grounding for the accuracy claims. However, the coverage claims—especially for DAT—depend on a self-referential Llama3-70B judge and a potentially leaky data split, and the abstract overstates results that the paper's own figures contradict. The work is relevant to the code-review automation community, but the coverage evidence needs substantial strengthening before the main claims can be accepted.

major comments (4)
  1. [Section III-C / Section IV-B, RQ4] The coverage evaluation for DAT is circular: Llama3-70B is used both to filter the DAT training data (threshold 8, Figure 4) and as the RQ4 'reference point' judge that ranks DAT reviews first in 49% of cases (Section IV-C4). The RQ2 validation (kappa 0.72) was for three-way accuracy classification on roughly 125 samples, not for exhaustive issue enumeration, so it does not justify the assumption that Llama3-70B 'can identify a comprehensive set of issues.' The authors should validate coverage with an independent ground truth (e.g., human enumeration of issues on a sample) or at minimum report the promised manual inspection details (Section IV-E) with inter-rater agreement.
  2. [Section III-C / Section IV-B] The augmented dataset Da is built from the original test set Do (the 27,267 Java entries from DMi's test set) and then randomly split into train/validation/test without any stated grouping by code-change identity. Because each code change appears in multiple data points (one per review comment), the same code change may appear in both MF T's training data and the evaluation test set, which would inflate DAT's accuracy and coverage via memorization rather than generalization. The authors must clarify whether the split was grouped by file/code change and, if not, repeat the DAT evaluation with a grouped split.
  3. [Abstract / Section I / Section IV-C3] The claim that all three hybrid strategies 'enhance the relevance, completeness, and overall quality of review comments' is contradicted by the paper's own results: RQ1 (Figure 8) and RQ3 (Figure 9) show that DAT and NCO do not improve accuracy over Mi, and DAT has the lowest LLM-judged accuracy. The supported accuracy claim is specific to RAG; the abstract and conclusion should be revised to avoid the blanket statement.
  4. [Section IV-C4] The win-tie-loss coverage analysis uses an arbitrary threshold of at least two rank levels for a win/loss and aggregates percentages without statistical tests or confidence intervals. Additionally, coverage is defined entirely by the judge's ranking, so a 'win' may reflect stylistic alignment with Llama3-70B rather than a genuinely broader set of issues. Reporting pairwise agreement rates, using multiple judges, and inspecting cases where rankings disagree with human judgment would strengthen the claim.
minor comments (5)
  1. [Section IV-B] The exact number of samples used in RQ1 is not stated; please report the precise count (10% of 1,245) and the human-human agreement before resolving conflicts.
  2. [Section III-C] Please specify the number of reviews retained before and after the threshold-8 filter and after the random balancing between KBS and LBS, since '78,776 samples' alone does not convey the filtering effect.
  3. [Section IV-D, Table I] The DAT example includes speculative suggestions (e.g., 'It is possible to remove the final keyword') that may be inaccurate; a representative example would make the qualitative analysis more convincing.
  4. [Section IV-B] The text uses inconsistent notation ('dif f' vs 'diff', 'structered') and reference [52] appears to misspell 'Tufano'; please proofread.
  5. [Figures 10 and 11] Please add error bars or confidence intervals, as the raw percentages are based on 1,245 samples and may not support the strength of the claims.

Circularity Check

1 steps flagged · score 6.0 of 10

DAT coverage ranking is circular: Llama3-70B filters DAT's training data and then serves as the RQ4 coverage judge, so DAT's Rank-1 coverage advantage may measure judge alignment.

  1. fitted input called prediction [Sec. III-C (Data Filtering / Fig. 4) and Sec. IV-B (RQ4 setup)]
    "We used Llama3-70b, inputting the code and its corresponding reviews. We instructed the model to rate each review on a 10-point scale. A threshold rating of 8 was set, with only reviews surpassing this threshold retained in the final dataset. ... For RQ4, We conducted a comparative evaluation by ranking the generated reviews based on their coverage ... we used the larger LLM, Llama3-70B, as a reference point, assuming it can identify a comprehensive set of issues."

    The RQ4 coverage metric is operationalized as agreement with Llama3-70B's ranking of which generated review covers a comprehensive set of issues. DAT's training set Da was constructed by keeping only reviews that Llama3-70B rated >=8 on the same code-review relevance task. Thus MF T is explicitly trained toward comments that Llama3-70B prefers, and the coverage evaluation then uses Llama3-70B as the referee. A DAT review that echoes Llama3-filtered comment styles is rewarded by construction; the reported 49% Rank-1 result for DAT can be explained by judge alignment rather than by an independent measure of issue completeness. The kappa=0.72 validation covers only three-way accuracy classification on ~125 samples, not exhaustive issue enumeration, so it does not break the circularity.

full rationale

The paper's accuracy claims for RAG are supported by an independent human evaluation (RQ1: RAG significantly outperforms Mi) and by LLM evaluation; those are not circular. The circularity burden is concentrated in the RQ4 coverage ranking. DAT's training set was filtered by Llama3-70B (reviews rated >=8 retained), and RQ4 then uses Llama3-70B as the 'reference point' for what counts as comprehensive coverage. Therefore the DAT coverage result (49% Rank 1) is at least partly a measure of agreement with the same model that curated its training data, not an independent measure of issue completeness. The RQ2 kappa=0.72 validation covers only three-way accuracy classification and does not validate exhaustive issue enumeration, so it cannot break this circularity. RAG's coverage is also judged by the same unvalidated reference point, though RAG was not trained on Llama3-filtered data; its accuracy finding retains independent content. The paper's self-citations ([5], [46]) are not load-bearing. Overall: partial circularity in the coverage conclusion for DAT, while the central accuracy conclusion for RAG is independently grounded; score 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its load-bearing choices are the filtering threshold, the balanced synthetic dataset, the rank-difference threshold, and the untested assumption that Llama3-70B can enumerate all issues in a code change. The most consequential assumption is the judge-as-ground-truth for coverage, which is partially circular with DAT data filtering.

free parameters (4)
  • Relevance filtering threshold = 8 on a 10-point scale
    Reviews rated below 8 by Llama3-70B are discarded from the augmented dataset (Section III-C).
  • Number of Mi-generated reviews per code change = 4
    Each code change is used to generate four synthetic LBS reviews for the augmented dataset (Section III-C).
  • KBS/LBS balance in augmented dataset = 50/50 equal distribution
    The final dataset Da is balanced so that half the reviews come from static analyzers and half from the LLM (Section III-C).
  • Win-tie-loss rank difference threshold = 2 rank levels
    In RQ4, a win or loss requires at least a two-level ranking difference; smaller differences count as ties (Section IV-B).
assumptions (4)
  • domain assumption CodeLlama-7b fine-tuned on DMi is a competent representative of learning-based code review generation.
    All comparisons treat the fine-tuned Mi model as the LBS baseline, without benchmarking it against other state-of-the-art review generation models (Section III-B).
  • domain assumption PMD and Checkstyle rule outputs can be treated as knowledge-based review comments comparable to LLM comments.
    KBS reviews are extracted rule messages with surrounding code context, but no evidence shows they match the format or intent of human review comments (Section III-B, III-C).
  • domain assumption Llama3-70B judgments are a reliable substitute for human judgments on the full test set.
    The substitution is justified only by Cohen's kappa 0.72 on a roughly 125-sample subset, and it is applied to both accuracy and coverage evaluations (Section IV-B, RQ2).
  • ad hoc to paper Merging overlapping code differences into a unified diff preserves the validity of paired KBS and LBS reviews.
    Section IV-B constructs unified diffs and triplets (diff_u, r1, r2) when two code changes overlap, which may alter the context the reviews refer to.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Combining Large Language Models with Static Analyzers for Code Review Generation." pith.science (2026). https://pith.science/paper/7OMB3JIT

@misc{pith2026250206633,
  author       = {Pith},
  title        = {Pith review of: Combining Large Language Models with Static Analyzers for Code Review Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OMB3JIT}},
  note         = {Machine review of arXiv:2502.06633}
}
read the original abstract

Code review is a crucial but often complex, subjective, and time-consuming activity in software development. Over the past decades, significant efforts have been made to automate this process. Early approaches focused on knowledge-based systems (KBS) that apply rule-based mechanisms to detect code issues, providing precise feedback but struggling with complex, context-dependent cases. More recent work has shifted toward fine-tuning pre-trained language models for code review, enabling broader issue coverage but often at the expense of precision. In this paper, we propose a hybrid approach that combines the strengths of KBS and learning-based systems (LBS) to generate high-quality, comprehensive code reviews. Our method integrates knowledge at three distinct stages of the language model pipeline: during data preparation (Data-Augmented Training, DAT), at inference (Retrieval-Augmented Generation, RAG), and after inference (Naive Concatenation of Outputs, NCO). We empirically evaluate our combination strategies against standalone KBS and LBS fine-tuned on a real-world dataset. Our results show that these hybrid strategies enhance the relevance, completeness, and overall quality of review comments, effectively bridging the gap between rule-based tools and deep learning models.

Figures

Figures reproduced from arXiv: 2502.06633 by the authors.

Figure 1
Figure 1. Different strategies to combine learning and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Proposed strategies to combine LBS and KBS [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Dataset augmentation pipeline [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Distribution of LBS and KBS Reviews in Our Dataset [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: Judgment of review comments using Llama3-70B To ensure a fair and scalable rating system, we leveraged large language models, which have demonstrated remarkable performance in similar assessment tasks [44]–[46]. By using these models, we achieved a more accurate and co…
Figure 6
Figure 6. Figure 6: Prompt used to generate review comments using RAG [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Merging LBS and KBS Changes into a Union Diff [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Accuracy levels for the different models based on [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Accuracy levels for the different models based on LLM [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Distribution of Ranks Across Models D. Discussion Based on the results from RQ3 and RQ4, our baseline models performed as expected, each with distinct strengths and limitations. The knowledge-based system (KBS) demonstrated high accuracy in detecting rule-based issues…
Figure 11
Figure 11. Figure 11: Win-tie-loss ratios of our proposed approaches compared to the baseline in terms of coverage. [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation

    cs.SE 2025-09 conditional novelty 6.0 of 10

    SWR-Bench is a PR-centric code review benchmark with objective LLM scoring; current ACR tools reach at best 19.4% F1, and multi-review aggregation yields relative F1 gains up to 43.7%.

  2. CRScore++: Reinforcement Learning with Verifiable Tool and AI Feedback for Code Review

    cs.SE 2025-05 conditional novelty 5.0 of 10

    A two-stage SFT+DPO pipeline using linter and code-smell tool outputs plus an LLM judge improves generated code review comments and appears to transfer from Python to Java and JavaScript, though the judge-based evalua...

Reference graph

Works this paper leans on

62 extracted references · 31 canonical work pages · cited by 2 Pith papers

  1. [1]

    A review of code reviewer recommendation studies: Challenges and future directions,

    H. A. C ¸ etin, E. Do ˘gan, and E. T ¨uz¨un, “A review of code reviewer recommendation studies: Challenges and future directions,” Science of Computer Programming, vol. 208, p. 102652, 2021

    work page 2021

  2. [2]

    A survey on source code review using machine learning,

    W. Xiaomeng, Z. Tao, X. Wei, and H. Changyu, “A survey on source code review using machine learning,” in 2018 3rd International Con- ference on Information Systems Engineering (ICISE) . IEEE, 2018, pp. 56–60

    work page 2018

  3. [3]

    Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,

    Y . Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,” in Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering, 2022, pp. 507–519

    2022

  4. [4]

    Four eyes are better than two: On the impact of code reviews on software quality,

    G. Bavota and B. Russo, “Four eyes are better than two: On the impact of code reviews on software quality,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2015, pp. 81–90

    2015

  5. [5]

    Improving the learning of code review successive tasks with cross-task knowledge distillation,

    O. Ben Sghaier and H. Sahraoui, “Improving the learning of code review successive tasks with cross-task knowledge distillation,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 1086–1106, 2024

    2024

  6. [6]

    Finding bugs is easy,

    D. Hovemeyer and W. Pugh, “Finding bugs is easy,” Acm sigplan notices, vol. 39, no. 12, pp. 92–106, 2004

    work page 2004

  7. [7]

    Tricorder: Building a program analysis ecosystem,

    C. Sadowski, J. Van Gogh, C. Jaspan, E. Soderberg, and C. Winter, “Tricorder: Building a program analysis ecosystem,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 1. IEEE, 2015, pp. 598–608

    work page 2015

  8. [8]

    Core: Resolving code quality issues using llms,

    N. Wadhwa, J. Pradhan, A. Sonwane, S. P. Sahu, N. Natarajan, A. Kanade, S. Parthasarathy, and S. Rajamani, “Core: Resolving code quality issues using llms,” Proceedings of the ACM on Software Engi- neering, vol. 1, no. FSE, pp. 789–811, 2024

    work page 2024

Show all 62 references
  1. [9]

    Code review automation: strengths and weaknesses of the state of the art,

    R. Tufano, O. Dabi ´c, A. Mastropaolo, M. Ciniselli, and G. Bavota, “Code review automation: strengths and weaknesses of the state of the art,” IEEE Transactions on Software Engineering , 2024

    2024

  2. [10]

    Towards contextually aware large language models for software requirements engineering: A retrieval augmented generation framework,

    M. S. Ibtasham, “Towards contextually aware large language models for software requirements engineering: A retrieval augmented generation framework,” 2024

    2024

  3. [11]

    Core: Automating review recommendation for code changes,

    J. K. Siow, C. Gao, L. Fan, S. Chen, and Y . Liu, “Core: Automating review recommendation for code changes,” in 2020 IEEE 27th Interna- tional Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2020, pp. 284–295

    2020

  4. [12]

    Automating code review,

    R. Tufano, “Automating code review,” in 2023 IEEE/ACM 45th Inter- national Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2023, pp. 192–196

    2023

  5. [13]

    A survey on modern code review: Progresses, challenges and opportunities,

    Z. Yang, C. Gao, Z. Guo, Z. Li, K. Liu, X. Xia, and Y . Zhou, “A survey on modern code review: Progresses, challenges and opportunities,” arXiv preprint arXiv:2405.18216, 2024

    2024

  6. [14]

    Code generation using machine learning: A systematic review,

    E. Dehaerne, B. Dey, S. Halder, S. De Gendt, and W. Meert, “Code generation using machine learning: A systematic review,” Ieee Access , vol. 10, pp. 82 434–82 455, 2022

    2022

  7. [15]

    Evaluating how static analysis tools can reduce code review effort,

    D. Singh, V . R. Sekar, K. T. Stolee, and B. Johnson, “Evaluating how static analysis tools can reduce code review effort,” in 2017 IEEE symposium on visual languages and human-centric computing (VL/HCC). IEEE, 2017, pp. 101–105

    2017

  8. [16]

    Evaluating bug finders– test and measurement of static code analyzers,

    A. Delaitre, B. Stivalet, E. Fong, and V . Okun, “Evaluating bug finders– test and measurement of static code analyzers,” in 2015 IEEE/ACM 1st International Workshop on Complex Faults and Failures in Large Software Systems (COUFLESS) . IEEE, 2015, pp. 14–20

    2015

  9. [17]

    Using static analysis to find bugs,

    N. Ayewah, W. Pugh, D. Hovemeyer, J. D. Morgenthaler, and J. Penix, “Using static analysis to find bugs,” IEEE software, vol. 25, no. 5, pp. 22–29, 2008

    2008

  10. [18]

    On adopting linters to deal with performance concerns in android apps,

    S. Habchi, X. Blanc, and R. Rouvoy, “On adopting linters to deal with performance concerns in android apps,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineer- ing, 2018, pp. 6–16

    2018

  11. [19]

    FindBugs,

    “FindBugs,” https://findbugs.sourceforge.net/, 2005

    2005

  12. [20]

    “PMD,” https://pmd.github.io/, 2000

    2000

  13. [21]

    Checkstyle,

    “Checkstyle,” https://checkstyle.org/, 2001

    2001

  14. [22]

    SonarQube,

    SonarSource, “SonarQube,” https://www.sonarsource.com/products/ sonarqube/, 2006

    2006

  15. [23]

    Static code analysis tools: A systematic literature review,

    D. Stefanovi ´c, D. Nikoli ´c, D. Daki ´c, I. Spasojevi´c, and S. Risti ´c, “Static code analysis tools: A systematic literature review,” in Ann. DAAAM Proc. Int. DAAAM Symp , vol. 31, no. 1, 2020, pp. 565–573

    2020

  16. [24]

    Analyzing the state of static analysis: A large-scale evaluation in open source software,

    M. Beller, R. Bholanath, S. McIntosh, and A. Zaidman, “Analyzing the state of static analysis: A large-scale evaluation in open source software,” in 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) , vol. 1. IEEE, 2016, pp. 470–481

    2016

  17. [25]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu et al. , “Automating code review activities by large-scale pre-training,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundat...

    2022

  18. [26]

    Using pre-trained models to boost code review automa- tion,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained models to boost code review automa- tion,” in Proceedings of the 44th international conference on software engineering, 2022, pp. 2291–2302

    2022

  19. [27]

    Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommenda- tion,

    V . Balachandran, “Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommenda- tion,” in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 931–940

    2013

  20. [28]

    Auger: automatically generating review comments with pre-training models,

    L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger: automatically generating review comments with pre-training models,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2022

  21. [29]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 2023, pp. 647–658

    2023

  22. [30]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022

    2022 arXiv

  23. [31]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023

    2023 arXiv

  24. [32]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  25. [33]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997 , 2023

    2023 arXiv

  26. [34]

    Business insights using rag–llms: a review and case study,

    M. Arslan, S. Munawar, and C. Cruz, “Business insights using rag–llms: a review and case study,” Journal of Decision Systems , pp. 1–30, 2024

    2024

  27. [35]

    Judgelm: Fine-tuned large language models are scalable judges,

    L. Zhu, X. Wang, and X. Wang, “Judgelm: Fine-tuned large language models are scalable judges,” arXiv preprint arXiv:2310.17631 , 2023

    2023 arXiv

  28. [36]

    Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms,

    L. Shi, W. Ma, and S. V osoughi, “Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms,” arXiv preprint arXiv:2406.07791 , 2024

    2024

  29. [37]

    Judging the judges: Evaluating alignment and vulnerabili- ties in llms-as-judges,

    A. S. Thakur, K. Choudhary, V . S. Ramayapally, S. Vaidyanathan, and D. Hupkes, “Judging the judges: Evaluating alignment and vulnerabili- ties in llms-as-judges,” arXiv preprint arXiv:2406.12624 , 2024

    2024 arXiv

  30. [38]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  31. [39]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  32. [40]

    A critical comparison on six static analysis tools: Detection, agreement, and precision,

    V . Lenarduzzi, F. Pecorelli, N. Saarimaki, S. Lujan, and F. Palomba, “A critical comparison on six static analysis tools: Detection, agreement, and precision,” Journal of Systems and Software , vol. 198, p. 111575, 2023

    2023

  33. [41]

    Comparing bug finding tools for java open source software,

    E. H. Oskouei and O. Kalıpsız, “Comparing bug finding tools for java open source software,” 2018

    2018

  34. [42]

    Why don’t software developers use static analysis tools to find bugs?

    B. Johnson, Y . Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 672–681

    2013

  35. [43]

    The effectiveness of supervised machine learning algorithms in predicting software refac- toring,

    M. Aniche, E. Maziero, R. Durelli, and V . H. Durelli, “The effectiveness of supervised machine learning algorithms in predicting software refac- toring,” IEEE Transactions on Software Engineering , vol. 48, no. 4, pp. 1432–1450, 2020

    2020

  36. [44]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  37. [45]

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers,

    H. Huang, Y . Qu, J. Liu, M. Yang, and T. Zhao, “An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers,” arXiv preprint arXiv:2403.02839 , 2024

    2024 arXiv

  38. [46]

    Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,

    M. Weyssow, A. Kamanda, and H. Sahraoui, “Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,” arXiv preprint arXiv:2403.09032 , 2024

    2024 arXiv

  39. [47]

    Active retrieval augmented generation,

    Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y . Yang, J. Callan, and G. Neubig, “Active retrieval augmented generation,” arXiv preprint arXiv:2305.06983, 2023

    2023 arXiv

  40. [48]

    Replication package,

    “Replication package,” https://github.com/ImenJaoua/ Hybrid-Code-Review

  41. [49]

    Datasets and results,

    Anonymous, “Datasets and results,” 2024. [Online]. Available: https://zenodo.org/records/14061110

    2024

  42. [50]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  43. [51]

    Intelligent code reviews using deep learning,

    A. Gupta and N. Sundaresan, “Intelligent code reviews using deep learning,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’18) Deep Learning Day, 2018

    2018

  44. [52]

    Towards automating code review activities,

    R. Tufan, L. Pascarella, M. Tufanoy, D. Poshyvanykz, and G. Bavota, “Towards automating code review activities,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , 2021, pp. 163–174

    2021

  45. [53]

    Repairagent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” arXiv preprint arXiv:2403.17134 , 2024

    2024 arXiv

  46. [54]

    Pyty: Repairing static type errors in python,

    Y . W. Chow, L. Di Grazia, and M. Pradel, “Pyty: Repairing static type errors in python,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  47. [55]

    Learning deep semantics for test completion,

    P. Nie, R. Banerjee, J. J. Li, R. J. Mooney, and M. Gligoric, “Learning deep semantics for test completion,” in 2023 IEEE/ACM 45th Interna- tional Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2111–2123

    2023

  48. [56]

    No more manual tests? evaluating and improving chatgpt for unit test generation,

    Z. Yuan, Y . Lou, M. Liu, S. Ding, K. Wang, Y . Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,” arXiv preprint arXiv:2305.04207 , 2023

    2023 arXiv

  49. [57]

    Adaptive test generation using a large language model,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “Adaptive test generation using a large language model,” 2023, cite arxiv:2302.06527. [Online]. Available: http://arxiv.org/abs/2302.06527

    2023 arXiv

  50. [58]

    Skipanalyzer: A tool for static code analysis with large language models,

    M. M. Mohajer, R. Aleithan, N. S. Harzevili, M. Wei, A. B. Belle, H. V . Pham, and S. Wang, “Skipanalyzer: A tool for static code analysis with large language models,” arXiv preprint arXiv:2310.18532 , 2023

    2023 arXiv

  51. [59]

    Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis,

    Y . Sun, D. Wu, Y . Xue, H. Liu, H. Wang, Z. Xu, X. Xie, and Y . Liu, “Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  52. [60]

    D2a: A dataset built for ai-based vulnerability detection methods using differential analysis,

    Y . Zheng, S. Pujar, B. Lewis, L. Buratti, E. Epstein, B. Yang, J. Laredo, A. Morari, and Z. Su, “D2a: A dataset built for ai-based vulnerability detection methods using differential analysis,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software En...

    2021

  53. [61]

    Reposvul: A repository-level high-quality vulnerability dataset,

    X. Wang, R. Hu, C. Gao, X.-C. Wen, Y . Chen, and Q. Liao, “Reposvul: A repository-level high-quality vulnerability dataset,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings , 2024, pp. 472–483

    2024

  54. [62]

    Stall+: Boosting llm-based repository-level code completion with static analysis,

    J. Liu, Y . Chen, M. Liu, X. Peng, and Y . Lou, “Stall+: Boosting llm-based repository-level code completion with static analysis,” arXiv preprint arXiv:2406.10018, 2024

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.