Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

DPO-F+: Aligning Code Repair Feedback with Developers' Preferences

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a frozen reward-model margin to preference training makes code-repair feedback more accurate and more aligned with developer needs, the paper argues.

desk verdict The empirical pipeline is real, but the abstract sells a human-preference result the paper doesn't contain, and the rubric circularity is load-bearing. read the letter →

arxiv 2511.01043 v2 pith:EYXDT3HH submitted 2025-11-02 cs.SE

Zihan Fang , Yifan Zhang , Yueke Zhang , Kevin Leach , Yu Huang This is my paper · ORCID
classification cs.SE
keywords coderepairfeedbackalignmentdirectpreferenceoptimizationreward-augmentedDPOdeveloperpreferencescomprehensionhuman-AIteamingLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that code repair assistance is not just about producing corrected code: the natural-language feedback accompanying the fix must match the developer's experience level and comprehension needs. It introduces DPO-f+, a training method that extends Direct Preference Optimization with a frozen reward-model margin, so preferred versus dispreferred feedback pairs are weighted by how decisive the preference is rather than treated as equal binary labels. On novice programming tasks, DPO-f+ raises Pass@1 by 5.71 percentage points over the baseline and 3.30 over standard DPO; on SWE-bench Lite it resolves 4.67 percentage points more issues than the baseline. The framework also scores highest on all seven feedback-alignment dimensions and wins a majority of pairwise comparisons, with beginners preferring DPO-f+ feedback in 71.5% of human evaluations (p=.0057). If correct, the paper shows that structured, developer-profiled alignment of natural-language guidance is a trainable property that improves both objective code accuracy and perceived feedback quality.

What carries the argument

The load-bearing mechanism is the reward-augmented DPO objective (Eq. 4), in which the policy's log-likelihood score for each response is combined with a frozen reward model's scaled score, and the loss contrasts the combined margin of the preferred versus rejected response against the reference model's margin. The frozen reward supplies a graded preference signal that standard DPO lacks. The other key piece is the seven-dimension rubric — Conciseness, Quality, Explainability, Understandability, Completeness, Actionability, and Contextual Relevance — interpreted differently for novice and experienced developers; it defines what 'alignment' means and drives the automatic construction of accep

What would settle it

Run a fully blind human study in which novice and experienced developers rate baseline, standard DPO, and DPO-f+ feedback on the same seven dimensions without knowing the source; if DPO-f+ does not beat the alternatives in blind ratings, or if GPT-4 rubric scores correlate at or below chance with human developers' ratings on a validation set larger than the reported 100-item check, the alignment claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that code-repair feedback — the natural-language explanations that accompany corrected code — is a first-class alignment target, not a byproduct of learning to repair. It presents DPO-f+, which augments Direct Preference Optimization with a learned reward model's margin signal: instead of treating every accepted/rejected preference pair as equally informative, the combined score in the DPO objective up-weights clear wins and down-weights near-ties. Empirically, DPO-f+ improves Pass@1 on novice programming tasks by 5.71 percentage points over the baseline and 3.30 over standard DPO, and it raises the issue-resolution rate on SWE-bench Lite by 4.67 points over the

Load-bearing premise

The alignment claim rests on the assumption that GPT-4's seven-dimension rubric scores are a reliable proxy for real developer preferences, and that the preference pairs built from those scores are trustworthy ground truth; if that proxy fails, the claim that DPO-f+ is aligned with developers loses its force even if the code-accuracy gains are genuine.

Editorial extensions

If this is right

  • If DPO-f+ works as reported, preference-aligned feedback is not merely a courtesy: the alignment gains transfer to code correctness, with Pass@1 improving by 5.71 points over baseline on novice tasks and SWE-bench Lite resolution rising by 4.67 points.
  • Preference pairs can be built automatically from execution-verified, rubric-scored code, so the framework can be re-applied to new tasks without manual preference labeling at each step.
  • The same training recipe generalizes from introductory C++ tasks to real GitHub issue-fix pairs, producing measurable gains under the official SWE-bench Lite evaluator with identical retrieval support.
  • Feedback alignment can be evaluated at scale with LLM-as-judge protocols: rubric scores and pairwise judgments both favor DPO-f+, and the reported human calibration shows 95% agreement on a 100-item sample.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate whether the frozen reward-margin signal is the active ingredient or whether the gains come from the extra training signal alone; a controlled ablation that keeps the reward model but removes the margin would identify the mechanism.
  • Preference-accuracy gains are much larger than downstream Pass@k gains, suggesting rubric alignment and bug-fixing skill are partially separate; a human task measuring comprehension — time-to-understand or ability to apply the suggested fix — would test whether alignment buys comprehension directly.
  • Because the human calibration used only a small sample, a larger blind evaluation with developers across experience levels would determine whether GPT-4 rubric scores track real preferences beyond the reported 71.5% preference rate.
  • The recipe could plausibly transfer to code review comments, test generation, or documentation, where natural-language guidance is similarly the bottleneck for human-AI teaming.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DPO-f+, a fine-tuning framework that augments Direct Preference Optimization (DPO) with a frozen learned reward-model margin, aiming to align code-repair feedback with the needs of novice and experienced developers. The framework defines a seven-dimension rubric for feedback quality, constructs preference pairs by scoring candidate feedback with GPT-4, trains a reward model and policy on those pairs, and evaluates the resulting model on novice programming tasks and SWE-bench Lite. The authors report improvements over a base model and standard DPO in preference accuracy, Pass@k/executability on novice tasks, SWE-bench resolution rate, and rubric-based alignment scores.

Significance. If the central claim were supported, the idea of adding a graded, frozen reward-margin signal to DPO for profile-aware code-repair feedback would be a useful and inexpensive contribution to the SE/LLM alignment literature. The paper also has positive features: it ships source data and scripts, evaluates on two base models, uses an official SWE-bench harness, and separates code-accuracy from feedback-quality metrics. However, the headline claim—that DPO-f+ aligns feedback with real developer preferences—is not established by the evidence in the manuscript. The abstract reports a 200-developer human study that never appears in the body; the alignment evaluation is circular because the same GPT-4 rubric that creates the training labels is also used to score the final outputs; and the objective accuracy improvements are reported without confidence intervals or significance tests, with the SWE-bench difference between DPO-f+ and DPO being within sampling noise. The core contribution may be salvageable with additional human evaluation and a reframed claim, but as written the central empirical assertion is unsupported.

major comments (4)
  1. [Abstract; §5.1.3; §5.2.3; §6.2; §7.1] The abstract states that a human study with 200 developers found DPO-f+ preferred in 71.5% of comparisons with p=0.0057. No such study is reported in the body. The evaluation sections only present LLM-as-judge rubric scores and pairwise comparisons; §6.2 explicitly says 'future work will include direct human evaluation to further validate our findings,' and §7.1 again defers human studies. Either the human study must be fully described (participants, materials, procedure, results) or the abstract claim must be removed. As it stands, the manuscript's headline finding is absent from the reported experiments.
  2. [§4.2.3; §4.4.3; Tables 2 and 4] The alignment evaluation is circular. Preference labels for training are produced by GPT-4 scoring feedback with the seven-dimension rubric (§4.2.3). The main alignment evaluation then uses GPT-4 to score generated feedback on the same seven dimensions and DeepSeek-V3 for pairwise comparison (§4.4.3). Therefore the reported preference accuracy, G-Eval scores, and win rates measure agreement with the GPT-4 rubric, not agreement with actual developer preferences. The 100-item human calibration mentioned in §6.2 does not resolve this, because the text does not specify that raters were from the target developer population, how the 100 items were selected, or whether the 95% figure is inter-rater agreement or agreement with GPT-4's scores.
  3. [§5.1.2, Table 3; §5.2.2, Table 5] The objective accuracy claims lack statistical grounding. Table 3 reports executability and Pass@k values without confidence intervals or significance tests. Table 5 shows SWE-bench Lite resolutions of 20/300 for DPO-f+, 15/300 for DPO, and 6/300 for the baseline; the DPO-f+ vs. DPO difference is within sampling noise (e.g., Fisher's exact test gives a large p-value). The claimed +1.67 pp over DPO is not compelling evidence of an improvement. Confidence intervals and appropriate tests should be reported for all headline accuracy comparisons.
  4. [§5.1.1, Table 2] Preference accuracy is computed on test pairs drawn from the same GPT-4-labeled distribution used for training. The paper itself acknowledges that this metric 'can be susceptible to overfitting to specific rubric cues.' The large gains (e.g., Qwen baseline 0.4511 to DPO-f+ 0.8184) may therefore reflect the model learning to exploit rubric-specific patterns rather than developing a generalizable ability to predict developer preferences. Independent validation on human preference judgments or on out-of-distribution pairs is needed before these numbers can be interpreted as alignment with developers.
minor comments (5)
  1. [Eq. (1), Eq. (4)] The inverse temperature β is mentioned in the text but does not appear in the equations. Also, the data-dependent weight λ(x) is never defined; please specify how it is computed.
  2. [§5.1.3] The text says 'Table 1 reports per-metric scores' but the actual per-metric alignment scores appear in Table 4. Please correct the cross-reference.
  3. [§1] There is a typo: 'i.e,.SWE-bench Lite' should be 'i.e., SWE-bench Lite'.
  4. [§4.4.2] The inference setup says 'default temperature' but does not give the temperature value. Please specify the decoding hyperparameters.
  5. [§4.3.4] The max sequence length of 64 tokens seems very short for natural-language feedback plus corrected code. Please clarify whether this is an input or output truncation value and whether it affects the evaluation.

Circularity Check

1 steps flagged · score 6.0 of 10

Feedback-alignment evaluation is a closed loop: GPT-4's seven-dimension rubric both creates the training labels (§4.2.3) and scores the final outputs (§4.4.3), so the reported alignment gains measure imitation of the labeling judge rather than demonstrated developer preference. The abstract's 200-developer human study appears nowhere in the body, which defers human evaluation to future work.

  1. self definitional [§4.2.3 Pair construction and §4.4.3 Feedback Alignment Evaluation; Eq. (4) in §4.3.3]
    "we used GPT-4 to score each feedback instance across all metrics ... A feedback instance is labeled accepted (i.e., preferred) if its revised code executes successfully, passes all unit tests, and achieves an average quality score of at least 4.0 across the seven metrics; otherwise, it is labeled rejected. ... We employed an LLM (i.e., GPT-4) to rate each feedback item on the seven metrics in Table 1 using a 1–5 scale (higher is better). ... An overall G-Eval score is the mean of the seven metric scores per item."

    The same GPT-4 seven-dimension rubric produces the training labels (accepted/rejected) and the G-Eval alignment scores used as the outcome. Eq. (4) optimizes the policy to increase the combined margin for accepted over rejected pairs, i.e., to imitate that rubric. The reported 'feedback alignment' improvements therefore show that DPO-f+ tracks the labeling judge, not that real developers prefer it. The paper's own limitation (§6.2) calls LLM-as-a-judge 'a proxy' and says 'future work will include direct human evaluation to further validate our findings.'

full rationale

The code-accuracy results are self-contained and not circular: Pass@1 is measured by executing corrected code against unit tests, and SWE-bench Lite is scored by the official evaluator on a training-disjoint split. The circularity is confined to the 'developer preference' alignment claim. In §4.2.3, GPT-4's seven-dimension rubric (plus test execution) decides which feedback is accepted/rejected; Eq. (4) trains DPO-f+ to increase the policy margin on those labels; and in §4.4.3, GPT-4 scores the final outputs on the same seven dimensions to produce G-Eval. Thus the alignment evaluation measures how well the model reproduces the labeling judge's rubric, not whether real developers prefer the outputs. The paper's own §6.2 concedes that LLM-as-a-judge is 'a proxy' and that 'future work will include direct human evaluation to further validate our findings.' Separately, and not as a circularity, the abstract's claim of 'a human study with 200 developers' (71.5% preference, p=0.0057) is not reported in the body; the only human check described is '100 feedback cases' with '95% inter-annotator agreement' (§4.2.3, §6.2), and §7.1 again defers human studies. That makes the headline human-preference evidence unsupported, while the remaining alignment evidence is the rubric loop described above. No load-bearing self-citation or imported uniqueness theorem is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of LLM-judge scores as developer-preference proxies and on several hand-chosen training choices (λ, γ, acceptance threshold). No new physical or conceptual entities are introduced; 'DPO-f+' is a training objective rather than an invented entity.

free parameters (3)
  • λ(x) data-dependent reward weight = unspecified
    Appears in Eq. (3) and Eq. (4) as the weight on the frozen reward-model score. No functional form or value is given, and the central training loss depends on it.
  • KL regularization weight γ = 0 or 0.02 (reported for DPO)
    The paper reports γ∈{0, 0.02} for DPO and says the same criterion was used for model selection; the value actually used for DPO-f+ is not stated.
  • Acceptance threshold for preferred feedback = 4.0
    In §4.2.3, a feedback instance is preferred only if its code executes, passes tests, and its average GPT-4 rubric score is at least 4.0. This threshold is hand-chosen and shifts the entire preference dataset.
assumptions (4)
  • domain assumption GPT-4/DeepSeek LLM judgments approximate human ratings of feedback quality
    Used in §4.2.3 to label accepted/rejected pairs and in §4.4.3 to score alignment. Only a 100-case human calibration is reported; the body says direct human evaluation is future work.
  • ad hoc to paper The seven fixed rubric dimensions are the right developer-preferred qualities for code-repair feedback
    Defined in §4.1 and Table 1 based on prior studies, but the choice of dimensions and equal weighting is not validated independently for the developer profiles used here.
  • standard math Standard DPO/RLHF pairwise-loss mathematics is valid for this setting
    Eqs. (1)-(4) rely on the log-sigmoid pairwise objective and KL regularization from Rafailov et al. [29]; this is accepted background.
  • domain assumption Code augmentation preserves feedback-relevant properties of the original programs
    The data pipeline in §4.2.1 expands 53 scripts to 534 via compression and identifier modification; if augmented scripts are unrealistic, the preference dataset and evaluation may not transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DPO-F+: Aligning Code Repair Feedback with Developers' Preferences." pith.science (2026). https://pith.science/paper/EYXDT3HH

@misc{pith2026251101043,
  author       = {Pith},
  title        = {Pith review of: DPO-F+: Aligning Code Repair Feedback with Developers' Preferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EYXDT3HH}},
  note         = {Machine review of arXiv:2511.01043}
}
abstract

Large Language Models (LLMs) are increasingly used in software engineering tasks, especially code repair. However, developers often struggle to interpret model outputs, limiting effective human--AI teaming, where humans and AI work toward a shared objective. Prior work mainly optimizes generated code, giving less attention to natural-language feedback that supports comprehension and iterative improvement. We present \textsc{DPO-f+}, a framework that aligns code-repair feedback with the needs of different developer groups, including novices and proficient developers. It (1) defines feedback-alignment metrics across seven fixed dimensions with task-specific descriptions; (2) automatically constructs pairwise preference datasets from code-repair tasks; (3) fine-tunes models using Direct Preference Optimization (DPO) augmented with a reward model; and (4) provides an automated protocol for evaluating feedback quality. Empirically, \textsc{DPO-f+} outperforms both the baseline and standard DPO in feedback accuracy and overall alignment. On novice programming tasks, \textsc{DPO-f+} improves Pass@1 by 5.71 percentage points (pp) over the baseline and 3.30 pp over DPO. On \textsc{SWE-Bench}, it improves issue-resolution rate by 1.67 pp over DPO and 4.67 pp over the baseline. It also improves feedback alignment by both LLM judges and a human study with 200 developers: beginners preferred \textsc{DPO-f+} in 71.5% of comparisons, with overall preference above chance ($p=0.0057$). By better aligning feedback with developer needs, \textsc{DPO-f+} turns LLM assistance from a one-shot output into a collaborative sense-making workflow, enhancing code comprehension and human--AI teaming in software engineering.

Figures

Figures reproduced from arXiv: 2511.01043 by the authors.

Figure 1
Figure 1. Comparative feedback to the same MinStack implementation produced by Baseline, DPO, and DPO-f+. Colors link each identified issue to its corresponding fix. yielding a defined and testable failure mode. The guidance is imper￾ative and stepwise, with short what and why clauses that make each action concrete and verifiable. It orders the fixes logically (types, then bounds, then guards). Each edit is also tied to a cle… view at source ↗
Figure 2
Figure 2. The overview of DPO-f+ framework. preference pairs, then present the reward-augmented DPO training procedure, and finally describe the evaluation protocol. We aim to answer the following research questions: (1) Does DPO-f+ improve the alignment of code-repair feedback for novice programming instruction, relative to Baseline and DPO? (2) Does DPO-f+ maintain strong performance on more chal￾lenging code-repair tasks? … view at source ↗
Figure 3
Figure 3. Three distinct prompts used during the feedback [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The example of prompt used to generate feedback [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering

    cs.SE 2026-08 conditional novelty 6.0 of 10

    AgentForge lets novices practice software repair by playing one of four roles alongside AI agents; in a 37-person study, completion and self-reported learning were high, while code review proved the hardest role.

  2. From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software

    cs.SE 2026-07 conditional novelty 6.0 of 10

    In OSS repos that commit AI chat logs, AI use is heavier in smaller, less collaborative projects; chats almost always precede commits, quality signals do not broadly worsen, and developers trust their own AI code more...

Reference graph

Works this paper leans on

54 extracted references · 12 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. InProceedings of the 2019 chi conference on human factors in computing systems. 1–13

    2019

  2. [2]

    Sara Mernissi Arifi, Ismail Nait Abdellah, Azeddine Zahi, and Rachid Benabbou

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    arXiv 2021

  4. [4]

    Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023. Grounded copilot: How programmers interact with code-generating models.Proceedings of the ACM on Programming Languages7, OOPSLA1 (2023), 85–111

    2023

  5. [5]

    Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of useful code reviews: An empirical study at microsoft. In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 146–156

    2015

  6. [6]

    Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. 2024. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms.Comput. Surveys(2024)

    2024

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    arXiv 2021

  8. [8]

    Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, and Yong Liu

Show all 54 references
  1. [9]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1469–1481

    2023

  2. [10]

    Zihan Fang, Jiliang Li, Anda Liang, Gina R Bai, and Yu Huang. 2025. A Compara- tive Study on ChatGPT and Checklist as Support Tools for Unit Testing Education. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 871–882

    2025

  3. [11]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999 (2022)

    2022 arXiv

  4. [12]

    Mark Grechanik, Chen Fu, and Qing Xie. 2012. Automatically finding perfor- mance problems with feedback-directed learning software testing. In2012 34th international conference on software engineering (ICSE). IEEE, 156–166

    2012

  5. [13]

    John Grundy, Hourieh Khalajzadeh, and Jenny McIntosh. 2020. Towards human- centric model-driven software engineering. InInternational Conference on Evalu- ation of Novel Approaches to Software Engineering 2020. Scitepress, 299–238

    2020

  6. [14]

    Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph E Gonzalez, and Ion Stoica. 2020. Contrastive code representation learning.arXiv preprint arXiv:2007.04973(2020)

    2020 arXiv

  7. [15]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    2023 arXiv

  8. [16]

    Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. Beyond code generation: An observational study of chatgpt usage in software engineering practice.Proceedings of the ACM on Software Engineering1, FSE (2024), 1819–1840

    2024

  9. [17]

    Fabian Kortum, Jil Klünder, and Kurt Schneider. 2019. Behavior-driven dynamics in agile development: The effect of fast feedback on teams. In2019 IEEE/ACM International Conference on Software and System Processes (ICSSP). IEEE, 34–43

    2019

  10. [18]

    Heiko Koziolek, Andreas Burger, Marie Platenius-Mohr, Julius Rückert, Hadil Abukwaik, Raoul Jetley, and Abdulla P P. 2020. Rule-based code generation in industrial automation: four large-scale case studies applying the cayenne method. InProceedings of the ACM/IEEE 42nd Interna...

    2020

  11. [19]

    Changshu Liu, Pelin Cetin, Yogesh Patodia, Baishakhi Ray, Saikat Chakraborty, and Yangruibo Ding. 2024. Automated code editing with search-generate-modify. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 398–399

    2024

  12. [20]

    Dominic Lohr, Hieke Keuning, and Natalie Kiesler. 2025. You’re (Not) My Type- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?Journal of Computer Assisted Learning41, 1 (2025), e13107

    2025

  13. [21]

    Bowen Lou, Tian Lu, TS Raghu, and Yingjie Zhang. 2025. Unraveling human-ai teaming: A review and outlook.arXiv preprint arXiv:2504.05755(2025)

    2025 arXiv

  14. [22]

    Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. Llama-reviewer: Ad- vancing code review automation with large language models through parameter- efficient fine-tuning. In2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 647–658

    2023

  15. [23]

    Srijoni Majumdar, Ayush Bansal, Partha Pratim Das, Paul D Clough, Kausik Datta, and Soumya Kanti Ghosh. 2022. Automated evaluation of comments to aid software maintenance.Journal of Software: Evolution and Process34, 7 (2022), e2463

    2022

  16. [24]

    Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the robustness of code generation techniques: An empirical study on github copilot. In2023 IEEE/ACM 45th International Conference on Soft...

    2023

  17. [25]

    Marcus Messer, Neil CC Brown, Michael Kölling, and Miaojing Shi. 2024. Au- tomated grading and feedback tools for programming education: A systematic review.ACM Transactions on Computing Education24, 1 (2024), 1–43

    2024

  18. [26]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (...

    2022

  19. [27]

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies.Transactions of the Association for Computational Linguistics12 (...

    2024

  20. [28]

    Luca Pascarella, Davide Spadini, Fabio Palomba, Magiel Bruntink, and Alberto Bacchelli. 2018. Information needs in contemporary code review.Proceedings of the ACM on human-computer interaction2, CSCW (2018), 1–27

    2018

  21. [29]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741

    2023

  22. [30]

    Abdul Razzaq, Jim Buckley, Qin Lai, Tingting Yu, and Goetz Botterweck. 2024. A systematic literature review on the influence of enhanced developer experience on developers’ productivity: Factors, practices, and recommendations.Comput. Surveys57, 1 (2024), 1–46

    2024

  23. [31]

    Alexander Scarlatos, Digory Smith, Simon Woodhead, and Andrew Lan. 2024. Im- proving the validity of automatically generated feedback via reinforcement learn- ing. InInternational Conference on Artificial Intelligence in Education. Springer, 280–294

    2024

  24. [32]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  25. [33]

    Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé F Bissyandé. 2024. Codeagent: Autonomous communicative agents for code review.arXiv preprint arXiv:2402.02172(2024)

    2024 arXiv

  26. [34]

    Xunzhu Tang, Jacques Klein, and Tegawendé F Bissyandé. 2025. Boosting Open- Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Rein- forcement Learning.arXiv preprint arXiv:2506.03921(2025)

    2025 arXiv

  27. [35]

    Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. 2022. Using pre-trained models to boost code review automation. InProceedings of the 44th international conference on software engineering. 2291–2302

    2022

  28. [36]

    Asif Kamal Turzo, Fahim Faysal, Ovi Poddar, Jaydeb Sarker, Anindya Iqbal, and Amiangshu Bosu. 2023. Towards automated classification of code review feedback to support analytics. In2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). ...

    2023

  29. [37]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a- judge in software engineering.Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1955–1977

    2025

  30. [38]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation.arXiv preprint arXiv:2109.00859(2021)

    2021 arXiv

  31. [39]

    Patricia Widjojo and Christoph Treude. 2023. Addressing compiler errors: Stack overflow or large language models?arXiv preprint arXiv:2307.10793(2023)

    2023 arXiv

  32. [40]

    Ratnadira Widyasari, Ting Zhang, Abir Bouraffa, Walid Maalej, and David Lo

  33. [41]

    Juliette Woodrow, Sanmi Koyejo, and Chris Piech. 2025. Improving generative ai student feedback: Direct preference optimization with teachers in the loop

    2025

  34. [42]

    Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. [n. d.]. AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization. InForty-second International Conference on Machine Learning

  35. [43]

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719(2024)

    2024 arXiv

  36. [44]

    Explaining explanations: An empirical study of explanations in code reviews.ACM Transactions on Software Engineering and Methodology34, 6 (2025), 1–30

    2025

  37. [45]

    Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. 2023. Evacrc: Evaluating code review comments. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 275–287

    2023

  38. [46]

    Huan Zhang, Wei Cheng, Yuhan Wu, and Wei Hu. 2024. A pair programming framework for code generation via multi-plan exploration and feedback-driven refinement. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1319–1331

    2024

  39. [47]

    Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gus- tavo Soares, and Gust Verbruggen. 2024. Pydex: Repairing bugs in introductory python assignments using llms.Proceedings of the ACM on Programming Lan- guages8, OOPSLA1 (2024), 1100–1124

    2024

  40. [48]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Conference’17, July 2017, Washington, DC, US...

    2024

  41. [49]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

    2023

  42. [50]

    Yangtian Zi, Luisa Li, Arjun Guha, Carolyn Anderson, and Molly Q Feldman. 2025. “I Would Have Written My Code Differently’: Beginners Struggle to Understand LLM-Generated Code. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1479–1488

    2025

  43. [52]

    Kechi Zhang, Ge Li, Jia Li, Yihong Dong, and Zhi Jin. 2025. Focused-dpo: Enhanc- ing code generation through focused preference optimization on error-prone points.arXiv preprint arXiv:2502.11475(2025)

    2025

  44. [2015]

    In2015 Third World Conference on Complex Systems (WCCS)

    Automatic program assessment using static and dynamic analysis. In2015 Third World Conference on Complex Systems (WCCS). IEEE, 1–6

  45. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  46. [2025]

    An empirical study on challenges for llm application developers.ACM Transactions on Software Engineering and Methodology(2025)

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.