Improving LLM-Generated Code Quality with GRPO

Laurence Aitchison; Maxime Robeyns

REVIEW 4 major objections 6 minor 1 cited by

Improving LLM-Generated Code Quality with GRPO

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A code-quality reward added to GRPO training makes LLM code better by human judgment, not just by unit tests.

desk verdict A solid incremental RL-for-code-quality result that overclaims on correctness: OLMo's validation correctness drops 0.305→0.217, contradicting the abstract. read the letter →

arxiv 2506.02211 v1 pith:GEDMB5JF submitted 2025-06-02 cs.AI

Maxime Robeyns , Laurence Aitchison This is my paper

classification cs.AI

keywords codequalityGRPOreinforcementlearninglargelanguagemodelsgenerationrewarddesignCISQstaticanalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that execution-feedback RL for code generation, which rewards only whether unit tests pass, misses maintainability, security, reliability, and performance. To close that gap, it builds a static-analysis-based quality scorer, codequal_analyzer, and uses the resulting score as a reward term in GRPO. On three open language models, adding this term raises automated quality scores on held-out problems, and blinded expert human annotators preferred the quality-trained model's output in 78.6% of pairwise comparisons. The paper's central claim is that code quality can be improved without sacrificing functional correctness, though the reported correctness results are mixed across the three models.

What carries the argument

The load-bearing object is codequal_analyzer, a library that runs static analyzers covering maintainability, security, performance, and reliability, assigns each finding a severity in {info, low, medium, high, critical} with weights 0.5, 1.0, 2.5, 5.0, 10.0, and collapses the weighted count $W$ into a reward $r_{quality} = 1/(1+W)$. This reward is combined with a format reward and a unit-test pass-rate reward as $r = 0.2 r_{format} + 0.3 r_{correct} + 0.5 r_{quality}$, and GRPO group-relative advantages are computed from these total rewards. The paper also contributes a synthetic dataset of code-editing problems designed to expose quality-relevant issues that standard benchmarks like MBPP, HumanEval, and APPS do not differentiate.

What would settle it

Run the same training recipe on additional models and have a diverse panel of professional programmers rate outputs on security and maintainability separately; if the automated quality score improves while human-preferred quality does not, or if security-critical issues such as hard-coded secrets or unsafe deserialization become more frequent, the reward proxy is failing. A cheaper observation already available in the paper is the OLMo correctness drop, which contradicts an unconditional no-trade-off reading of the central claim.

Watch

Extended reading notes

Core claim

The central discovery is that a single scalar reward derived from CISQ-aligned static analyzers, inserted into the GRPO objective alongside format and test-pass rewards, shifts the policy toward code that human experts judge to be better. The effect is large on the model where it was measured: 78.6% preference for the quality-trained model over the control, with a 95% confidence interval of 71.6% to 84.3%. Automated quality scores improved for all three models, while correctness improved or stayed flat for two of the three and dropped for the third.

Load-bearing premise

The load-bearing premise is that the hand-tuned severity weights and the particular reward formula faithfully capture what humans mean by code quality, with the human-preference check run only on one of the three models.

Editorial extensions

If this is right

The quality reward can be added to existing execution-feedback RL pipelines as one extra scalar term, at negligible additional compute during training.
The generated code tends to be shorter when the quality reward is present, so deployment costs could drop even before downstream maintainability benefits appear.
A CISQ-aligned analyzer suite can serve as a reusable reward signal for other LLM coding tasks, not just the three models studied here.
The synthetic dataset design, which targets quality-relevant categories such as dead code, security, and loop efficiency, may be reused for future code-quality RL datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The severity weights and the reward form are hand-chosen; a principled calibration against pairwise human judgments or professional code-review scores could shift the optimal policy and should precede deployment.
With a 50% weight on quality and 30% on correctness, the objective explicitly trades test-passing against clean code; the OLMo correctness drop suggests a Pareto frontier that should be measured and reported.
Because the reward is deterministic given a generated program, the same analyzer could serve as a post-hoc filter or reranker over candidate outputs, which may be cheaper than RL training.
The current analyzers are Python-specific, so the results do not automatically transfer to other programming languages without porting the CISQ issue taxonomy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper's real contribution is a concrete, CISQ-rooted code-quality scoring library used as a GRPO reward, plus a blind human preference test on Qwen (78.6% preferring the quality-trained model). That human result is a genuine external signal, and the side-by-side examples show real style improvements. The library is substantial work and the paper is clear about its reward design.

What's new is the breadth of the quality scorer as a reward, not the concept: ACECode already used efficiency as a quality facet. Extending to maintainability, security, and reliability, and packaging it as a reusable library, is a reasonable contribution.

The soft spots, in order of severity. First, the abstract and conclusion claim improved quality "without any degradation in functional correctness," but Table 2 shows OLMo 2 1B validation correctness dropping from 0.305 to 0.217, a roughly 29% relative drop. That is an internal inconsistency. The claim needs to be scoped to Qwen and Llama, or reframed as a trade-off. Second, the primary evaluation metric is the same codequal_analyzer score used as the reward, so the quality gain is partly the model optimizing its own reward. The human test partially rescues this, but covers only Qwen and annotator expertise is unspecified. The hand-tuned severity weights and 1/(1+W) form are not fatal—any aggregation has to start somewhere—but an ablation or external validation of the score would help. The synthetic dataset is small (200 problems) and not fully released, and the evidence covers only three small models. Security, a headline category, isn't evaluated separately.

These are addressable issues, not a reason to dismiss the approach. The method plausibly works; the paper just needs to report the correctness result accurately and broaden the human evaluation.

Audience: people doing RL for code generation, especially reward design. Worth a serious referee: I would send it out, but with a clear request to fix the correctness overclaim, add a human eval on at least one more model, and release the dataset and eval harness.

Referee Report

4 major / 6 minor

Summary. The paper introduces codequal_analyzer, a composite code-quality scorer based on CISQ-style weakness categories, and uses it as a reward component in a GRPO pipeline for code generation. The reward is a weighted sum of a format score, a held-out unit-test correctness score, and the code-quality score. The authors train Qwen 2.5 3B, Llama 3.2 3B, and OLMo 2 1B with and without the quality reward, reporting higher validation quality scores for all three models, higher or comparable correctness for two of them, and a blinded human preference for Qwen output (78.6%, p<0.001). The paper's central claim is that GRPO with a code-quality reward improves code quality without sacrificing functional correctness.

Significance. If the claim held in general, this would be a useful contribution: it would provide an automated, RL-compatible quality signal and a concrete open-source library (codequal_analyzer) for other researchers. The blinded human-preference experiment is a genuine independent check, and basic statistics are reported. However, the primary quality metric is the very same scorer used as the reward, the human check covers only one model and its annotation protocol is underspecified, and one of the three models shows a large correctness drop. These issues currently limit the generality of the central claim, but they are addressable.

major comments (4)

[§4 (Table 2) and §1/§5] The claim that adding the quality reward works 'without any degradation in the functional correctness' is contradicted by the OLMo row of Table 2: validation correctness drops from 0.305 to 0.217, a relative decrease of about 29%. No confidence intervals or multiple seeds are reported, so it is impossible to tell whether this is noise or a genuine trade-off. The conclusion as written is too strong; please scope the correctness claim to Qwen and Llama, report uncertainty about the OLMo result, or present the result as a trade-off.
[§3.1 and §4 (Table 2, Fig. 3)] The primary quality evaluation uses codequal_analyzer, which is the same scorer used as the reward signal, so the reported quality improvement is partly the expected consequence of optimizing that exact objective. The severity weights and the 1/(1+W) functional form are chosen by hand and are not calibrated against any external standard. An independent evaluation of code quality, such as a second analyzer not used in training or a larger blind human study across models, is needed to establish that 'quality' itself improved rather than just the reward score.
[§4 (Human Preference)] The blinded human-preference experiment was performed only on Qwen 2.5 3B, and the manuscript does not report the number of annotators, their qualifications, or inter-annotator agreement. The abstract calls the annotators 'expert,' but the methods text only says they were told to 'Choose which code snippet you think is of higher quality.' Since this is the only independent evidence for the quality improvement, please provide the full annotation protocol and, ideally, extend the preference study to at least one more model.
[§3.3, Eq. (2) and Table 2] The 'Total Reward' column is inconsistent with the stated linear combination r_i = 0.2 r_format + 0.3 r_correct + 0.5 r_quality. For example, using the Table 2 deltas for OLMo (Δcorrectness = -0.088, Δquality = +0.073, Δtotal = +0.221), the implied change in the format reward would be about 1.05, which is impossible because r_format is bounded in [0,1]. The table, the reward definition, or the reward computation needs to be corrected, and the format-reward values should be reported.

minor comments (6)

[§1] The sentence beginning 'As a concrete starting point, considered the list' is missing a subject; it should read 'we considered the list'.
[§3.2] The displayed expression for D_KL involving π_ref/π_θ − log(π_ref/π_θ) − 1 is an unbiased estimator of the KL divergence rather than the KL divergence itself; please clarify the notation.
[Figure 3] The training curves show no error bars or seed variability; please state the number of independent runs used to generate the plots.
[Table 1] There is a typo in 'Y AML' under Security, and the entry 'Functions with high complexity scores' is ambiguous about which analyzer produced it.
[§3.4] The synthetic dataset contains only 200 problems, but the train/validation split is not specified throughout the text; this limits the precision of the correctness estimates and should be stated explicitly.
[Appendix B] The note that the side-by-side examples 'may not be functionally identical, with some potentially containing bugs' weakens the qualitative comparison; please mark which examples are functionally correct.

Circularity Check

1 steps flagged · score 5.0 of 10

Automated quality gain is the training objective itself; only the human-preference study provides independent support.

self definitional [Section 3.1 (r_quality definition), Section 3.3 Eq. (2), Table 2, Abstract]
"We find GRPO increases code quality according to this measure ... we obtain a score between 0 and 1 using the following formula ... rquality = 1/(1 + W). ... ri = 2/10 ri,format + 3/10 ri,correct + 5/10 ri,quality. ... Table 2: 'Validation Quality' ... + quality reward training ... 0.878(+0.112)."

The quantity reported as 'Validation Quality' in Table 2 is exactly r_quality, the same codequal_analyzer score that is inserted into the GRPO reward with weight 0.5 in Eq. (2). The model trained with the quality reward is therefore being rewarded to maximize the very number that later appears as the evidence of improvement. Calling this an empirical confirmation that GRPO 'increases code quality according to this measure' is a restatement of the training objective, not an independent prediction. The blinded human-preference study is the only non-circular evidence of quality improvement, and it was measured only for Qwen 2.5 3B.

full rationale

The automated quality-score result is circular by construction: the intervention is defined as adding r_quality to the reward, and the main reported outcome is r_quality itself. This does not, by itself, invalidate the paper, because the human-preference experiment is external to the reward and supports the qualitative claim for Qwen. That external evidence is limited to one model and is subjective, but it is real evidence. Separately, the paper's conclusion that quality training causes 'no degradation in functional correctness' is contradicted by its own Table 2 for OLMo 2 1B Instruct (validation correctness drops from 0.305 to 0.217); this is an internal-consistency problem rather than a circularity. No load-bearing self-citation or imported uniqueness theorem appears. Overall, the central automated-metric claim reduces to the reward definition, while the human study gives independent but partial support, so the paper is partially circular rather than fully circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new scientific entities are postulated. The central claim rests on three domain assumptions: the validity of the hand-built quality score, the representativeness of the synthetic dataset, and the validity of the subjective human preference as ground truth.

free parameters (3)

Severity weights w_info, w_low, w_medium, w_high, w_critical = 0.5, 1.0, 2.5, 5.0, 10.0
Chosen by hand in Section 3.1 to weight issue counts in the quality score; no derivation or calibration against external data is provided.
Quality score functional form = r_quality = 1/(1+W)
Arbitrary decaying function chosen in Section 3.1; not derived from any principle or benchmark.
Reward combination weights = format 0.2, correctness 0.3, quality 0.5
Chosen by hand in Section 3.3 with a 'slight emphasis on the code quality'; no ablation or sensitivity analysis is reported.

assumptions (3)

domain assumption Code quality can be quantified by counting issues with the chosen severity weights.
The entire reward is built on this premise; Section 3.1 defines the score but does not validate it against any known ground truth.
domain assumption The synthetic dataset of 200 Gemini-generated problems is representative of real code-editing tasks that LLMs face.
Section 3.4 argues MBPP/HumanEval are too simple, but no evidence is given that the generated problems capture real-world quality concerns.
domain assumption Blind human raters' subjective preference for one snippet is a valid ground truth for code quality.
Section 4 describes the preference test but gives no details on rater expertise, inter-rater reliability, or agreement with the automated score.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving LLM-Generated Code Quality with GRPO." pith.science (2026). https://pith.science/paper/GEDMB5JF

@misc{pith2026250602211,
  author       = {Pith},
  title        = {Pith review of: Improving LLM-Generated Code Quality with GRPO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEDMB5JF}},
  note         = {Machine review of arXiv:2506.02211}
}

read the original abstract

Large Language Models (LLMs) are gaining widespread use for code generation. Recent training procedures use execution feedback as a reward signal, typically focusing on the functional correctness of the code, using unit test pass rate as a reward signal. However, this reward signal fails to capture notions of maintainability, quality and safety of the code produced. We address this under-explored area and develop a comprehensive library to quantify various aspects of code quality, and use it as a reward in GRPO. We find GRPO increases code quality according to this measure, which is confirmed by expert, blinded human annotators.

Figures

Figures reproduced from arXiv: 2506.02211 by the authors.

**Figure 2.** GRPO advantage calculation. In our experiments, we ablate the code quality score to [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Validation code quality score and correctness throughout Qwen 2.5 3B training. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Human preferences of Qwen 2.5 3B output with and without the quality reward signal. We applied GRPO on Llama 3.2 3B Instruct [Team, 2024], Qwen2.5 3B Instruct [Qwen et al., 2025] and Olmo 2 1B Instruct [OLMo et al., 2025] with a dataset of 200 Python coding problems g…

**Figure 5.** Figure 5: Example code from models trained with and without the quality score. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
cs.AI 2026-02 conditional novelty 5.0 of 10

EMO-R3, which combines a three-step emotional reasoning prompt with a reward for the model agreeing with its own image–emotion judgments, raises visual emotion-recognition accuracy by about one point over plain GRPO.

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

Program synthesis with large language models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

arXiv
[3]

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Xuanjing Huang, and Tao Gui. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback. arXiv preprint arXiv:2402.01391,

arXiv
[4]

arXiv preprint arXiv:2410.02089

URL https://openreview.net/forum? id=zPPy79qKWe. arXiv preprint arXiv:2410.02089. Maurice H. Halstead. Elements of Software Science. Elsevier North-Holland,

arXiv
[7]

Python Code Quality Authority

URL http://arxiv.org/abs/2501.00656. Python Code Quality Authority. Bandit: A tool designed to find common security issues in python code. URL https://github.com/PyCQA/bandit. PyUp.io. Safety: Check your python dependencies for known security vulnerabilities. URL https: //github.com/pyupio/safety. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, B...

arXiv
[8]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

URL http://arxiv.org/abs/2412.15115. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs], August

arXiv
[10]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al

Notion Blog. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

arXiv
[13]

PRLCoder: Leveraging Process-Supervised Reinforcement Learning to Enhance Code Generation

Yufan Ye, Ting Zhang, Wenbin Jiang, and Hua Huang. PRLCoder: Leveraging Process-Supervised Reinforcement Learning to Enhance Code Generation. arXiv preprint arXiv:2502.01715,

arXiv
[14]

operation_audit.log

URL http://arxiv. org/abs/2503.14476. 10 A Synthetic Code Problem Examples Our dataset generation is structured across the following problem categories: ’algorithm selection’, ’array manipulation’, ’custom structures’, ’data structure choice’, ’decomposition’, ’edge cases’, ’exception handling’, ’extract function’, ’function composition’, ’graph algorithm...

arXiv

Show all 14 references

[1977]

Measuring coding challenge competence with APPS

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. arXiv preprint arXiv:2105.09938,

arXiv
[2007]

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C.H. Hoi. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. arXiv preprint arXiv:2207.01780,

arXiv
[2017]

Jendrik Seipp et al

URL http://arxiv.org/abs/ 1707.06347. Jendrik Seipp et al. Vulture: Find dead python code. URL https://github.com/jendrikseipp/ vulture. Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Si- mon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Y...

arXiv
[2023]

Iterative self-training for code generation via reinforced re-ranking

Nikita Sorokin, Ivan Sedykh, and Valentin Malykh. Iterative self-training for code generation via reinforced re-ranking. arXiv preprint arXiv:2504.09643,

arXiv
[2024]

The Mypy Team

URL http://arxiv.org/abs/ 2407.21783. The Mypy Team. Mypy: Optional static typing for python. URL https://github.com/python/ mypy. The Pylint Team. Pylint: code analysis for python. Webpage. URL https://pylint.org/. Chengran Yang, Hong Jin Kang, Jieke Shi, and David Lo. ACECod...

arXiv
[2025]

Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang, and Lin Yan

URL https://www.it-cisq.org/standards/. Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang, and Lin Yan. Process supervision-guided policy optimization for code generation. arXiv preprint arXiv:2410.17621,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Program synthesis with large language models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

arXiv

[2] [3]

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Xuanjing Huang, and Tao Gui. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback. arXiv preprint arXiv:2402.01391,

arXiv

[3] [4]

arXiv preprint arXiv:2410.02089

URL https://openreview.net/forum? id=zPPy79qKWe. arXiv preprint arXiv:2410.02089. Maurice H. Halstead. Elements of Software Science. Elsevier North-Holland,

arXiv

[4] [7]

Python Code Quality Authority

URL http://arxiv.org/abs/2501.00656. Python Code Quality Authority. Bandit: A tool designed to find common security issues in python code. URL https://github.com/PyCQA/bandit. PyUp.io. Safety: Check your python dependencies for known security vulnerabilities. URL https: //github.com/pyupio/safety. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, B...

arXiv

[5] [8]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

URL http://arxiv.org/abs/2412.15115. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs], August

arXiv

[6] [10]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al

Notion Blog. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

arXiv

[7] [13]

PRLCoder: Leveraging Process-Supervised Reinforcement Learning to Enhance Code Generation

Yufan Ye, Ting Zhang, Wenbin Jiang, and Hua Huang. PRLCoder: Leveraging Process-Supervised Reinforcement Learning to Enhance Code Generation. arXiv preprint arXiv:2502.01715,

arXiv

[8] [14]

operation_audit.log

URL http://arxiv. org/abs/2503.14476. 10 A Synthetic Code Problem Examples Our dataset generation is structured across the following problem categories: ’algorithm selection’, ’array manipulation’, ’custom structures’, ’data structure choice’, ’decomposition’, ’edge cases’, ’exception handling’, ’extract function’, ’function composition’, ’graph algorithm...

arXiv