{"id":"f71f23eb-676c-42e3-b20e-de60e1876301","arxiv_id":"2506.02211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding a hand-built code-quality score to the GRPO reward makes a 3B model produce code that blind raters prefer, but evidence rests on one model and a synthetic benchmark.","lead":"The authors train small language models with GRPO, adding an automated code-quality scorer to the standard test-passing reward. The quality-trained Qwen model produced code that blind raters preferred 78.6% of the time, but correctness actually dropped for one of the three models and no standard benchmarks were used.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conclusion overclaims no correctness degradation: OLMo's validation correctness drops from 0.305 to 0.217, directly contradicting the paper's stated claim that quality rewards preserve correctness.","rationale":"The reader's strongest_claim correctly flags the OLMo correctness drop as an overreach, and I agree that this is a serious problem. However, the reader's weakest_assumption focuses on the hand-tuned codequal_analyzer metric's validity. That is a legitimate external-validity concern, but it is conditional: it would matter even if the reported numbers were internally consistent. The correctness drop is a direct contradiction between the text and the paper's own Table 2, so it is the most load-bearing issue for the central claim: if the claimed no-degradation property fails, the deployment argument collapses regardless of how valid the quality metric is. Since the discrepancy is fixable by scoping the claim or adding replicates, and the human-preference result for Qwen provides partial independent support, the correct outcome remains a conditional assessment rather than a rejection. I mark agreement as partial because I adopt the reader's strongest-claim concern but not their chosen weakest assumption.","tokens_in":9783,"tokens_out":3896,"duration_ms":35923,"concrete_test":"Re-run the OLMo 2 1B Instruct GRPO condition with and without the quality reward using at least 3 seeds and the reported hyperparameters, then compare held-out validation correctness at matched training steps with confidence intervals. If the 0.305 to 0.217 drop persists beyond noise, the paper's 'without degradation' claim is empirically false and must be revised; if it does not persist, the current Table 2 presentation still needs correction before the claim can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central pitch, repeated in the introduction and conclusion, is that adding the codequal_analyzer reward improves code quality 'without any degradation in the functional correctness' and that the quality-trained model 'performed as well or even better' than the control on correctness. Table 2 contradicts this for OLMo 2 1B Instruct: validation correctness falls from 0.305 to 0.217, a roughly 29 percent relative drop. This is not a matter of external validity or speculative reward hacking; it is an internal inconsistency in the reported results. Qwen and Llama correctness are improved or flat, so the method may still be useful, but the general claim 'without sacrificing correctness' is false for one of the three evaluated models. The human-preference evidence covers only Qwen, so it cannot rescue the OLMo correctness claim. The load-bearing issue is that a reader cannot accept the abstract and conclusion as written: the paper should either scope the claim to Qwen and Llama, report a trade-off, or provide additional evidence that the OLMo drop is noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces codequal_analyzer, a composite code-quality scorer based on CISQ-style weakness categories, and uses it as a reward component in a GRPO pipeline for code generation. The reward is a weighted sum of a format score, a held-out unit-test correctness score, and the code-quality score. The authors train Qwen 2.5 3B, Llama 3.2 3B, and OLMo 2 1B with and without the quality reward, reporting higher validation quality scores for all three models, higher or comparable correctness for two of them, and a blinded human preference for Qwen output (78.6%, p<0.001). The paper's central claim is that GRPO with a code-quality reward improves code quality without sacrificing functional correctness.","tokens_in":9997,"tokens_out":5874,"duration_ms":51594,"significance":"If the claim held in general, this would be a useful contribution: it would provide an automated, RL-compatible quality signal and a concrete open-source library (codequal_analyzer) for other researchers. The blinded human-preference experiment is a genuine independent check, and basic statistics are reported. However, the primary quality metric is the very same scorer used as the reward, the human check covers only one model and its annotation protocol is underspecified, and one of the three models shows a large correctness drop. These issues currently limit the generality of the central claim, but they are addressable.","major_comments":[{"comment":"The claim that adding the quality reward works 'without any degradation in the functional correctness' is contradicted by the OLMo row of Table 2: validation correctness drops from 0.305 to 0.217, a relative decrease of about 29%. No confidence intervals or multiple seeds are reported, so it is impossible to tell whether this is noise or a genuine trade-off. The conclusion as written is too strong; please scope the correctness claim to Qwen and Llama, report uncertainty about the OLMo result, or present the result as a trade-off.","section":"§4 (Table 2) and §1/§5"},{"comment":"The primary quality evaluation uses codequal_analyzer, which is the same scorer used as the reward signal, so the reported quality improvement is partly the expected consequence of optimizing that exact objective. The severity weights and the 1/(1+W) functional form are chosen by hand and are not calibrated against any external standard. An independent evaluation of code quality, such as a second analyzer not used in training or a larger blind human study across models, is needed to establish that 'quality' itself improved rather than just the reward score.","section":"§3.1 and §4 (Table 2, Fig. 3)"},{"comment":"The blinded human-preference experiment was performed only on Qwen 2.5 3B, and the manuscript does not report the number of annotators, their qualifications, or inter-annotator agreement. The abstract calls the annotators 'expert,' but the methods text only says they were told to 'Choose which code snippet you think is of higher quality.' Since this is the only independent evidence for the quality improvement, please provide the full annotation protocol and, ideally, extend the preference study to at least one more model.","section":"§4 (Human Preference)"},{"comment":"The 'Total Reward' column is inconsistent with the stated linear combination r_i = 0.2 r_format + 0.3 r_correct + 0.5 r_quality. For example, using the Table 2 deltas for OLMo (Δcorrectness = -0.088, Δquality = +0.073, Δtotal = +0.221), the implied change in the format reward would be about 1.05, which is impossible because r_format is bounded in [0,1]. The table, the reward definition, or the reward computation needs to be corrected, and the format-reward values should be reported.","section":"§3.3, Eq. (2) and Table 2"}],"minor_comments":[{"comment":"The sentence beginning 'As a concrete starting point, considered the list' is missing a subject; it should read 'we considered the list'.","section":"§1"},{"comment":"The displayed expression for D_KL involving π_ref/π_θ − log(π_ref/π_θ) − 1 is an unbiased estimator of the KL divergence rather than the KL divergence itself; please clarify the notation.","section":"§3.2"},{"comment":"The training curves show no error bars or seed variability; please state the number of independent runs used to generate the plots.","section":"Figure 3"},{"comment":"There is a typo in 'Y AML' under Security, and the entry 'Functions with high complexity scores' is ambiguous about which analyzer produced it.","section":"Table 1"},{"comment":"The synthetic dataset contains only 200 problems, but the train/validation split is not specified throughout the text; this limits the precision of the correctness estimates and should be stated explicitly.","section":"§3.4"},{"comment":"The note that the side-by-side examples 'may not be functionally identical, with some potentially containing bugs' weakens the qualitative comparison; please mark which examples are functionally correct.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's real contribution is a concrete, CISQ-rooted code-quality scoring library used as a GRPO reward, plus a blind human preference test on Qwen (78.6% preferring the quality-trained model). That human result is a genuine external signal, and the side-by-side examples show real style improvements. The library is substantial work and the paper is clear about its reward design.\n\nWhat's new is the breadth of the quality scorer as a reward, not the concept: ACECode already used efficiency as a quality facet. Extending to maintainability, security, and reliability, and packaging it as a reusable library, is a reasonable contribution.\n\nThe soft spots, in order of severity. First, the abstract and conclusion claim improved quality \"without any degradation in functional correctness,\" but Table 2 shows OLMo 2 1B validation correctness dropping from 0.305 to 0.217, a roughly 29% relative drop. That is an internal inconsistency. The claim needs to be scoped to Qwen and Llama, or reframed as a trade-off. Second, the primary evaluation metric is the same codequal_analyzer score used as the reward, so the quality gain is partly the model optimizing its own reward. The human test partially rescues this, but covers only Qwen and annotator expertise is unspecified. The hand-tuned severity weights and 1/(1+W) form are not fatal—any aggregation has to start somewhere—but an ablation or external validation of the score would help. The synthetic dataset is small (200 problems) and not fully released, and the evidence covers only three small models. Security, a headline category, isn't evaluated separately.\n\nThese are addressable issues, not a reason to dismiss the approach. The method plausibly works; the paper just needs to report the correctness result accurately and broaden the human evaluation.\n\nAudience: people doing RL for code generation, especially reward design. Worth a serious referee: I would send it out, but with a clear request to fix the correctness overclaim, add a human eval on at least one more model, and release the dataset and eval harness.","headline":"A solid incremental RL-for-code-quality result that overclaims on correctness: OLMo's validation correctness drops 0.305→0.217, contradicting the abstract.","tokens_in":10525,"tokens_out":1409,"would_cite":false,"duration_ms":15347,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A code-quality reward added to GRPO training makes LLM code better by human judgment, not just by unit tests.","keywords":["code quality","GRPO","reinforcement learning","large language models","code generation","reward design","CISQ","static analysis"],"falsifier":"Run the same training recipe on additional models and have a diverse panel of professional programmers rate outputs on security and maintainability separately; if the automated quality score improves while human-preferred quality does not, or if security-critical issues such as hard-coded secrets or unsafe deserialization become more frequent, the reward proxy is failing. A cheaper observation already available in the paper is the OLMo correctness drop, which contradicts an unconditional no-trade-off reading of the central claim.","tokens_in":9579,"feed_emoji":"💻","tokens_out":9291,"duration_ms":72159,"temperature":0.7,"pith_summary":"The paper argues that execution-feedback RL for code generation, which rewards only whether unit tests pass, misses maintainability, security, reliability, and performance. To close that gap, it builds a static-analysis-based quality scorer, codequal_analyzer, and uses the resulting score as a reward term in GRPO. On three open language models, adding this term raises automated quality scores on held-out problems, and blinded expert human annotators preferred the quality-trained model's output in 78.6% of pairwise comparisons. The paper's central claim is that code quality can be improved without sacrificing functional correctness, though the reported correctness results are mixed across the three models.","feed_headline":"Code-quality reward wins blind expert preference 78.6% of the time","feed_subtitle":"Blinded experts preferred quality-trained model output in 78.6% of 159 comparisons.","key_machinery":"The load-bearing object is codequal_analyzer, a library that runs static analyzers covering maintainability, security, performance, and reliability, assigns each finding a severity in {info, low, medium, high, critical} with weights 0.5, 1.0, 2.5, 5.0, 10.0, and collapses the weighted count $W$ into a reward $r_{quality} = 1/(1+W)$. This reward is combined with a format reward and a unit-test pass-rate reward as $r = 0.2 r_{format} + 0.3 r_{correct} + 0.5 r_{quality}$, and GRPO group-relative advantages are computed from these total rewards. The paper also contributes a synthetic dataset of code-editing problems designed to expose quality-relevant issues that standard benchmarks like MBPP, HumanEval, and APPS do not differentiate.","core_discovery":"The central discovery is that a single scalar reward derived from CISQ-aligned static analyzers, inserted into the GRPO objective alongside format and test-pass rewards, shifts the policy toward code that human experts judge to be better. The effect is large on the model where it was measured: 78.6% preference for the quality-trained model over the control, with a 95% confidence interval of 71.6% to 84.3%. Automated quality scores improved for all three models, while correctness improved or stayed flat for two of the three and dropped for the third.","pith_inferences":["The severity weights and the reward form are hand-chosen; a principled calibration against pairwise human judgments or professional code-review scores could shift the optimal policy and should precede deployment.","With a 50% weight on quality and 30% on correctness, the objective explicitly trades test-passing against clean code; the OLMo correctness drop suggests a Pareto frontier that should be measured and reported.","Because the reward is deterministic given a generated program, the same analyzer could serve as a post-hoc filter or reranker over candidate outputs, which may be cheaper than RL training.","The current analyzers are Python-specific, so the results do not automatically transfer to other programming languages without porting the CISQ issue taxonomy."],"forward_implications":["The quality reward can be added to existing execution-feedback RL pipelines as one extra scalar term, at negligible additional compute during training.","The generated code tends to be shorter when the quality reward is present, so deployment costs could drop even before downstream maintainability benefits appear.","A CISQ-aligned analyzer suite can serve as a reusable reward signal for other LLM coding tasks, not just the three models studied here.","The synthetic dataset design, which targets quality-relevant categories such as dead code, security, and loop efficiency, may be reused for future code-quality RL datasets."],"supporting_citations":[{"why":"Supplies the GRPO algorithm whose objective carries the whole training pipeline.","marker":"[Shao et al., 2024]"},{"why":"Defines the ISO 5055 standards and weaknesses that codequal_analyzer turns into a reward.","marker":"[CISQ, 2025]"},{"why":"Supplies the clipped GRPO hyperparameters (epsilon_low=0.2, epsilon_high=0.28) used in all runs.","marker":"[Yu et al., 2025]"},{"why":"Provides the style and structure analyzers that feed the issue counts in codequal_analyzer.","marker":"[The Pylint Team]"},{"why":"Provides the Bandit security analyzers that detect injection and unsafe-deserialization issues in the reward signal.","marker":"[Python Code Quality Authority]"},{"why":"Provides the Vulture analyzer for dead code, a major maintainability component of the quality score.","marker":"[Seipp et al.]"}],"fun_headline_variants":["GRPO quality reward wins blind expert 78.6%","Code-quality GRPO beats standard 78.6% in expert blind","78.6% expert preference for GRPO quality-trained code","Beyond tests: GRPO code-quality reward boosts expert preference","Quality reward in GRPO flips human preference to 78.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-tuned severity weights and the particular reward formula faithfully capture what humans mean by code quality, with the human-preference check run only on one of the three models.","fun_headline_variants_meta":{"raw":{"variants":["GRPO quality reward wins blind expert 78.6%","Code-quality GRPO beats standard 78.6% in expert blind","78.6% expert preference for GRPO quality-trained code","Beyond tests: GRPO code-quality reward boosts expert preference","Quality reward in GRPO flips human preference to 78.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00101,"raw_usage":{"total_tokens":4168,"prompt_tokens":746,"completion_tokens":3422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":362,"completion_tokens_details":{"reasoning_tokens":3332}},"tokens_in":362,"tokens_out":3422,"duration_ms":23766,"temperature":1.0,"reasoning_tokens":3332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:27:58.184813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training recipe on additional models and have a diverse panel of professional programmers rate outputs on security and maintainability separately; if the automated quality score improves while human-preferred quality does not, or if security-critical issues such as hard-coded secrets or unsafe deserialization become more frequent, the reward proxy is failing. A cheaper observation already available in the paper is the OLMo correctness drop, which contradicts an unconditional no-trade-off reading of the central claim.","supporting_citations":[],"review_version":1}