Pith. sign in

REVIEW 4 major objections 5 minor 39 references

From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hierarchy-aware penalty improves CWE prediction when delivered as a GRPO reward, and fails when delivered through supervised losses.

desk verdict A real, clearly written result about delivery mechanisms for a hierarchy-aware CWE penalty, with one load-bearing gap: the CWE extraction step is never specified or validated, so the headline number needs tempering before it can be trusted. read the letter →

arxiv 2607.21069 v1 pith:YVMFQLNW submitted 2026-07-23 cs.LG

Muntasir Adnan , Manile Srun , Carlos C. N. Kuhn This is my paper
classification cs.LG
keywords CWEpredictionvulnerabilitydetectionhierarchicalpenaltyGRPOreinforcementlearningdistributionshiftsupervisedfine-tuningPythonsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the ALPHA penalty—a taxonomy-aware scoring function for CWE-level vulnerability prediction—can serve as a training signal, and that its effectiveness depends entirely on how it is delivered to the model. The authors compare supervised fine-tuning, a dual-head classification loss that includes an expected-penalty term, and reinforcement learning via GRPO with a dense reward equal to one minus the normalised penalty. Under distribution shift from SecurityEval to SVEN, both supervised mechanisms regress below the zero-shot baseline, while every GRPO configuration from the base backbone improves; the best configuration reduces the cumulative ALPHA penalty by 27.9% and matches a 4.5x larger teacher. The paper concludes that a hierarchical penalty is squandered unless it reaches the parameters directly, and that RL provides that directness.

What carries the argument

The load-bearing object is the ALPHA penalty graph: a normalised, direction-aware distance over the MITRE CWE hierarchy that penalises a wrong CWE prediction less when it is taxonomically close to the truth, and penalises over-generalisation more than over-specification. The paper's machinery is the contrast between two gradient structures: the supervised expected-penalty term L_ALPHA = sum_v p(v|x) Phat(v,c_true), whose gradient on the true class is p(c_true)L_ALPHA and vanishes when the model is confident and wrong, versus GRPO's group-relative advantage with reward R = 1 - Phat(c_hat, c_true), whose policy gradient scales as 1/pi_theta and remains corrective. The KL anchor to the base ref

What would settle it

Run the same GRPO pipeline with the reward computed by a validated parser or directly from the model's internal classification head; if the 27.9% improvement disappears, the gains are extraction artifacts. Or train a binary exact-match reward GRPO baseline; if it matches 543.1, the hierarchical grading is not the active ingredient.

Watch

Extended reading notes

Core claim

The central claim is that the ALPHA penalty works as a training objective only when delivered as a dense reinforcement-learning reward, not when it is folded into a supervised loss. The paper proves the point twice: analytically, by showing that the expected-penalty gradient scales with p_theta(k) and therefore 'goes silent' exactly when the model is confidently wrong, whereas the GRPO policy gradient scales inversely with probability and gives large updates to low-probability correct actions; and empirically, by showing that supervised mechanisms degrade under shift (the best supervised score is 2799.2 vs 753.2 zero-shot baseline) while GRPO reaches 543.1 under greedy decoding, a 27.9% redu

Load-bearing premise

The central result depends on the unstated reliability of extracting the predicted CWE from the model's free-form text; if that extraction has non-trivial error, the reward is noisy and the measured gains could be artifacts.

Editorial extensions

If this is right

  • GRPO with a dense hierarchy-aware reward improves CWE-level prediction under distribution shift (27.9% best, 8.3% default) without needing a larger backbone.
  • Both supervised mechanisms (SFT and dual-head expected-penalty loss) regress below the zero-shot 7B baseline on SVEN, so in-distribution gains do not transfer to deployment.
  • The expected-penalty gradient is confirmatory: it can widen a lead but not create one, so any loss of the form sum_v p(v|x) P(v,c) needs a corrective term like cross-entropy to train from a weak start.
  • The improvement is consistent across all six base-backbone GRPO configurations, with sign-test p=0.0156, so the direction of the effect does not depend on hyperparameter selection.
  • The KL anchor to the base reference policy is essential: initialising GRPO from the SFT checkpoint more than doubles the penalty, because the anchor prevents escape from a biased starting distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A binary-reward GRPO baseline would separate the value of the graded hierarchy from RL's free output vocabulary; the paper leaves this open, so part of the 27.9% may not be due to the hierarchy.
  • The unvalidated CWE-extraction step from free-form text is the weakest link: if extraction is noisy, the dense reward is partly rewarding parsing artifacts; a test is to evaluate extraction accuracy and compare with a hidden-state-based reward.
  • The large reference-policy effect suggests an alternative explanation: RL's benefit may come from staying anchored to a balanced pretrained policy (KL) rather than from the penalty; one could test by using the same ALPHA reward with different reference policies.
  • The finding could be generalised to other hierarchical taxonomies and to iterative repair pipelines where the graded penalty's direction-aware asymmetry (over-generalisation penalised more) is presumably useful; deploying the trained policy in a repair loop and measuring remediation is the natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper validates the proposal from the authors' earlier ALPHA work that a taxonomy-aware CWE penalty can be used not only as an evaluation metric but also as a training signal. Three delivery mechanisms are compared: supervised fine-tuning, a dual-head classification loss with an expected-penalty term, and GRPO with a dense reward derived from the normalized penalty. On an in-distribution SecurityEval split, the dual-head loss behaves as predicted by the paper's gradient analysis; under distribution shift to SVEN, both supervised mechanisms regress below a 7B zero-shot baseline, while all base-backbone GRPO configurations improve. The best configuration reduces the SVEN ALPHA penalty from 753.2 to 543.1 (27.9%) under greedy decoding, and the paper concludes that the value of the hierarchical penalty depends more on the directness of delivery than on the penalty itself.

Significance. If the central claim holds, the paper makes a useful empirical contribution: it demonstrates that a hierarchy-aware CWE penalty can serve as a dense RL training signal and that the delivery mechanism is decisive for out-of-distribution robustness. The paper's strengths include a clean and correct gradient derivation (Appendix A), in-distribution results that match the derived predictions, a configuration-robust baseline (690.6, 8.3% improvement) that does not rely on test-set selection, a sign test across six configurations (p=0.0156), transparent ablations of reference policy, data composition, group size and KL coefficient, and an unusually honest threats-to-validity section. The main empirical result, if substantiated, would be of interest to the LLM security community. However, a load-bearing component of the RL pipeline — the extraction of a predicted CWE from free-form text — is never specified or validated, and several secondary claims outrun the evidence. These gaps require substantive revision.

major comments (4)
  1. [§3.3, Eq. (4); §4 and §5] The reward R = 1 - P(ĉ, c_true)/P_max depends entirely on the predicted CWE ĉ 'extracted from the text' of the generated response, but the extraction procedure is never described, validated, or given an error rate. All ALPHA scores in Tables 2 and 3, and therefore the headline 27.9% reduction, depend on this extraction. If the extractor is noisy or biased — e.g., matching the first CWE identifier substring, or defaulting to a frequent ID — the dense reward can reward surface textual patterns rather than genuine CWE classification. Because CWE-89 accounts for 54% of SVEN, a policy could inflate reward by emitting that identifier; the KL anchor constrains divergence from the reference but cannot correct a systematically biased reward. The manuscript must specify the extraction rule, report its accuracy and failure modes, and ideally validate it on a labeled sample or compare against a stru
  2. [§5.2 and §8, Table 3] The headline 543.1 (27.9%) is the best value selected on the SVEN test set across hyperparameters, data recipe, and epoch number, and it comes from a single training run. The paper acknowledges this, and the 690.6 default-configuration result and the sign test are helpful mitigations. However, the six 'independent configurations' are not independent in a statistical sense: they are six hyperparameter combinations sharing the same backbone, data sources, and a single seed per configuration, and continued-epoch rows in Table 3 are repeated evaluations of the same run. The claim that 'GRPO is the only mechanism that demonstrates robustness under shift' would be more defensible if at least the default configuration were repeated with multiple training seeds and the variance reported. Without seed variance, the configuration-robust value 690.6 is also a single draw.
  3. [§3.1, §5.1, and §7] The supervised mechanisms (SFT and dual-head) are evaluated as delivery mechanisms, but their training depends on a teacher-generated explanation corpus whose quality is never checked. The teacher is asked to omit CWE identifiers in the classifier corpus, yet there is no verification that it did so, nor any measure of the mechanistic quality of the explanations. If the teacher corpus is weak or noisy, the supervised failure under shift could be attributed to teacher quality rather than to the delivery mechanism. Because the paper's central contrast is between supervised delivery and direct RL delivery, the teacher corpus must be validated (e.g., human or automatic checks for identifier leakage, explanation correctness, and diversity) or the competing explanation must be explicitly ruled out.
  4. [Abstract, §7, and §9] The conclusion that 'the value of a hierarchical penalty as a training signal depends largely on the directness of its delivery' is broader than the evidence supports. The paper does not run a binary-reward GRPO baseline, so the graded hierarchical structure of the reward is never isolated from the effect of RL with a dense reward and an unrestricted vocabulary. The manuscript acknowledges this in §7 and leaves it as future work, but the abstract's conclusion is stated without this caveat. Either add a binary-reward GRPO condition (or an exact-match CWE reward) to separate the graded-penalty contribution, or soften the abstract and conclusion to say that directly delivered dense reward works, while the marginal value of hierarchy within RL remains open.
minor comments (5)
  1. [§4, Table A1] Typographical errors: 'coefficeint' should be 'coefficient'; in Section 2, 'exemplifly' should be 'exemplify'; in Appendix A, 'orso f tmax' should be 'or softmax'.
  2. [Table 3] The Δ(%) column is confusing: negative values indicate a worse (higher) penalty for SFT rows, while positive values indicate improvement for base-backbone rows. The sign convention should be stated explicitly in the caption.
  3. [§5.2] The phrase 'reaching statistical parity with its 4.5x larger zero-shot teacher' is not backed by a formal parity test. The 543.1 greedy score and the 561.3 sampled mean lie within one reported SD of the 32B model's mean (574.7 ± 42.4), but no test of equivalence or non-inferiority is reported. Consider replacing 'statistical parity' with 'numerical parity' or adding an appropriate test.
  4. [References and related work] Reference [27] (online RL with a vulnerability reward model) is highly related but is not discussed in the body. Since the paper claims no prior work uses a hierarchy-aware CWE penalty as an RL reward, a sentence distinguishing [27]'s reward model from the ALPHA penalty would strengthen the novelty statement.
  5. [§8] The threats section does not mention the unvalidated CWE extraction from Eq. (4). This is a distinct threat from test-set selection and should be acknowledged, preferably with a targeted validation.

Circularity Check

1 steps flagged · score 4.0 of 10

The paper's evaluation and training signal are both the authors' own ALPHA penalty, creating self-reference, but the central delivery-mechanism comparison is not forced by construction.

  1. self citation load bearing [Abstract; §2; §3.2 Eq. (1); §3.3 Eq. (4); Appendix C]
    "The original ALPHA benchmark [1] introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penalty could theoretically also serve as a training signal. This paper provides that validation. [...] The ALPHA penalty graph is taken from [1]."

    The paper's success metric, its supervised L_ALPHA term (Eq. 1), and its GRPO reward (Eq. 4) are all the same normalised ALPHA penalty P(v,c_true)/P_max from the authors' own prior paper [1]. Thus the validation measures [1]'s proposal with [1]'s own quantity: optimising the reward is literally minimising the evaluation metric. This is self-referential rather than by-construction because training is on SecurityEval/combined and evaluation on shifted SVEN, so the policy could have failed; indeed, supervised delivery of the same penalty does fail. However, the load-bearing evaluation criterion is imported from the authors' own, not independently re-validated benchmark, so the central claim rests on a self-citation chain.

full rationale

The GRPO result is not circular by construction: the reward R = 1 - P(c_hat,c_true)/P_max is the same ALPHA penalty used as the test metric, but the policy is trained on SecurityEval/combined and evaluated on the out-of-distribution SVEN set, so improvement is an empirical generalisation result rather than an analytical identity. The supervised mechanisms also use the same penalty (Eq. 1) and fail under shift, which makes the delivery-mechanism comparison informative rather than tautological. The main circularity burden is self-reference: ALPHA [1] is the authors' own prior work, and the paper explicitly frames itself as validating [1]'s proposal using [1]'s own penalty as both loss/reward and evaluation metric. This is load-bearing but not forced. Separate validity threats, which I treat as risks rather than circularity, include the unvalidated CWE-extraction step in the RL reward loop (§3.3), the selection of the best configuration on the SVEN test set (§8, explicitly acknowledged), and the absence of a binary-reward GRPO baseline (§7, explicitly acknowledged).

Assumptions & free parameters 6 free parameters · 8 assumptions · 0 invented entities

The central experiment rests on the ALPHA penalty definition imported from [1], on the reliability of SVEN and SecurityEval labels, on the teacher corpus generated by Qwen2.5-Coder-32B under a label-withholding protocol, and on an unspecified CWE-extraction step for RL rewards. No new physical entities are introduced; the main free choices are the loss weight λ, GRPO hyperparameters (G, β, epochs) that were selected on the test set, and the temperature/pooling choices.

free parameters (6)
  • lambda (dual-head loss weight) = 3.0
    Chosen by hand in §3.2 to balance L_CE ([0,4.23]) and normalized L_ALPHA ([0,1]) on a commensurate scale; the value is not learned or tuned on a validation set.
  • GRPO group size G = 16 (best); 8 explored
    Selected on the SVEN test set (§5.2, §8); the paper acknowledges the best configuration was identified using test results.
  • KL coefficient beta = 0.02 (best); 0.04 explored
    Selected on the SVEN test set; ablation in §6.3 shows it changes the magnitude of improvement.
  • GRPO epochs / early stopping = 3 (best); 2, 4, 6 explored
    The peak epoch was identified by its score on SVEN, which is the test benchmark (§6.3, §8).
  • Sampling temperature = 0.8
    Matches ollama default; used during training rollouts and sampled-decoding evaluation (§4, Table A1).
  • Pooling choice (mean vs last-token) = mean pooling (superior in-distribution)
    Architecture choice; the paper finds pooling is the dominant factor in-distribution and inverts under shift (§5.1).
assumptions (8)
  • domain assumption ALPHA penalty P(v,c_true) and its normalization Pmax from [1] are a valid measure of CWE-prediction utility and are correctly computed.
    The reward (Eq. 4) and the test metric both depend entirely on the ALPHA penalty, which is imported from the authors' own prior paper without definition in this preprint.
  • domain assumption SVEN is a representative out-of-distribution benchmark with reliable labels.
    The central distribution-shift claim is evaluated on SVEN's 342 samples; if its labels or selection are biased, the evaluation is biased. §4 introduces SVEN as the OOD benchmark.
  • domain assumption The teacher (Qwen2.5-Coder-32B) produces faithful mechanistic explanations and complies with the label-withholding constraint.
    The SFT and classifier corpora are generated by this teacher (§3.1); no quality, leakage, or consistency checks are reported.
  • domain assumption A predicted CWE can be reliably extracted from free-form GRPO rollouts.
    The reward R(a,c_true) in Eq. (4) depends on an extracted CWE; the extraction procedure and its error rate are never described (§3.3).
  • domain assumption The published zero-shot baselines (7B: 753.2; 32B: 574.7±42.4) are correct and comparable under matched decoding conditions.
    The statistical claims compare new runs against baseline numbers reported in [1] and [38]; the protocol details are not restated here (§5.2).
  • domain assumption SecurityEval and the filtered CVEfixes subset provide adequate training signal for CWE prediction.
    The 121 SecurityEval samples across 69 CWEs and the filtered CVEfixes subset determine what the model can learn; class sparsity is acknowledged as a binding constraint (§9).
  • standard math Standard softmax calculus: ∂p_k/∂z_j = p_k(δ_kj - p_j), used in Appendix A derivations.
    The gradient analysis of L_CE and L_ALPHA rests on standard differentiation, which is correct as presented.
  • domain assumption GRPO's group-relative advantage baseline and KL anchor behave as standard; skipping groups with reward std < 1e-6 is a valid safeguard.
    The effectiveness of GRPO with the dense penalty reward is assumed; no comparison against a binary-reward baseline is provided to isolate the hierarchy structure (§7, §9).

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python." pith.science (2026). https://pith.science/paper/YVMFQLNW

@misc{pith2026260721069,
  author       = {Pith},
  title        = {Pith review of: From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YVMFQLNW}},
  note         = {Machine review of arXiv:2607.21069}
}
read the original abstract

The original ALPHA benchmark introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penalty could theoretically also serve as a training signal. This paper provides that validation. We compare three delivery mechanisms: supervised fine-tuning, a dual-head classification loss, and reinforcement learning with a dense reward derived from the normalised penalty. We find that supervised approaches consistently regress below the zero-shot baseline under distribution shift, while GRPO succeeds. Our best policy reduces the cumulative ALPHA penalty of Qwen2.5-Coder-7B on Security Hardening and Adversarial Testing (SVEN) dataset by 27.9% under greedy decoding, and by 25.5% under sampled decoding(p = 0.005, Welch's t-test), reaching statistical parity with its 4.5x larger zero-shot teacher. We conclude that the value of a hierarchical penalty as a training signal depends largely on the directness of its delivery.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 1 canonical work pages

  1. [1]

    Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python

    Adnan, M.; Kuhn, C.C.N. Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python. In Proceedings of the 2026 International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML), 2026, pp. 193–200. https://doi.org/10.1 109/AAIML67890.2026.11498216

    arXiv 2026

  2. [3]

    Measuring and mitigating debugging effectiveness decay in code language models.Scientific Reports2025,15, 44120

    Adnan, M.; Kuhn, C.C. Measuring and mitigating debugging effectiveness decay in code language models.Scientific Reports2025,15, 44120

  3. [4]

    Large Language Model Guided Self-Debugging Code Genera- tion, 2025, [arXiv:cs.SE/2502.02928]

    Adnan, M.; Xu, Z.; Kuhn, C.C.N. Large Language Model Guided Self-Debugging Code Genera- tion, 2025, [arXiv:cs.SE/2502.02928]

    arXiv 2025

  4. [5]

    Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions.Commun

    Pearce, H.; Ahmad, B.; Tan, B.; Dolan-Gavitt, B.; Karri, R. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions.Commun. ACM2025,68, 96–105. https: //doi.org/10.1145/3610721

    doi:10.1145/3610721

  5. [6]

    How secure is AI-generated code: A large-scale comparison of large language models.Empirical Software Engineering2025,30, 47

    Tihanyi, N.; Bisztray, T.; Ferrag, M.A.; Jain, R.; Cordeiro, L.C. How secure is AI-generated code: A large-scale comparison of large language models.Empirical Software Engineering2025,30, 47

  6. [7]

    When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs.Transactions of the Association for Computational Linguistics2024,12, 1417–1440

    Kamoi, R.; Zhang, Y.; Zhang, N.; Han, J.; Zhang, R. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs.Transactions of the Association for Computational Linguistics2024,12, 1417–1440. https://doi.org/10.1162/tacl_a_00713

    doi:10.1162/tacl_a_00713

  7. [8]

    SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques

    Siddiq, M.L.; Santos, J.C.S. SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques. In Proceedings of the Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security, 2022. https://doi.org/10.1145/3549035.3561184

    arXiv 2022

  8. [9]

    CVEfixes: automated collection of vulnerabilities and their fixes from open-source software

    Bhandari, G.; Naseer, A.; Moonen, L. CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, 2021, pp. 30–39

    2021

Show all 39 references
  1. [10]

    Large language models for code: Security hardening and adversarial testing

    He, J.; Vechev, M. Large language models for code: Security hardening and adversarial testing. In Proceedings of the Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, 2023, pp. 1865–1879

    2023

  2. [11]

    Instruction tuning for secure code generation

    He, J.; Vero, M.; Krasnopolska, G.; Vechev, M. Instruction tuning for secure code generation. In Proceedings of the Proceedings of the 41st International Conference on Machine Learning. JMLR.org, 2024, ICML’24

    2024

  3. [12]

    Comparison of static application security testing tools and large language models for repo-level vulnerability detection.arXiv preprint arXiv:2407.162352024

    Zhou, X.; Tran, D.M.; Le-Cong, T.; Zhang, T.; Irsan, I.C.; Sumarlin, J.; Le, B.; Lo, D. Comparison of static application security testing tools and large language models for repo-level vulnerability detection.arXiv preprint arXiv:2407.162352024. https://doi.org/10.3390/1010000...

    arXiv

  4. [13]

    DLAP: A Deep Learning Augmented Large Language Model Prompting framework for software vulnerability detection.J

    Yang, Y.; Zhou, X.; Mao, R.; Xu, J.; Yang, L.; Zhang, Y.; Shen, H.; Zhang, H. DLAP: A Deep Learning Augmented Large Language Model Prompting framework for software vulnerability detection.J. Syst. Softw.2025,219. https://doi.org/10.1016/j.jss.2024.112234

    2025

  5. [14]

    An Empirical Study of Vulnerabilities in Python Packages and Their Detection.arXiv preprint arXiv:2509.042602025

    Quan, H.; Wang, J.; Li, X.; Zhuo, T.Y.; Chen, X.; Du, X. An Empirical Study of Vulnerabilities in Python Packages and Their Detection.arXiv preprint arXiv:2509.042602025

    arXiv

  6. [15]

    Common weakness enumeration (CWE) status update.Ada Lett.2008, XXVIII, 88–91

    Martin, R.A.; Barnum, S. Common weakness enumeration (CWE) status update.Ada Lett.2008, XXVIII, 88–91. https://doi.org/10.1145/1387830.1387835

    2008

  7. [16]

    Making Better Mistakes: Leveraging Class Hierarchies With Deep Networks

    Bertinetto, L.; Mueller, R.; Tertikas, K.; Samangooei, S.; Lord, N.A. Making Better Mistakes: Leveraging Class Hierarchies With Deep Networks. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 12503–12512. https://doi.o...

    2020

  8. [17]

    Deep reinforcement learning from human preferences

    Christiano, P .F.; Leike, J.; Brown, T.B.; Martic, M.; Legg, S.; Amodei, D. Deep reinforcement learning from human preferences. In Proceedings of the Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2017; NIPS’17, p....

    2017

  9. [18]

    Training language models to follow instructions with human feedback

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.L.; Mishkin, P .; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. Training language models to follow instructions with human feedback. In Proceedings of the Proceedings of the 36th International Conference on Neural ...

    2022

  10. [19]

    Proximal Policy Optimization Algorithms, 2017, [arXiv:cs.LG/1707.06347]

    Schulman, J.; Wolski, F.; Dhariwal, P .; Radford, A.; Klimov, O. Proximal Policy Optimization Algorithms, 2017, [arXiv:cs.LG/1707.06347]

    2017 arXiv

  11. [20]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.033002024

    Shao, Z.; Wang, P .; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.033002024

    arXiv

  12. [21]

    CodeRL: mastering code generation through pretrained models and deep reinforcement learning

    Le, H.; Wang, Y.; Gotmare, A.D.; Savarese, S.; Hoi, S.C. CodeRL: mastering code generation through pretrained models and deep reinforcement learning. In Proceedings of the Proceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY, ...

    2022

  13. [22]

    Execution-based code generation using deep reinforcement learning.arXiv preprint arXiv:2301.138162023

    Shojaee, P .; Jain, A.; Tipirneni, S.; Reddy, C.K. Execution-based code generation using deep reinforcement learning.arXiv preprint arXiv:2301.138162023

    arXiv

  14. [23]

    RLTF: Reinforcement Learning from Unit Test Feedback.Transactions on Machine Learning Research2023

    Liu, J.; Zhu, Y.; Xiao, K.; FU, Q.; Han, X.; Wei, Y.; Ye, D. RLTF: Reinforcement Learning from Unit Test Feedback.Transactions on Machine Learning Research2023

  15. [24]

    Stepcoder: improving code generation with reinforcement learning from compiler feedback

    Dou, S.; Liu, Y.; Jia, H.; Zhou, E.; Xiong, L.; Shan, J.; Huang, C.; Wang, X.; Fan, X.; Xi, Z.; et al. Stepcoder: improving code generation with reinforcement learning from compiler feedback. In Proceedings of the Proceedings of the 62nd Annual Meeting of the Association for C...

    2024

  16. [25]

    Llm-powered code vulnerability repair with reinforcement learning and semantic reward

    Islam, N.T.; Khoury, J.; Seong, A.; Karkevandi, M.B.; Parra, G.D.L.T.; Bou-Harb, E.; Najafirad, P . Llm-powered code vulnerability repair with reinforcement learning and semantic reward. arXiv preprint arXiv:2401.033742024

    arXiv

  17. [26]

    Code Security Vulnerability Repair Using Reinforce- ment Learning with Large Language Models, 2024, [arXiv:cs.CR/2401.07031]

    Islam, N.T.; Karkevandi, M.B.; Najafirad, P . Code Security Vulnerability Repair Using Reinforce- ment Learning with Large Language Models, 2024, [arXiv:cs.CR/2401.07031]

    2024 arXiv

  18. [27]

    Secure Code Generation via On- line Reinforcement Learning with Vulnerability Reward Model, 2026, [arXiv:cs.CR/2602.07422]

    Wu, T.; Du, M.; Liu, Y.; Yang, C.; Zhuo, T.Y.; Zhang, J.; Ng, S.K. Secure Code Generation via On- line Reinforcement Learning with Vulnerability Reward Model, 2026, [arXiv:cs.CR/2602.07422]

    2026

  19. [28]

    R+ r: Security vulnerability dataset quality is critical

    Yadav, A.S.; Wilson, J.N. R+ r: Security vulnerability dataset quality is critical. In Proceedings of the 2024 Annual Computer Security Applications Conference (ACSAC). IEEE, 2024, pp. 1047–1061

    2024

  20. [29]

    Analyzing source code vulnerabilities in the D2A dataset with ML ensembles and C-BERT.Empirical Softw

    Pujar, S.; Zheng, Y.; Buratti, L.; Lewis, B.; Chen, Y.; Laredo, J.; Morari, A.; Epstein, E.; Lin, T.; Yang, B.; et al. Analyzing source code vulnerabilities in the D2A dataset with ML ensembles and C-BERT.Empirical Softw. Engg.2024,29. https://doi.org/10.1007/s10664-023-10405-9

    2024 doi

  21. [30]

    Lora: Low-rank adaptation of large language models.Iclr2022,1, 3

    Hu, E.J.; Shen, Y.; Wallis, P .; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. Lora: Low-rank adaptation of large language models.Iclr2022,1, 3

  22. [31]

    Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models

    ENNADIR, S.; Zólyomi, L.; Smirnov, O.; Wang, T.; Pertoft, J.; Cornell, F.; Cao, L. Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models. In Proceedings of the The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. https://doi.org/10....

    2026 doi

  23. [32]

    Unleashing Artificial Cognition: Integrating Multiple AI Systems.Australasian Conference on Information Systems2024

    Adnan, M.; Gamage, B.; Xu, Z.; Herath, D.; Kuhn, C.C. Unleashing Artificial Cognition: Integrating Multiple AI Systems.Australasian Conference on Information Systems2024

  24. [33]

    Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation.arXiv preprint arXiv:2506.230342025

    Yan, H.; Vaidya, S.S.; Zhang, X.; Yao, Z. Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation.arXiv preprint arXiv:2506.230342025

    arXiv

  25. [34]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences2017,114, 3521–3526

    Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A.A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences2017,114, 3521–3526

  26. [35]

    Sutton, R.S.; Barto, A.G.Reinforcement Learning: An Introduction, second ed.; MIT Press: Cam- bridge, MA, 2018

    2018

  27. [36]

    On Information and Sufficiency.Annals of Mathematical Statistics1951, 22, 79–86

    Kullback, S.; Leibler, R.A. On Information and Sufficiency.Annals of Mathematical Statistics1951, 22, 79–86

  28. [37]

    Adam: A method for stochastic optimization

    Kingma, D.P .; Ba, J. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015

    2015

  29. [38]

    Hui, B.; Yang, J.; Cui, Z.; Yang, J.; Liu, D.; Zhang, L.; Liu, T.; Zhang, J.; Yu, B.; Lu, K.; et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.121862024

    arXiv

  30. [39]

    Parameter-efficient fine-tuning of large-scale pre-trained language models.Nat Mach Intell 5 2023, pp

    Ding, N.; Qin, Y.; Yang, G.; Wei, F.; Yang, Z.; Su, Y.; Hu, S.; Chen, Y.; Chan, C.M.; Chen, W.; et al. Parameter-efficient fine-tuning of large-scale pre-trained language models.Nat Mach Intell 5 2023, pp. 220––235. https://doi.org/https://doi.org/10.1038/s42256-023-00626-4

    2023 doi

  31. [40]

    A study of BFLOAT16 for deep learning training

    Kalamkar, D.; Mudigere, D.; Mellempudi, N.; Das, D.; Banerjee, K.; Avancha, S.; Vooturi, D.T.; Jammalamadaka, N.; Huang, J.; Yuen, H.; et al. A study of BFLOAT16 for deep learning training. arXiv preprint arXiv:1905.123222019. https://doi.org/10.3390/1010000

    1905 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.