Pith.

Software Engineering · arXiv:2609.05879 v1 · submitted 5 Sep 2026 · 25 references

Correct Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test Generation

Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, Shiwen Ni

The paper says

“Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output.”

more of the abstract

These requirements need not improve together: a model can increase test correctness by choosing easier inputs, or discover useful inputs whose expected outputs it cannot predict. We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement. Our generator jointly emits five input–output tests in one response. During training, audited reference programs provide correctness feedback, while a fixed bank of faulty programs provides two utility signals: potential input kill and effective kill after checking the generated output. An additive GRPO objective preserves both signals without requiring execution at inference time. On an audited TC-Bench split with 506 training and 142 evaluation tasks, three independently trained Qwen3.5-9B runs at step 75 increase full-test correctness from 28.59% to 42.54%, input kill from 24.06% to 25.27%, and effective full kill from 12.23% to 14.15%. Matched 50-step ablations reveal a trade-off: removing kill rewards yields higher correctness and slightly higher full kill, but lowers input kill to 21.60%. A fixed-input source–oracle crossover on 64 training-pool tasks attributes the principal NoKill-to-FullKill difference to harder input selection rather than worse output prediction on identical inputs. These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 9 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: minor revision.

A useful decomposition of oracle conversion in test generation, but the core metrics depend on an unvalidated reference program bank. This line is wrong

2 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 25 references say.

The paper says, at Section 8 (Threats) and Eqs. (2)–(5), (9)

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The central quantitative claims (e.g., Table 3: 28.59%→42.54% correctness, 24.06%→25.27% input kill, 12.23%→14.15% effective kill) are all computed against the same audited consensus oracle from the reference program bank. The paper acknowledges (Section 8) that “multiple references can share a defect or process an out-of-domain input identically.”

more of objection 1

Because training reward (Eq. 9) and every evaluation metric (Eqs. 2–5) use this same oracle, a systematic judge error would not average out across seeds; it would coherently bias all reported scores. The authors should either provide manual validation evidence for a representative sample of generated inputs and outputs, or perform a sensitivity analysis (e.g., randomly corrupting a fraction of reference programs) to bound the potential impact on the main comparisons. Without such evidence, the reader cannot assess how much of the reported improvement is genuine spec-level correctness versus agreement with a possibly flawed consensus.

The paper says, at Section 5.1 and Section 8

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The 142-task evaluation set was inspected during prior experiments and used to select the step-75 training length (seed 103). The paper transparently discloses this, but it means the absolute scores are not unbiased estimates on a newly untouched test set.

more of objection 2

The methods paper presents the step-75 numbers as the main result, yet the set is effectively a project-validation set rather than a held-out test set. The authors should clarify exactly how many prior inspections occurred and, if possible, perform a final evaluation on a genuinely held-out set (e.g., from a different source or a later-data release) to confirm generalization.

5 minor objections

The paper says, at Table 1

Pith says: minor objection 1 of 5

The first row of the table renders as “NoKill0.05𝐹+0.05𝑄+0.15𝑉 𝑞+0.25𝐶𝑞−0.10𝐿”. Insert a space or colon after “NoKill” for readability, and use consistent notation for subscripts (e.g., V_q, C_q) as in the text.

The paper says, at Table 7

Pith says: minor objection 2 of 5

The proxy endpoint is labeled “GPT-5.6-sol”. GPT-5.6 is not a standard model identifier and may confuse readers; consider either redacting the specific version or clarifying that this is a proxy name for an unverified backend.

show 3 more

The paper says, at Section 6.4

Pith says: minor objection 3 of 5

Seed 101 at step 75 has 9 failed tasks due to unterminated JSON. It would be helpful to report what fraction of those failures correspond to test-time length limits (the 4096-token response limit) versus other causes, since that directly affects the practical deployability of the method.

The paper says, at Figure 1 caption

Pith says: minor objection 4 of 5

The caption says “Effective kill per test (%)” but the y-axis label in panel (b) reads “Effective kill per test (%)”. It is unclear whether this uses the suite-union denominator or the per-test rate; the text clarifies it is the latter, but the caption could state this explicitly: “per-test effective kill, not suite-union.”

The paper says, at Section 3.4

Pith says: minor objection 5 of 5

The bin notation uses parentheses inconsistently: (0,0.10] is fine, but later the text writes “(0, 0.10]” with spaces. Use consistent spacing and include a legend for the bin intervals in the figure.

What would settle it

Independently re-derive consensus outputs for a stratified sample of evaluation slots. Draw roughly 60 tasks from the 142-task evaluation set spanning all five input-kill bins; for each sampled reference-valid input, have two independent oracle implementers (or human annotators) write solutions from the natural-language spec without seeing the TC-Bench reference programs, and compare their outputs with y*_q(x). Also inspect a sample of inputs the references labeled invalid to check whether rejection is due to a real constraint violation or a reference bug. Recompute C, O, and K on the sample using the corrected oracle. If disagreement exceeds roughly 2% of sampled slots, or if the Additive-minus-Base deltas change sign on the sample, the Table 3 claims are not established as spec-correct; if agreement is high, the consensus assumption is supported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 9 Sep 2026

  1. Reader holds up confidence high
  2. Skeptic agrees: says it holds up only with conditions 1 min 11 s · 18,327 tokens read, 6,066 written All three headline metrics (C, O, K) are computed against the same audited consensus oracle; if the 5–8 reference solutions share a systematic defect, the training reward and every reported number in Table 3 shift coherently with that error.
  3. Editor wrote the verdict above 21 s · 19,086 tokens read, 2,057 written
  4. Referee minor revision: 2 major objections, 5 minor 32 s · 18,561 tokens read, 3,168 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper makes a clean contribution by separating potential and effective kill in specification-based test generation, and by using a fixed-input crossover to tease apart input selection from oracle difficulty. The experimental design is careful: multiple seeds, exact continuations, transparent per-seed reporting, and honest discussion of limitations. The additive GRPO objective is a plausible way to preserve input utility during training, and the measurement protocol (oracle conversion efficiency, suite-union kill) is reusable.

The soft spots are real. First, the absolute improvements are modest—effective kill goes from 12.23% to 14.15%. Second, and more important, every headline number depends on the correctness of the reference program bank. The authors acknowledge this in Section 8, but they do not test how sensitive the results are to reference set composition or to systematic defects shared across references. If the references all mishandle a certain input type, the consensus oracle is wrong, and the reported correctness and kill rates shift coherently. Three-seed replication won't catch that. The stress-test note is accurate here.

Other concerns: the study uses one model family (Qwen3.5-9B), one benchmark subset, and modest training budgets. The Additive method doesn't dominate NoKill on effective kill; the trade-off is real and well-documented. Generalizability is uncertain.

Who this is for: researchers working on RL for test generation or reward design for code generation. The fixed-input crossover and the oracle conversion measurement are worth adopting.

I would send this to peer review. The core idea is sound, the experiments are transparent, and the limitations are acknowledged. But I would ask the authors to add a robustness analysis—for example, jackknifing the reference programs or using an independent oracle for a subset of tasks. Without that, the quantitative claims rest on a threat that the paper flags but does not address.

The strongest objection

Every headline number depends on the correctness and completeness of the reference bank. Equation (2) defines full correctness as agreement with the consensus output y*_q(x); Equations (3)–(5) build potential and effective kill from the same consensus; Equation (9) feeds that consensus into the training reward. If the retained references share a defect, or all process some out-of-domain input identically, then an incorrect generated output is labeled fully correct, faulty programs whose outputs agree with the true spec are counted as killed, and inputs that are valid for the spec but crash the references are discarded as reference-invalid. Section 8 states exactly this risk: 'multiple references can share a defect or process an out-of-domain input identically.' Because the same consensus drives both training and evaluation, a systematic judge error would not average out across the three seeds; it would coherently shift correctness and both kill rates in the same direction. The three-seed replication therefore does not protect the central quantitative claims (28.59→42.54 C, 24.06→25.27 O, 12.23→14.15 K). The paper scopes its correctness claims to the audited protocol, but the abstract and title present these as 'correct tests' and 'oracle conversion,' which requires spec-level correctness, not merely self-consistency of the reference programs.

The paper is internally careful: the metrics in Equations (3)–(8) are well defined, the reward decomposition in Equation (9) is transparent, the nested ablations are matched, per-seed numbers are shown, failure cases remain in the denominator, and the fixed-input crossover is a genuinely useful diagnostic. The reader's weakest assumption correctly identifies the reference-program consensus as the foundation. I agree that this is the single most load-bearing assumption: a systematic defect in the reference bank would contaminate the training signal and every evaluation column in the same direction, so replication across seeds and internal consistency cannot rescue the claim. The paper's own Section 8 admission makes this more than a hypothetical. That said, the concern is testable: a modest independent audit of the consensus oracle would settle whether the reported correctness and kill numbers reflect spec-level test quality or merely agreement with a possibly flawed reference set. Because the manuscript's strongest claims are framed as improvements in correct tests and oracle conversion, and because the current evidence does not independently validate the oracle, I would accept only conditionally on that audit rather than as a fully established empirical result.

The referee's report

If the results hold, this work makes a valuable conceptual contribution by formalizing oracle conversion as a measurable bottleneck in specification-based test generation, and by providing a principled training objective (Additive GRPO) that balances correctness with input utility without requiring execution at inference time. The three-seed replication, the careful ablations (NoKill, FullKill-only), and the crossover diagnostic are methodologically strong and set a good standard for future work in this area. The paper also releases its evaluation protocol and artifact details, supporting reproducibility. The findings are directly relevant to the growing body of work on LLM-based test generation and could inform the design of reward functions for other program synthesis tasks that require joint reasoning about inputs and outputs.

Pith's reading of the paper (a machine's summary, not the authors')

This paper shows that when a language model generates tests from a natural-language specification, improving test correctness and improving fault detection can work against each other: a model can become more accurate by choosing easier inputs, thereby missing harder-to-detect bugs. The authors introduce oracle conversion as a suite-level measure of how many potentially detectable faults are actually caught by the model's own correct outputs. They then propose Additive GRPO, a reinforcement learning objective that rewards both full correctness and potential fault detection (input kill), even when the generated output is wrong. On an audited subset of the TC-Bench benchmark, Additive GRPO improves both test correctness and effective fault detection relative to the base model, while preserving input utility that correctness-only training sacrifices. The work identifies oracle conversion as a concrete bottleneck and provides a measurement protocol for future improvements.

What carries the argument. Two central mechanisms carry the argument: (1) a suite-level oracle-conversion measurement protocol that reports validity, correctness, potential input kill (union of faulty programs distinguished by reference-valid inputs), effective full kill (union distinguished by fully correct tests), and the conversion efficiency ratio; (2) an Additive GRPO objective function that decomposes the suite-level reward into format, count, validity, correctness, effective-kill, and potential-kill components, with the potential-kill term preserving utility even on inputs whose outputs are wrong. The group-relative advantage normalization of GRPO ensures that differences in potential kill among sampled suites drive policy updates.

Core claim. The paper establishes that the bottleneck in specification-based test generation is not only discovering revealing inputs but converting those inputs into complete correct tests. It defines potential input kill (the set of faulty programs an input could distinguish if its output were correct) and effective full kill (the set actually distinguished by fully correct tests), and shows that the gap between them—oracle conversion—can be measured and partially addressed through reward design. The key empirical finding is that Additive GRPO, which rewards both effective kill and potential input kill, preserves input utility (potential kill) while improving full correctness and effective kill, unlike correctness-only training which sacrifices input utility. However, the absolute effective gains are modest (from 12.23% to 14.15% full kill), and the main improvement comes from selecting more difficult inputs rather than improving output prediction on unchanged inputs.

The premise it rests on. The reference programs used to compute the consensus oracle are correct and complete for each task; if they share a defect or process an out-of-domain input identically, the correctness signal and kill counts become unreliable, compromising both training and evaluation.

If the paper is right (Pith's inferences, not the authors')

Test generation evaluations should report potential kill alongside effective kill to separate input-discovery ability from oracle accuracy, avoiding the misleading conclusion that a model that selects easier inputs is a better test generator.

Reinforcement learning objectives that include potential-input-kill reward can prevent the collapse of input diversity under correctness pressure, making the trained policy a stronger candidate for test-driven development where both correct tests and challenging inputs are desired.

The oracle-conversion protocol provides a diagnostic tool for identifying whether a generator's bottleneck is input selection or output prediction, enabling targeted interventions (e.g., harder-input training vs. output-reasoning training).

The modest absolute gains (≈2 percentage points on effective kill) suggest that additive reward is insufficient alone; future work must combine it with stronger oracle reasoning, perhaps through multi-turn refinement or dedicated oracle models.

The oracle-conversion gap could serve as a universal metric for any task where a system generates both a probe and its expected answer (e.g., question answering, program synthesis), not just software testing.

The source–oracle crossover method (freezing inputs and swapping output policies) is a reusable ablation that should become standard in decomposed generation evaluations, as it separates input-selection effects from output-prediction effects.

If training were extended to reward per-input conversion directly (e.g., penalizing incorrect outputs on high-kill inputs), the additive reward might achieve larger effective gains than the reported 14.15% full kill.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output. These requirements need not improve together: a model can increase test correctness by choosing easier inputs, or discover useful inputs whose expected outputs it cannot predict. We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement. Our generator jointly emits five input–output tests in one response. During training, audited reference programs provide correctness feedback, while a fixed bank of faulty programs provides two utility signals: potential input kill and effective kill after checking the generated output. An additive GRPO objective preserves both signals without requiring execution at inference time. On an audited TC-Bench split with 506 training and 142 evaluation tasks, three independently trained Qwen3.5-9B runs at step 75 increase full-test correctness from 28.59% to 42.54%, input kill from 24.06% to 25.27%, and effective full kill from 12.23% to 14.15%. Matched 50-step ablations reveal a trade-off: removing kill rewards yields higher correctness and slightly higher full kill, but lowers input kill to 21.60%. A fixed-input source–oracle crossover on 64 training-pool tasks attributes the principal NoKill-to-FullKill difference to harder input selection rather than worse output prediction on identical inputs. These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.85 · human participants 0.04 · controlled comparison 0.88 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.82 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output. background 1.00 0.07 0.03 asserted 1.00
These requirements need not improve together: a model can increase test correctness by choosing easier inputs, or discover useful inputs whose expected outputs it cannot predict. background 0.71 0.12 0.06 hedged possibility 0.59
We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement. contribution 0.55 0.93 0.05 asserted 0.97
Our generator jointly emits five input--output tests in one response. method 0.98 0.94 0.03 asserted 1.00
During training, audited reference programs provide correctness feedback, while a fixed bank of faulty programs provides two utility signals: potential input kill and effective kill after checking the generated output. method 1.00 0.92 0.03 asserted 0.99
An additive GRPO objective preserves both signals without requiring execution at inference time. method 0.99 0.83 0.05 asserted 1.00
On an audited TC-Bench split with 506 training and 142 evaluation tasks, three independently trained Qwen3.5-9B runs at step 75 increase full-test correctness from 28.59\% to 42.54\%, input kill from 24.06\% to 25.27\%, and effective full kill from 12.23\% to 14.15\%. result 1.00 0.97 0.12 asserted 1.00
Matched 50-step ablations reveal a trade-off: removing kill rewards yields higher correctness and slightly higher full kill, but lowers input kill to 21.60\%. result 0.98 0.96 0.09 asserted 1.00
A fixed-input source--oracle crossover on 64 training-pool tasks attributes the principal NoKill-to-FullKill difference to harder input selection rather than worse output prediction on identical inputs. interpretation 0.51 0.94 0.11 asserted 0.97
These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation. interpretation 1.00 0.94 0.87 asserted 1.00

What kind of paper the classifier reports this is: method or tool 0.57 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,689 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (2)
Figure 1 of the paper
Figure 1. Oracle conversion across generated-input kill bins on the 142-task evaluation set. Trained-policy slots
Figure 2 of the paper
Figure 2. Cross-model capability references. Curves are conditional on each model’s generated reference-valid

Figures are the authors', reproduced from the paper.

References (25)
  1. [1] Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated Unit Test Improvement using Large Language Models at Meta. arXiv:2402.09171 [cs.SE] https://arxiv.org/abs/2402.09171 on PithPith's reviewarXiv
  2. [2] Juan Altmayer Pizzorno and Emery D. Berger. 2025. CoverUp: Effective High Coverage Test Generation for Python. Proceedings of the ACM on Software Engineering2, FSE (June 2025), 2897–2919. doi:10.1145/3729398 on Pithdoi:10.1145/3729398
  3. [3] Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey.IEEE Transactions on Software Engineering41, 5 (May 2015), 507–525. doi:10.1109/tse.2014.2372785 on PitharXiv
  4. [4] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv:2304.05128 [cs.CL] https://arxiv.org/abs/2304.05128 on PithPith's reviewarXiv
  5. [5] Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: a neural method for test oracle generation. InProceedings of the 44th International Conference on Software Engineering (ICSE ’22). ACM, 2130–2141. doi:10.1145/3510003.3510141 on PitharXiv
  6. [6] Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering (ESEC/FSE’11). ACM, 416–419. doi:10.1145/20251... on PitharXiv
  7. [7] Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li. 2025. HardTests: Synthesizing High-Quality Test Cases for LLM Coding. arXiv:2505.24098 [cs.CL] https: //arxiv.org/abs/2505.24098 on PithPith's reviewarXiv
  8. [8] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. arXiv:2105.09938 [cs.SE] https://arxiv.org/abs/2105.09938 on PithPith's reviewarXiv
  9. [9] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL] https://arxiv.org/abs/2106.09685 on PithPith's reviewarXiv
  10. [10] Dong Huang, Mark Harman, Jie M. Zhang, Zhijiang Guo, Mingzhe Du, and See Kiong Ng. 2026. Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning. arXiv:2608.24473 [cs.SE] https://arxiv.org/abs/2608.24473 on PithPith's reviewarXiv
  11. [11] Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing.IEEE Transactions on Software Engineering37, 5 (Sept. 2011), 649–678. doi:10.1109/tse.2010.62 on Pithdoi:10.1109/tse.2010.62
  12. [12] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180 [cs.LG] https://arxiv.org/abs/2309.06180 on PithPith's reviewarXiv
  13. [13] Dongjun Lee, Changho Hwang, and Kimin Lee. 2026. Learning to Generate Unit Test via Adversarial Reinforcement Learning. InInternational Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026. 148091–14811... on Pith
  14. [14] Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. TACO: Topics in Algorithmic COde generation dataset. arXiv:2312.14852 [cs.AI] https://arxiv.org/abs/2312.14852 on PithPith's reviewarXiv
  15. [15] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale... on PithPith's reviewarXiv
  16. [16] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv.org/abs/2305.01210 on PithPith's reviewarXiv
  17. [17] Xianzhen Luo, JinYang Huang, Wenzhen Zheng, Qingfu Zhu, Mingzheng Xu, Yiheng Xu, YuanTao Fan, and Wanx- iang Che. 2026. How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary- Matrix Perspective. InInternational Conference on Learning Represent... on Pith
  18. [18] Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In29th International Conference on Software Engineering (ICSE’07). IEEE, 75–84. doi:10.1109/icse.2007.37 on Pithdoi:10.1109/icse.2007.37
  19. [19] Archiki Prasad, Elias Stengel-Eskin, Justin Chih-Yao Chen, Zaid Khan, and Mohit Bansal. 2025. Learning to Generate Unit Tests for Automated Debugging. InConference on Language Modeling. arXiv:2502.01619 [cs.SE] https://openreview. net/forum?id=yeVBHPLXxi on PithPith's reviewarXiv
  20. [20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https://arxiv.org/abs/1707.06347 on PithPith's reviewarXiv
show 5 more
  1. [21] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300 on PithPith's reviewarXiv
  2. [22] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. InProceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25). ACM, 1279–1297. d... on PitharXiv
  3. [23] Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. 2025. Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation. arXiv:2310.02368 [cs.SE] https://arxiv.org/abs/2310.02368 on PithPith's reviewarXiv
  4. [24] Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. 2025. Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning. InAdvances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. ... on Pithdoi:10.52202/085713-4809
  5. [25] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https://arxiv.org/abs/2201.11903 A Prompt and Output Protocol T... on PithPith's reviewarXiv
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment