Pith. sign in

REVIEW 4 major objections 5 minor 47 references

SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAGE claims that a fine-tuned open-source LLM with verifiable-reward reinforcement learning can extract Context-Free Grammars with Counters (CCFGs) from natural-language specifications, reaching 96.66% set-based validity and 95.92%…

desk verdict Useful empirical system for grammar induction with real gains over baselines, but the headline generality number hinges on a k=5 reward and evaluation protocol that needs an independent check. read the letter →

arxiv 2506.11081 v1 pith:RLMRB4DF submitted 2025-06-04 cs.CL

Aditi , Hyunwoo Park , Sicheol Sung , Yo-Sub Han , Sang-Ki Ko This is my paper
classification cs.CL
keywords testcasegenerationcontext-freegrammarswithcountersreinforcementlearninglargelanguagemodelscompetitiveprogramminggrammarinductionverifiablerewardsspecification-to-grammartranslation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SAGE is a pipeline that gets an open-source language model to translate natural-language problem specifications into Context-Free Grammars with Counters (CCFGs), the formal objects that describe a competitive-programming input format. The paper claims that supervised fine-tuning on fewer than a thousand specification-grammar pairs, followed by reinforcement learning that rewards grammars whose sampled test cases are valid and general, is enough to reach 96.66% set-based validity, 95.92% set-based generality, and 80.67% set-based effectiveness on 240 held-out problems. If true, this closes the gap between open and closed models for grammar-based test generation, because the winning system is a 14-billion-parameter open model that outperforms 17 baselines including commercial ones. The central mechanism is a verifiable reward: the model's grammar is rewarded by parsing its generated test cases with a reference grammar and vice versa, so no human preference labels are needed.

What carries the argument

The load-bearing object is the Context-Free Grammar with Counters (CCFG), which extends an ordinary CFG by letting production rules read counter variables set earlier in the derivation, so that, for example, the number of array elements derived matches the value n sampled for the array size. The argument is carried by a two-stage training loop: supervised fine-tuning aligns the model to the spec-to-grammar mapping, and GRPO then optimizes the reward R = RV · RG, where RV is the fraction of k sampled test cases from the generated grammar that the ground-truth grammar parses, and RG is the fraction of k sampled ground-truth test cases the generated grammar parses. Test cases are drawn from a CCFG by a Las Vegas sampling algorithm that tracks feasible value intervals for counters and resamples until constraints are satisfied.

What would settle it

Re-annotate a random subset of the 240 evaluation problems with independent experts, or run the same generated grammars through a different CCFG parser, and check whether the 96.66% validity and 95.92% generality figures hold; alternatively, generate 100 test cases per grammar instead of 10 and see whether set-based effectiveness moves toward 100% rather than staying near the ground-truth ceiling.

Watch

Extended reading notes

Core claim

The paper's central claim is that an open-source LLM, after supervised fine-tuning and GRPO reinforcement learning with grammar-based verifiable rewards, induces CCFGs from specifications that are nearly as good as human-annotated ground-truth grammars. On the 240-problem evaluation split, SAGE achieves 97.28% element-based validity, 96.66% set-based validity, 96.75% element-based generality, 95.92% set-based generality, and 80.67% set-based effectiveness, exceeding the prior CCFGT5 system by 15.92 percentage points in set-based validity and 12.34 points in set-based effectiveness. Ablations show supervised fine-tuning is the foundation, GRPO adds a consistent further gain, and iterative feedback, while helpful for closed-source models, adds little after fine-tuning because ill-formed grammars become rare.

Load-bearing premise

The human-annotated CCFGs from [8] used both as the reward oracle and as the evaluation reference are correct, complete, and unbiased, so errors in them would be inherited by the training signal and the reported metrics.

Editorial extensions

If this is right

  • A small open model (14 billion parameters) can outperform much larger and closed models on grammar-based test generation when given a verifiable reward signal.
  • Test suites generated from SAGE grammars distinguish incorrect solutions at 80.67% set-based effectiveness, close to the 83.70% ceiling of human-written grammars sampled to 10 test cases.
  • The reward design means the approach needs no human preference annotations beyond the reference grammars used for parsing.
  • Iterative feedback is largely redundant for a fine-tuned and RL-trained model, which produces well-formed grammars in almost all cases.
  • The authors state that the recipe is intended to extend to more expressive grammar representations and real-world tasks like program synthesis and automated grading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reward is computed from only k=5 samples per grammar, the RL signal is noisy; a more careful sample-size study could reveal whether the reported gains are stable or partly an artifact of small-sample variance.
  • The evaluation inherits any bias in the human-annotated ground-truth grammars; a cross-check with independently written grammars or a different CCFG parser would be the direct test of the validity and generality numbers.
  • Set-based effectiveness is computed with only 10 generated test cases per problem, which caps the score below the grammar's true ceiling; generating more test cases would likely push SAGE's effectiveness closer to, or past, the ground-truth 83.70%.
  • The method suggests a general template: if a target output domain has any executable validator, SFT plus reward-guided RL can teach a small model to produce structured artifacts in that domain, without needing large human-curated preference datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SAGE, a pipeline in which an open-source LLM (DeepSeek-R1-Distill-Qwen-14B) is first supervised fine-tuned on specification-to-CCFG pairs and then trained with GRPO using verifiable rewards computed from a CCFG parser. The learned grammar is used to generate test cases, and the authors evaluate validity, generality, and effectiveness on 240 CodeContests problems with human-annotated CCFGs from prior work. They compare against 17 open- and closed-source LLMs, mutation-based fuzzing, direct test-case generation, and the CCFGT5 baseline, and they report set-based validity of 96.66%, set-based generality of 95.92%, and set-based effectiveness of 80.67%, which they describe as state-of-the-art. The paper also includes ablations for SFT, GRPO, prompt components, and iterative feedback.

Significance. The combination of grammar-based test-case generation with reinforcement learning from verifiable rewards is timely, and the use of parser-based rewards is a sensible way to reduce dependence on human feedback. Notable strengths are the detailed hyperparameter disclosure in Table I, the component ablation in Table V, the prompt ablation in Table VI, and the anonymous release of code and data. If the reported metrics were robust, the improvements over few-shot LLM baselines and over the prior CCFGT5 system would be practically valuable. However, the reward design and evaluation protocol have load-bearing weaknesses that need to be addressed before the state-of-the-art claim is fully supported.

major comments (4)
  1. [III.C / Algorithm 1] The prose in Section III.C states that the total reward is a weighted sum of R_V and R_G, but Algorithm 1 returns the product R_V * R_G. These are different training objectives: with a product, a failure on either dimension drives the reward to zero, whereas with a sum the two terms can trade off. The authors need to state which objective was actually used in the GRPO runs, specify the weights if a sum was used, and reconcile the text with the algorithm. This matters because the answer to RQ2 depends on the reward function that was actually optimized.
  2. [IV.C / Algorithm 1] The generality reward R_G is estimated from only k=5 test cases sampled from the ground-truth grammar G_GT, and the set-based generality numbers in Table IV appear to use the same protocol. With k=5, a grammar that parses exactly those five strings and nothing else in the language receives R_G=1, so the reward actively encourages grammars that fit a tiny sample rather than grammars with true language-level coverage. The reported 95.92% set-based generality is therefore an upper bound on specification coverage, not a measure of it. Please evaluate generality with a substantially larger sample or with all known valid test cases, report sensitivity to k, and, ideally, test on an independent set of valid inputs not used during reward computation.
  3. [V.A / Table IV] No error bars, confidence intervals, or significance tests are reported. The evaluation uses 240 problems, so differences such as 96.66% versus 91.85% set-based validity between SAGE and multi-turn Claude 3.5 may be meaningful, but the paper provides no measure of variance. Reporting standard errors, bootstrap confidence intervals, or paired tests across problems would make the state-of-the-art claim falsifiable and would let readers judge whether the improvements over the second-best systems are reliable.
  4. [IV.A / II.A] The CCFG formalism, the human-annotated grammars, the parser used for reward computation, the dataset, and the CCFGT5 state-of-the-art baseline all originate from the same prior work by the same research group. Systematic annotation or parser errors would therefore contaminate both the RL training signal and the reported validity and generality numbers in the same direction. Please add an annotation-agreement or error-audit analysis, or validate with an independent parser and independent annotations, to show that the ground truth does not favor the proposed method by construction.
minor comments (5)
  1. [Table II] The source column labels appear to be swapped: closed-source commercial models such as ChatGPT, Gemini, and Claude are listed under "Open", while open-source models such as Gemma and Qwen are listed under "Closed". Please correct this.
  2. [II.C] In the GRPO equations, the probability ratio is written with symbols ψ_θ and ψ_θold that are never defined, and the surrounding notation is inconsistent with the standard PPO/GRPO formulation. Please define all symbols or use the conventional π notation.
  3. [Example 3, Section V.F] The text says "105 was mistakenly interpreted as105", which appears to be a formatting error for 10^5. Please clarify the intended exponent notation.
  4. [VI.B] There is a typo in the sentence "including FuzzIL [38], and and approaches based on code fragments Ifuzzer [39]"; the duplicated "and" should be removed and the sentence restructured.
  5. [VI.A] ChatTester is cited as both [24] and [33] with the same title and authors; please consolidate the duplicate reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; reward and evaluation are anchored to externally annotated ground-truth grammars, and the reported gains are empirical rather than forced by construction.

full rationale

SAGE's derivation is not circular. The RL reward in Algorithm 1 uses R_V and R_G computed by parsing strings sampled from the human-annotated ground-truth CCFG G_GT from prior work [8], and the final validity/generality metrics use the same G_GT as an oracle on a held-out 240-problem evaluation split. This is measurement against an external, human-supplied standard rather than a quantity derived from the model's own outputs or from fitted parameters renamed as predictions. The fact that R_G coincides with the evaluation metric is the intended RLVR design, not a tautology: the model can fail to maximize it, and the ablation in Table V shows that the supervised-only baseline scores lower on the same metric. The state-of-the-art comparison includes the intra-lab baseline CCFGT5 [8], but Table IV also reports 17 independent open- and closed-source LLMs, so the central claim does not reduce to a self-citation chain. The k=5 sampling in Algorithm 1 is a legitimate benchmarking limitation, since set-based generality over five sampled strings is a weak proxy for full-language coverage, but it is a correctness/robustness concern rather than a definitional equivalence or forced prediction. The inconsistency between Section III.C's prose (weighted sum) and Algorithm 1's product return is a reproducibility issue, not circularity. No load-bearing step reduces by construction to its own input, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The claim rests on the quality of the CCFG annotations and parser from prior work [8], on the validity of sampled estimates for reward computation, and on the representativeness of the CodeContests split. No new entities are introduced; the free parameters are standard training hyperparameters plus the hand-chosen reward sample size k=5.

free parameters (6)
  • k (reward sample count) = 5
    Number of test cases sampled in Algorithm 1 to estimate element-based validity and generality; a hand-chosen value that directly shapes the RL reward and can bias metric estimates.
  • SFT learning rate = 1e-5
    Chosen hyperparameter for supervised fine-tuning, Table I(a).
  • RL learning rate = 5e-6
    Chosen hyperparameter for GRPO, Table I(b).
  • Temperature = 0.9
    Sampling temperature for RL generation, Table I(b).
  • Top-P = 0.9
    Nucleus sampling parameter for RL, Table I(b).
  • Clip ratio = 0.2
    Clipping ratio for the GRPO objective, Table I(b).
assumptions (5)
  • domain assumption Human-annotated CCFGs from [8] are correct, complete, and unbiased ground truth for validity and generality.
    Used in Section IV.A as training and evaluation data, and in Algorithm 1 as the parser oracle; errors would propagate to the reward and all reported metrics.
  • domain assumption The CCFG parser and validator from [8] correctly determine well-formedness, validity, and generality.
    The reward and all reported metrics depend on this external parser; the paper provides no independent verification of the parser's correctness.
  • ad hoc to paper Sampling k=5 test cases per grammar estimates element-based validity and generality accurately enough for RL.
    Algorithm 1 uses k=5; this is a small sample chosen without justification, and can make reward signals noisy or biased.
  • domain assumption The 240 held-out CodeContests problems are representative of the test distribution.
    Generalization claims in RQ1 and RQ2 rely on this held-out split from the same dataset as training.
  • domain assumption Element-based and set-based validity, generality, and effectiveness are appropriate proxies for test case quality.
    The paper's central quality claims rest on these metrics; they may not fully capture real-world defect detection ability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs." pith.science (2026). https://pith.science/paper/RLMRB4DF

@misc{pith2026250611081,
  author       = {Pith},
  title        = {Pith review of: SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RLMRB4DF}},
  note         = {Machine review of arXiv:2506.11081}
}
read the original abstract

Grammar-based test case generation has proven effective for competitive programming problems, but generating valid and general grammars from natural language specifications remains a key challenge, especially under limited supervision. Context-Free Grammars with Counters (CCFGs) have recently been introduced as a formalism to represent such specifications with logical constraints by storing and reusing counter values during derivation. In this work, we explore the use of open-source large language models (LLMs) to induce CCFGs from specifications using a small number of labeled examples and verifiable reward-guided reinforcement learning. Our approach first fine-tunes an open-source LLM to perform specification-to-grammar translation, and further applies Group Relative Policy Optimization (GRPO) to enhance grammar validity and generality. We also examine the effectiveness of iterative feedback for open and closed-source LLMs in correcting syntactic and semantic errors in generated grammars. Experimental results show that our approach SAGE achieves stronger generalization and outperforms 17 open and closed-source LLMs in both grammar quality and test effectiveness, improving over the state-of-the-art by 15.92%p in grammar validity and 12.34%p in test effectiveness. We provide our implementation and dataset at the following anonymous repository:https://anonymous.4open.science/r/SAGE-5714

Figures

Figures reproduced from arXiv: 2506.11081 by the authors.

Figure 1
Figure 1. Overview of the proposed framework ⋂ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of validity and generality scores. Set-based validity is [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the iterative grammar refinement process. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Changes in (set-based) validity, generality, and effectiveness over [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 29 canonical work pages

  1. [1]

    An orchestrated survey of methodologies for automated software test case generation,

    S. Anand, E. K. Burke, T. Y . Chen, J. A. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, and P. McMinn, “An orchestrated survey of methodologies for automated software test case generation,”Journal of systems and software, vol. 86, no. 8, pp. 1978– 2001, 2013

    work page 1978

  2. [2]

    EvoSuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” inSIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering and 13th European Software Engineering Conference. ACM, 2011, pp. 416– 419

    work page 2011

  3. [3]

    CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks,

    R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. R. Choudhury, L. Decker, V . Thost, L. Buratti, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss, “CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks,” inProceedings of the Neural Information Processing Systems Track on Dat...

    work page 2021

  4. [4]

    RePair: Automated program repair with process-based feedback,

    Y . Zhao, Z. Huang, Y . Ma, R. Li, K. Zhang, H. Jiang, Q. Liu, L. Zhu, and Y . Su, “RePair: Automated program repair with process-based feedback,” inFindings of the Association for Computational Linguistics, 2024, pp. 16 415–16 429

    work page 2024

  5. [5]

    Predicting patch correctness based on the similarity of failing test cases,

    H. Tian, Y . Li, W. Pian, A. K. Kabor ´e, K. Liu, A. Habib, J. Klein, and T. F. Bissyand ´e, “Predicting patch correctness based on the similarity of failing test cases,”ACM Transactions on Software Engineering and Methodology, vol. 31, no. 4, pp. 77:1–77:30, 2022

    work page 2022

  6. [6]

    Effective test generation using pre-trained large language models and mutation testing,

    A. M. Dakhel, A. Nikanjam, V . Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,”Information and Software Technology, vol. 171, p. 107468, 2024

    2024

  7. [7]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” inAdvances in Neural Information Processing Systems, 2023

    work page 2023

  8. [8]

    Logicase: Effective test case generation from logical description in competitive programming,

    S. Sung, Aditi, D. Kim, Y .-S. Han, and S.-K. Ko, “Logicase: Effective test case generation from logical description in competitive programming,” inProceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI-25), 2025, to appear. [Online]. Available: https://arxiv.org/abs/2505.15039

    arXiv 2025

Show all 47 references
  1. [9]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V . Zhao, Y . Huang, A. Dai, H. Y...

    2022

  2. [10]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” inThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

    2023

  3. [11]

    What makes large language models reason in (multi- turn) code generation?

    K. Zheng, J. Decugis, J. Gehring, T. Cohen, B. N ´eXuanjinggrevergne, and G. Synnaeve, “What makes large language models reason in (multi- turn) code generation?” inThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. Op...

    2025

  4. [12]

    Gramatron: Effective grammar-aware fuzzing,

    P. Srivastava and M. Payer, “Gramatron: Effective grammar-aware fuzzing,” inProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 2021, pp. 244– 256

    2021

  5. [13]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y . K. Li, Y . Wu, and D. Guo, “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”CoRR, vol. abs/2402.03300, 2024

    2024 arXiv

  6. [14]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”CoRR, vol. abs/1707.06347, 2017

    2017 arXiv

  7. [15]

    Competition- level code generation with AlphaCode,

    Y . Li, D. H. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Ko...

    2022

  8. [16]

    Gemma: Open models based on gemini research and technology,

    T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Love, P. Tafti, L. Hussenot, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. H ´eliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, ...

    2024 doi arXiv

  9. [17]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. ...

    2023 doi arXiv

  10. [18]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Rozi `ere, B. Biron, B. ...

    2024 doi arXiv

  11. [19]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y . Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W. Li, M. Risdal, J. Li, J. Zhu, T. Y . Zhuo, E. Zheltonozhskii, N....

    2024 doi arXiv

  12. [20]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,”CoRR, vol. abs/2310.06825,

    arXiv

  13. [21]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y . Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. L...

    2025

  14. [22]

    Available: https://doi.org/10.48550/arXiv.2310.06825

    [Online]. Available: https://doi.org/10.48550/arXiv.2310.06825

    doi arXiv

  15. [23]

    Codet: Code generation with generated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J. Lou, and W. Chen, “Codet: Code generation with generated tests,” inThe Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    2023

  16. [24]

    TestAug: A framework for augmenting capability-based NLP tests,

    G. Yang, M. Haque, Q. Song, W. Yang, and X. Liu, “TestAug: A framework for augmenting capability-based NLP tests,” inProceedings of the 29th International Conference on Computational Linguistics. International Committee on Computational Linguistics, 2022, pp. 3480– 3495

    2022

  17. [25]

    Testbench: Evaluating class-level test case generation capability of large language models,

    Q. Zhang, Y . Shang, C. Fang, S. Gu, J. Zhou, and Z. Chen, “Testbench: Evaluating class-level test case generation capability of large language models,”CoRR, vol. abs/2409.17561, 2024

    2024 arXiv

  18. [27]

    Automated unit test improvement using large language models at meta,

    N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at meta,” inCompanion Proceedings of the 32nd ACM International Conference on the Foun- dations of Sof...

    2024

  19. [28]

    Learning deep semantics for test completion,

    P. Nie, R. Banerjee, J. J. Li, R. J. Mooney, and M. Gligoric, “Learning deep semantics for test completion,” in45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 2111–2123

    2023

  20. [29]

    Testspark: Intellij idea’s ultimate test generation companion,

    A. Sapozhnikov, M. Olsthoorn, A. Panichella, V . Kovalenko, and P. Derakhshanfar, “Testspark: Intellij idea’s ultimate test generation companion,” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE Companion 20...

    2024

  21. [30]

    Adaptive test generation using a large language model,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “Adaptive test generation using a large language model,”CoRR, vol. abs/2302.06527, 2023

    2023 arXiv

  22. [31]

    Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code,

    Y . Qing, B. Zhu, M. Du, Z. Guo, T. Y . Zhuo, Q. Zhang, J. M. Zhang, H. Cui, S.-M. Yiu, D. Huang, S.-K. Ng, and L. A. Tuan, “Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code,” 2025. [Online]. Available: https://arxiv.org/abs/2505.13004

    2025 arXiv

  23. [32]

    TESTEV AL: benchmarking large language models for test case generation,

    W. Wang, C. Yang, Z. Wang, Y . Huang, Z. Chu, D. Song, L. Zhang, A. R. Chen, and L. Ma, “TESTEV AL: benchmarking large language models for test case generation,”CoRR, vol. abs/2406.04531, 2024

    2024 arXiv

  24. [33]

    Evaluating and improving chatgpt for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,”Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 1703–1726, 2024

    2024

  25. [34]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, Porto de Galinhas, Brazil, July 15-19, 2024, M...

    2024

  26. [35]

    Grammar-based whitebox fuzzing,

    P. Godefroid, A. Kiezun, and M. Y . Levin, “Grammar-based whitebox fuzzing,” inProceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). ACM, 2008, pp. 206–215

    2008

  27. [36]

    American fuzzy lop (afl),

    M. Zalewski, “American fuzzy lop (afl),” 2014, https://lcamtuf. coredump.cx/afl/

    2014

  28. [37]

    CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language mod- els,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language mod- els,” inProceedings of the 45th International Conference on Software Engineering. IEEE, 2023, pp. 919–931

    2023

  29. [38]

    Superion: grammar-aware greybox fuzzing,

    J. Wang, B. Chen, L. Wei, and Y . Liu, “Superion: grammar-aware greybox fuzzing,” inProceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25- 31, 2019, J. M. Atlee, T. Bultan, and J. Whittle, Eds. IEEE / ACM, 2019, pp. 724–735

    2019

  30. [39]

    Ifuzzer: An evolutionary interpreter fuzzer using genetic programming,

    S. Veggalam, S. Rawat, I. Haller, and H. Bos, “Ifuzzer: An evolutionary interpreter fuzzer using genetic programming,” inComputer Security - ESORICS 2016 - 21st European Symposium on Research in Computer Security, Heraklion, Greece, September 26-30, 2016, Proceedings, Part I, ...

    2016

  31. [40]

    Fuzzil: Coverage guided fuzzing for javascript engines,

    S. Groß, “Fuzzil: Coverage guided fuzzing for javascript engines,” Department of Informatics, Karlsruhe Institute of Technology, 2018

    2018

  32. [41]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions...

    2022

  33. [42]

    Evosuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” inSIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hung...

    2011

  34. [43]

    Mercury: A code efficiency benchmark for code large language models,

    M. Du, A. T. Luu, B. Ji, Q. Liu, and S. Ng, “Mercury: A code efficiency benchmark for code large language models,” inAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, Dece...

    2024

  35. [44]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” 2024

    2024

  36. [45]

    Effibench: Benchmarking the efficiency of automatically generated code,

    D. Huang, Y . Qing, W. Shang, H. Cui, and J. M. Zhang, “Effibench: Benchmarking the efficiency of automatically generated code,” in Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. ...

    2024

  37. [46]

    ACE- CODER: acing coder RL via automated test-case synthesis,

    H. Zeng, D. Jiang, H. Wang, P. Nie, X. Chen, and W. Chen, “ACE- CODER: acing coder RL via automated test-case synthesis,”CoRR, vol. abs/2502.01718, 2025

    2025 arXiv

  38. [48]

    Absolute zero: Reinforced self-play reasoning with zero data,

    A. Zhao, Y . Wu, Y . Yue, T. Wu, Q. Xu, Y . Yue, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang, “Absolute zero: Reinforced self-play reasoning with zero data,” 2025

    2025

  39. [2023]

    OpenReview.net, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.