Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that translating competitive-programming input specifications into Context-Free Grammars with Counters, using a fine-tuned CodeT5 model, produces test cases that are more often fully valid as sets and catch more…

desk verdict A genuinely new method that likely works, but the evaluation leans on the authors' own grammars as ground truth, so the headline numbers are plausible yet not firmly established. read the letter →

arxiv 2505.15039 v1 pith:7RV7WORG submitted 2025-05-21 cs.SE cs.AI

Sicheol Sung , Aditi , Dogyu kim , Yo-Sub Han , Sang-Ki Ko This is my paper
classification cs.SEcs.AI
keywords testcasegenerationcontext-freegrammarswithcounterscompetitiveprogrammingCodeT5grammar-basedfuzzinginputspecificationtranslationContestsLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a grammar-first pipeline for automated test-case generation in competitive programming. Instead of asking a large language model to write test cases directly, the authors fine-tune a CodeT5 model to translate each problem's natural-language input specification into a Context-Free Grammar with Counters (CCFG), and then sample test cases from that grammar. They claim this approach produces test suites whose entire set is valid more often and that expose incorrect algorithms more effectively than direct ChatGPT or Gemini generation: on the CodeContests evaluation set, CCFGT5-10 reaches 81.18% set-based validity and 67.73% set-based effectiveness, compared with 78.97% and 63.54% for direct ChatGPT generation. The practical stake is that a correct grammar, once learned, can generate any number of specification-compliant tests for free, without per-case validation or further model calls.

What carries the argument

The central object is the Context-Free Grammar with Counters (CCFG), a context-free grammar in which some nonterminals carry an index that acts as a counter, for example $T_i$ with $T_t$ expanded $t$ times before reaching $T_1$; the counter is set by an integer already read from the input and decremented at each application. This lets the grammar enforce that a variable's value determines the number of following lines, array elements, or queries, so both syntax and the semantic constraints that depend on earlier values live inside the grammar. The translation is carried out by CCFGT5, two fine-tuned CodeT5 modules, one for grammar structure and one for numerical constraints, plus a CCFG tokenizer that labels symbols with descriptors such as 'variable' and 'nonterminal.' The machinery's job is to reduce test-case generation to grammar sampling: once the grammar is faithful, every generated test case is valid, and varying the sampling interval for the counter-driven variables produces short, medium, and long cases that catch different incorrect algorithms.

What would settle it

Take roughly 30 CodeContests problems where CCFGT5-10's grammar is scored as semantically equivalent to the ground truth, sample the ten generated test cases per problem, and have independent human annotators check each input against the original problem statement; if any sampled case violates a stated numerical constraint, such as an array longer than the declared bound, or an implied structural rule, the validity and effectiveness numbers are inflated.

Watch

Extended reading notes

Core claim

The central claim is that CCFG-based test cases with the best CCFGT5 configuration exhibit the highest set-based validity and both types of effectiveness, achieving 81.18%, 42.26%, and 67.73% for set-based validity, element-based effectiveness, and set-based effectiveness, respectively. Ground-truth grammars, which the authors use as an upper bound, reach 100%, 52.51%, and 83.40% on the same metrics. The paper argues that the mechanism is a formal grammar with counters: the grammar records that an integer read earlier, such as $t$ or $n$, controls how many times a subsequent block repeats, so generation is valid by construction rather than by checking. On normal and hard specifications, CCFGT5-10's set-based validity remains 76.24% and 90.91%, while direct ChatGPT falls to 65.35% and 63.64%, which the authors cite as evidence that the formalism pays off precisely where specifications are complex.

Load-bearing premise

The evaluation assumes the 1,500 manually written CCFGs are correct and complete formalizations of the intended input specifications, kept only when they parse all public and private CodeContests test cases, since every validity, generality, and effectiveness score compares generated grammars against these author-created labels.

Editorial extensions

If this is right

  • Once a CCFG is correct, no per-test-case validation is needed; validity is a property of the grammar, so arbitrary numbers of valid test cases can be sampled cheaply.
  • The approach's advantage concentrates where specifications are complex: on hard specifications CCFGT5-10 reaches 90.91% set-based validity, while direct ChatGPT generation reaches 63.64%.
  • Mixed-length suites are the most effective setting: the union of short and long test cases outperforms either length alone, and the paper's 4-short/3-medium/3-long mix gives the best 10-case set-based effectiveness.
  • Set-based generality tracks set-based validity closely for CCFGT5-10, meaning the generated grammars cover nearly the same test-case space as the ground truth when they are valid.
  • Because grammar generation is a one-time cost, scaling to 30 test cases is straightforward and improves set-based effectiveness, unlike direct generation whose cost grows with each additional LLM prompt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A generated grammar that passes the paper's filters, parsing all public and private CodeContests tests for its problem and resembling training grammars syntactically, could be trusted to generate tests without a human-written ground truth; this suggests a fully automatic validation loop using only existing tests, though the paper itself keeps ground-truth grammars as labels.
  • The counter formalism and the translation model could transfer to other specification-driven generation tasks, such as synthesizing input data for program repair or property-based testing, because the CCFG is an executable model of the spec; whether transfer holds across domains is untested.
  • The failure analysis implies that normalizing constraint language, for instance rewriting 'the number of digits does not exceed 10^5' into $n \le 10^5$, and standardizing variable naming between grammar and constraint modules would likely close most of the gap between CCFGT5-10 and ground-truth grammars.
  • The comparison of set-based versus element-based validity suggests a cheap quality signal: when a generated grammar is valid on every sampled test case but its set-based validity is lower than its element-based validity, the grammar likely misses a rare branching condition, so validity alone should not be used to certify completeness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Context-Free Grammars with Counters (CCFGs), an extension of CFGs intended to capture counting and repetition constraints in competitive programming input specifications. It fine-tunes CodeT5 in two modules (grammar and constraints) to translate natural-language specifications into CCFGs, then samples test cases from the resulting grammars. On the CodeContests dataset, with ground-truth CCFGs manually written by the authors for 1,500 problems (1,200 training, 300 evaluation, reduced to 271 after filtering), the paper reports that CCFGT5 with beam size 10 achieves the highest set-based validity and set-based effectiveness among the compared methods, including direct ChatGPT generation: 81.18% vs 78.97% set-based validity and 67.73% vs 63.54% set-based effectiveness. The paper also reports observations about specification difficulty, test-case length, and grammar generality.

Significance. If the evaluation is trustworthy, the paper makes a useful contribution to automated test case generation: it provides a grammar-driven framework that avoids per-problem manual grammar writing at inference time, and the released code and data support reproduction and further study. The CCFG formalism, once precisely defined, could also inform grammar-based fuzzing and benchmark construction. However, the headline numerical gains are modest, and the evaluation's validity oracle is the authors' own manual CCFG annotations, so the central comparative claim is not yet firmly established. The paper is transparent about some failure cases in Section 4.6, which is a positive feature.

major comments (5)
  1. [Section 3.1] The paper never gives a formal definition of CCFG. Example 3 uses subscripted nonterminals such as T_i and T_{i-1} together with informal counter operations ('set the value of an internal counter to t', 'decrement the counter by 1'), but no formal syntax or semantics is provided. Because CCFG is the central contribution, and because the paper claims it is substantially different from the grammar model of Chistikov et al. [2018], the authors need to define the formalism precisely: how counters are declared, initialized, decremented, and how acceptance of a string is characterized. Without this, the implementation, the undecidability remark in Section 4.4, and the reproducibility of the evaluation cannot be assessed.
  2. [Sections 4.1, 4.4, 4.5] All validity, generality, and equality metrics in Tables 2 and 5 are computed against the authors' manually written ground-truth CCFGs. The only validation reported for these labels is that each grammar parses all public and private CodeContests test cases (Section 4.5). This is a necessary but insufficient check: an over-restrictive grammar that omits valid inputs can still pass because CodeContests test cases are a sparse sample of the input space. Since CCFGT5 is fine-tuned on the same annotation style, its outputs are likely to match the labels, whereas direct LLM generation may produce valid but unconventional test cases that are marked invalid. This bias directly affects the headline comparison in Table 2 (81.18% vs 78.97% set-based validity; 67.73% vs 63.54% set-based effectiveness). The authors should mitigate this concern by, for example, having independent annotators re-label a sample of problems, manually auditing test cases rejected by the ground-truth grammars, or reporting how the headline gaps change under plausible label perturbations.
  3. [Section 4.5, Table 2] The paper reports no variance or significance information. Set-based validity is a per-problem binary outcome and set-based effectiveness is a per-problem proportion, so the observed differences of 2.21 and 4.19 percentage points over roughly 270 problems may be within sampling noise. In addition, the rows of Table 2 report different 'Well-defined(#)' counts (e.g., 271 for direct ChatGPT vs 264 for CCFGT5 10) without explaining how these counts enter the denominators of the reported percentages, which prevents a reader from reconstructing the paired per-problem comparison. Please provide a contingency table, confidence intervals, and a paired statistical test such as a Wilcoxon signed-rank test or a bootstrap over problems.
  4. [Section 4.5, Table 3, Observation 1] The claim that CCFGs are especially effective for complex input specifications rests on the Hard category, which contains only 11 evaluation problems; CCFGT5 10's reported 90.91% hard set-based validity corresponds to 10 of 11 problems. This sample is too small to support the qualitative conclusion, and it is also the category where annotation errors are most plausible. The authors should either collect substantially more hard problems or soften Observation 1 accordingly.
  5. [Section 4.1] The evaluation excludes 29 of the 300 evaluation problems: 6 because CodeContests has no incorrect solutions for them and 23 because the authors' CCFG implementation cannot process their human-labeled grammars. If the excluded problems differ systematically from the included ones, the results in Table 2 may not generalize. The authors should report the difficulty distribution and specification features of the excluded problems and show that the main comparisons are stable under different inclusion criteria.
minor comments (6)
  1. [Section 4.4] In the definition of set-based generality, the phrase 'if and only if the test case-based generality is also 1' should presumably read 'element-based generality'; the terminology is inconsistent.
  2. [Table 2] The table is hard to read because numbers are run together in the extracted text (e.g., '26482.3281.18'); use clear column separation and consistent decimal formatting.
  3. [Section 4.3] The paper does not report the exact prompt templates, the number of chain-of-thought examples, or the sampling parameters used for ChatGPT and Gemini, which are needed for reproducing the direct-generation baselines.
  4. [Section 3.2] The text says the authors select 'a pair of grammar and constraints among top-k grammars and constraints' but the value of k is never specified, making the CCFGT5 inference procedure underspecified.
  5. [Section 4.4] Element-based effectiveness is defined to be 0 whenever the test-case set is invalid, so it is not actually an average over valid individual test cases; the authors should clarify why this coupling was chosen.
  6. [Section 5] The sentence 'We will expand CCFGs to handle broader input domains and optimizing sampling strategies' is ungrammatical; 'and optimizing' should likely be 'and optimize'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CCFGT5 is evaluated on held-out author-annotated grammars, a standard supervised benchmark setup; no prediction reduces to its training input.

full rationale

The paper's derivation chain is empirical rather than derivational. CCFGT5 is fine-tuned on 1,200 author-written CCFGs and evaluated on 271 disjoint problems (Section 4.1). Validity and generality are defined with respect to these human-labeled ground-truth grammars (Section 4.4), which are sanity-checked by requiring them to parse all public and private CodeContests test cases (Section 4.5). This is a supervised learning setup with a labeled benchmark: the ground-truth grammars are inputs/annotations, not outputs of the model, and the reported 81.18% set-based validity and 67.73% set-based effectiveness are measured on held-out labels. There is no fitted parameter that is renamed as a prediction, no load-bearing self-citation, and no uniqueness theorem imported from the authors. One could question whether author-written grammars are over-restrictive and thus bias the comparison against direct LLM generation, but that is a benchmark-validity concern, not a circular-reasoning defect: the metrics remain well-defined even if the ground truth is imperfect. Consequently, no circular step can be exhibited with a quote-and-reduction, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The core technical contribution, CCFG, is an invented formal construct rather than a derived quantity, and the evaluation depends on author-labeled ground-truth grammars. The free parameters listed are handpicked experimental choices that affect the reported effectiveness numbers. There are no fitted physical constants or derivations in the paper, so the ledger mostly captures evaluation-design dependencies.

free parameters (4)
  • Sampling interval choices = (n+k), (n+log k), (n+log log k), minimum n
    Section 4.2: test case lengths are controlled by these handpicked intervals, and they directly affect the reported effectiveness numbers. No ablation is shown for alternative intervals.
  • Test case length mix = 4 short, 3 medium, 3 long
    Section 4.4: set-based effectiveness is computed on a selected ten-test-case mix; this handpicked configuration influences the comparison with other methods.
  • Fine-tuning hyperparameters = learning rate 1e-5, repetition penalty 2.5, length penalty 1.0
    Section 3.2: chosen without a reported search, and they affect the quality of generated grammars.
  • Timeout multiplier = 2x original timeout
    Section 4.4: outputs exceeding twice the original timeout are treated as incorrect. This threshold is arbitrary and can change effectiveness scores.
assumptions (4)
  • domain assumption The manually created ground-truth CCFGs are correct and complete representations of the input specifications.
    Section 4.1 states that the authors manually created CCFGs for 1,500 problems, and Section 4.4 uses these grammars as the reference for validity, generality, and semantic equivalence.
  • domain assumption CodeContests public and private test cases, plus majority-vote outputs from up to ten correct solutions, are authoritative for determining correct behavior.
    Section 4.4: the correct output for a test case is the most frequent output among up to ten correct algorithms, and the incorrect algorithm set is sampled up to ten per problem.
  • domain assumption CCFG validity and generality can be reliably approximated by empirical parsing over sampled test cases.
    Section 4.4 acknowledges that exact CCFG validity is undecidable and instead uses element-based and set-based empirical measures, which may miss specification mismatches not exposed by the sampled tests.
  • domain assumption The 271-problem filtered evaluation set is representative of competitive programming input specifications.
    Section 4.1 excludes 29 problems because they lack incorrect solutions or cannot be processed by the implementation, and ground-truth grammars are retained only when they parse all public and private CodeContests test cases, biasing the set toward problems the method can handle.
invented entities (2)
  • CCFG (Context-Free Grammars with Counters) independent evidence
    purpose: A formal device that adds counters to nonterminals so that the number of production applications can be tied to integer values read from the input, capturing repetition constraints in specifications.
    The formalism is used to generate test cases, and those test cases are checked against ground-truth grammars and CodeContests data, providing an observable handle beyond the paper's own definitions.
  • CCFGT5 translation model independent evidence
    purpose: A pair of fine-tuned CodeT5 modules that translate natural language specifications into CCFG grammar templates and constraints.
    The model's outputs are empirically evaluated against human-labeled ground-truth grammars and against LLM baselines, so the entity carries a falsifiable performance claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming." pith.science (2026). https://pith.science/paper/7RV7WORG

@misc{pith2026250515039,
  author       = {Pith},
  title        = {Pith review of: LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RV7WORG}},
  note         = {Machine review of arXiv:2505.15039}
}
read the original abstract

Automated Test Case Generation (ATCG) is crucial for evaluating software reliability, particularly in competitive programming where robust algorithm assessments depend on diverse and accurate test cases. However, existing ATCG methods often fail to meet complex specifications or generate effective corner cases, limiting their utility. In this work, we introduce Context-Free Grammars with Counters (CCFGs), a formalism that captures both syntactic and semantic structures in input specifications. Using a fine-tuned CodeT5 model, we translate natural language input specifications into CCFGs, enabling the systematic generation of high-quality test cases. Experiments on the CodeContests dataset demonstrate that CCFG-based test cases outperform baseline methods in identifying incorrect algorithms, achieving significant gains in validity and effectiveness. Our approach provides a scalable and reliable grammar-driven framework for enhancing automated competitive programming evaluations.

Figures

Figures reproduced from arXiv: 2505.15039 by the authors.

Figure 1
Figure 1. Overview of the proposed framework for generating test cases for competitive programming problems. The deep learning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SAGE combines supervised fine-tuning and GRPO reinforcement learning to generate CCFG grammars from competitive programming specs, achieving 96.66% set-based validity and 80.67% set-based effectiveness on 240 held-out...

Reference graph

Works this paper leans on

27 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Burke, Tsong Yueh Chen, John A

    Saswat Anand, Edmund K. Burke, Tsong Yueh Chen, John A. Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, and Phil McMinn. An orchestrated survey of methodologies for automated software test case generation. Journal of systems and software , 86(8):1978--2001, 2013

    work page 1978

  2. [2]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. CoRR , abs/2107.03374, 2021

    arXiv 2021

  3. [3]

    Context-free commutative grammars with integer counters and resets

    Dmitry Chistikov, Christoph Haase, and Simon Halfon. Context-free commutative grammars with integer counters and resets. Theoretical Computer Science , 735:147--161, 2018

    work page 2018

  4. [4]

    Desmarais

    Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology , 171:107468, 2024

    work page 2024

  5. [5]

    CodeBERT : A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT : A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics , volume EMNLP 2020, pages 1536--1547. Association for Computational Linguistics, 2020

    work page 2020

  6. [6]

    EvoSuite : A utomatic test suite generation for object-oriented software

    Gordon Fraser and Andrea Arcuri. EvoSuite : A utomatic test suite generation for object-oriented software. In SIGSOFT/FSE '11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering and 13th European Software Engineering Conference , pages 416--419. ACM , 2011

    work page 2011

  7. [7]

    Patrice Godefroid, Adam Kiezun, and Michael Y. Levin. Grammar-based whitebox fuzzing. In Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation , pages 206--215. ACM , 2008

    work page 2008

  8. [8]

    UniXcoder : U nified cross-modal pre-training for code representation

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. UniXcoder : U nified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages 7212--7225. Association for Computational Linguistics, 2022

    work page 2022

Show all 27 references
  1. [9]

    Hopcroft, Rajeev Motwani, and Jeffrey D

    John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. Introduction to automata theory, languages, and computation, 3rd Edition . Addison-Wesley, 2007

    2007

  2. [10]

    Lahiri, and Siddhartha Sen

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. CodaMosa : E scaping coverage plateaus in test generation with pre-trained large language models. In Proceedings of the 45th International Conference on Software Engineering , pages 919--931. IEEE , 2023

    2023

  3. [11]

    Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \' e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po - Sen Huang, Johannes Welbl, Sven ...

    2022

  4. [12]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems , 2023

    2023

  5. [13]

    Miller, Lars Fredriksen, and Bryan So

    Barton P. Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of UNIX utilities. Communications of the ACM , 33(12):32--44, 1990

    1990

  6. [14]

    Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R

    Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. CodeNet : A lar...

    2021

  7. [15]

    Code Llama : O pen foundation models for code

    Baptiste Rozi \` e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \' e r \' e my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton - Ferrer, Aaron Grattafiori, Wenhan Xiong,...

    2023 arXiv

  8. [16]

    Gramatron: E ffective grammar-aware fuzzing

    Prashast Srivastava and Mathias Payer. Gramatron: E ffective grammar-aware fuzzing. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis , pages 244--256. ACM , 2021

    2021

  9. [17]

    Gemma : O pen models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma : O pen models based on gemini research and technology. CoRR , abs/2403.08295, 2024

    2024 arXiv

  10. [18]

    Bissyand \' e

    Haoye Tian, Yinghua Li, Weiguo Pian, Abdoul Kader Kabor \' e , Kui Liu, Andrew Habib, Jacques Klein, and Tegawend \' e F. Bissyand \' e . Predicting patch correctness based on the similarity of failing test cases. ACM Transactions on Software Engineering and Methodology , 31(4...

    2022

  11. [19]

    Joty, and Steven C

    Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. CodeT5 : I dentifier-aware unified pre-trained encoder-decoder models for code understanding and generation. CoRR , abs/2109.00859, 2021

    2021 arXiv

  12. [20]

    Saurous, and Yoon Kim

    Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A. Saurous, and Yoon Kim. Grammar prompting for domain-specific language generation with large language models. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems , 2023

    2023

  13. [21]

    TESTEVAL: benchmarking large language models for test case generation

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. TESTEVAL: benchmarking large language models for test case generation. CoRR , abs/2406.04531, 2024

    2024 arXiv

  14. [22]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-Thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Inf...

    2022

  15. [23]

    Fuzz4all: Universal fuzzing with large language models

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Fuzz4all: Universal fuzzing with large language models. In Proceedings of the 46th International Conference on Software Engineering , pages 126:1--126:13. ACM , 2024

    2024

  16. [24]

    TestAug : A framework for augmenting capability-based NLP tests

    Guanqun Yang, Mirazul Haque, Qiaochu Song, Wei Yang, and Xueqing Liu. TestAug : A framework for augmenting capability-based NLP tests. In Proceedings of the 29th International Conference on Computational Linguistics , pages 3480--3495. International Committee on Computational ...

    2022

  17. [25]

    No more manual tests? evaluating and improving ChatGPT for unit test generation

    Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. No more manual tests? evaluating and improving ChatGPT for unit test generation. CoRR , abs/2305.04207, 2023

    2023 arXiv

  18. [26]

    RePair : A utomated program repair with process-based feedback

    Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, and Yu Su. RePair : A utomated program repair with process-based feedback. In Findings of the Association for Computational Linguistics , pages 16415--16429, 2024

    2024

  19. [27]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.