Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Rethinking Repetition Problems of LLMs in Code Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that penalizing repetition at the level of grammar rules—not surface text—breaks the structural loops that trap LLMs during code generation, and that doing so also improves functional correctness on standard benchmarks.

desk verdict Novel framing of structural repetition in code generation and a useful new dataset, but the core penalty formula as printed cannot implement the claimed selective decay; the experiments are encouraging, so the paper deserves a serious referee despite the formal bug. read the letter →

arxiv 2505.10402 v1 pith:YGFGVR3K submitted 2025-05-15 cs.CL cs.AIcs.LGcs.SE

Yihong Dong , Yuchen Liu , Xue Jiang , Zhi Jin , Ge Li This is my paper · ORCID
classification cs.CLcs.AIcs.LGcs.SE
keywords codegenerationstructuralrepetitiongrammar-baseddecodingpushdownautomatonpenaltylargelanguagemodelsstrategyRepetEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to solve a failure mode of large language models that write code: the model gets stuck generating endless variations of the same code shape, such as repeated elif branches or repeated import statements, even though the exact text differs each time. It calls this 'structural repetition' and claims it is far more common than the content-level repetition studied before. The proposed fix, RPG, is a decoding-time method that maps each generated token to the grammar rule it belongs to, detects when a sequence of grammar rules starts repeating, and applies an exponential decay to the likelihood of the tokens inside that repeated pattern. The paper argues that this realigns generation back to a correct path, and reports that RPG outperforms standard sampling methods and content-repetition baselines on its new CodeRepetEval dataset as well as on HumanEval and MBPP, with relative Pass@1 gains up to 11.3%.

What carries the argument

The central object is the pushdown automaton (PDA) reduction of a token stream into grammar-rule symbols, $\hat{x}_t = g(x_t) = [q_t, z_t]$, where $q_t$ is the PDA state and $z_t$ the stack symbol at step $t$. This mapping lets structurally identical but textually different code collapse to the same rule sequence after merging adjacent identical symbols. Repetition is then detected on that reduced sequence with suffix arrays and LCP arrays ($O(n\log n)$ time, $O(n)$ space), and the penalty is applied as an exponential decay $\lambda^{\mathrm{Count}(\mathrm{Rep}(X_{1:t}))}$ in the token scoring function. The PDA adaptation to BPE tokenization, where one token can correspond to several grammar terminals or several tokens to one terminal, is what makes the whole pipeline usable with LLM vocabularies.

What would settle it

Feed generated code with known parse trees through the BPE-adapted PDA and compare the resulting grammar-rule sequence, token by token, against the sequence read off the true parse tree; if the two differ at any token that RPG then penalizes for repetition, the central reduction is unreliable and the reported gains rest on a noisy detector.

Watch

Extended reading notes

Core claim

The paper's central discovery is that structural repetition in LLM code generation is a grammar-level phenomenon, not just a text-level one: repeated code fragments that look different on the surface share the same sequence of context-free grammar rules. RPG exploits this by using a pushdown automaton built on the programming language's grammar to reduce each generated token to a unique grammar-rule symbol, $\hat{x}_t = g(x_t) = [q_t, z_t]$, merging adjacent identical symbols into a reduction sequence $\hat{R}_{1:t}$. Suffix-array and longest-common-prefix structures then locate repeated sub-sequences in $\hat{R}_{1:t}$, and a dynamic weight $P_n(x_t \mid x_{<t}) = \lambda^{\mathrm{Count}(\mathrm{Rep}(X_{1:t}))}$ decays the scores of tokens whose grammar rules participate in the repetition. The paper claims this both stops the endless repetition and restores correct generation, and presents experiments showing consistent gains across model families and sizes, across programming languages, and on functional correctness benchmarks.

Load-bearing premise

The load-bearing premise is that the pushdown automaton's reduction of each generated token to a unique grammar rule (Eq. 3) is correct even under BPE tokenization and on partial or incomplete code; if that mapping is noisy, the repetition detector misidentifies patterns and the penalty hits the wrong tokens.

Editorial extensions

If this is right

  • Decoding-time grammar penalties can be dropped into existing LLMs without retraining or fine-tuning.
  • Repetition metrics defined on grammar rules, such as TR-N and TR-S, give a way to measure code quality beyond exact string matching.
  • The approach transfers to other programming languages simply by swapping in the target language's grammar, as demonstrated with Go.
  • RPG improves functional correctness, not just repetition avoidance: it reports relative Pass@1 gains of 8.0% on HumanEval, 11.3% on HumanEval-ET, 6.4% on MBPP, and 10.3% on MBPP-ET over greedy sampling.
  • Content repetition is a special case of structural repetition, so a grammar-level detector subsumes earlier content-repetition methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The grammar-reduction view suggests a testable extension: repetition penalties could be applied at parse-tree node level rather than token level, which might be more robust when BPE tokens split grammar terminals.
  • If the mechanism is as effective as reported, it implies that self-reinforcement in repetition is driven by the model's own confidence in the repeated structure; a penalty that flattens that confidence may generalize to other structured outputs like JSON, SQL, or formal proofs, where grammar rules are also available.
  • The paper's own data show that the probability of the repetition start token rises with each repetition, so early intervention may matter more than penalty strength; adaptive decay schedules could outperform the fixed $\lambda=0.9$.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper identifies a distinction between content repetition and structural repetition in LLM code generation, where structural repetition consists of repeated code patterns that share the same grammar-rule structure. It proposes RPG (Repetition Penalization based on Grammar), a decoding-time method that (i) reduces each generated token to a grammar-rule label via a pushdown automaton, (ii) detects repeated grammar-rule blocks using suffix arrays and LCP arrays, and (iii) applies an exponential decay weight to token scores to discourage continuations of detected repetitions. The authors construct a new dataset, CodeRepetEval, with three scenarios (artificial synthesis, code generation benchmarks, and real-world repositories), and report that RPG substantially outperforms baselines on repetition-mitigation metrics as well as on HumanEval(-ET) and MBPP(-ET) Pass@1.

Significance. The conceptual framing of structural repetition as a grammar-level phenomenon is a useful contribution, and the proposed CodeRepetEval dataset could benefit the community if it is released with clear construction details. The experiments cover multiple base models, model sizes, and an additional programming language (Go), which strengthens the generality claims. The paper also provides pseudo-code and a full Python grammar specification in the appendix, which aids reproducibility. However, the formal specification of the penalty mechanism in §4.3 is inconsistent with the prose and, as written, cannot produce the claimed token-selective effect. The primary evaluation also has a circularity concern because TR-N and TR-S are computed on the grammar-reduced representation that RPG directly manipulates. The independent Pass@k results provide some evidence of benefit, but the improvements over greedy are modest and the comparison mixes decoding policies. The central claim as stated is therefore not fully supported, though the underlying direction is promising.

major comments (4)
  1. [§4.3, Eq. (6)–(8)] Equation (6) defines Pn(xt|x<t) = λ^Count(Rep(X1:t)); at decoding time X1:t is fixed, so the weight is the same positive constant for every candidate token xt. Multiplying all candidate scores by a constant does not change the argmax in Eq. (8) and only uniformly scales the sampling distribution, so the printed equations cannot implement the claimed selective penalization of tokens that continue a repetition. The prose in §4.3 states that the weight depends on the frequency and recency of the token's associated grammar rule (g(xt)), but g(xt) appears nowhere in Eqs. (6)–(7). Please specify the token-dependent variant actually used, for example by recomputing the repetition detection for each candidate extended prefix, and give precise definitions of Count and recency.
  2. [§5.3, §6.1, Tables 1–2] TR-N and TR-S are defined on G(x), the grammar-reduced sequence that RPG directly manipulates through the PDA reduction and repetition detection. Because RPG is designed to lower repetition counts in exactly this representation, large improvements on these two metrics are partly attributable to the evaluation being aligned with the mechanism, rather than to an external property of the generated code. The independent EGP and CCP gains on CodeRepetEval and the Pass@k results provide some supporting evidence, but the paper should acknowledge this circularity and supplement the CodeRepetEval analysis with repetition metrics computed directly on the raw token sequence.
  3. [§5.2, §5.4, §6.2, Table 4] The main comparison mixes decoding policies: RPG uses deterministic argmax (Eq. (8)) while the baselines in Table 4 are sampled with temperature 0.8 (as stated in §5.4). Since sampling at temperature 0.8 substantially lowers Pass@1 relative to greedy (e.g., 0.226 vs 0.301 on HumanEval), the reported 'substantial outperformance' over the best-performing baselines is confounded. The appropriate direct comparison is against Greedy, and the improvements there are modest (absolute Pass@1 increases of 0.024–0.031 in Table 4); please add greedy-matched baselines, such as Repetition Penalty applied with greedy decoding or with the same sampling policy as RPG, and report the variance across the five trials.
  4. [§4.1, Appendix E] The claim that Eq. (3) reduces every generated token to a unique grammar rule [q_t, z_t] is not established for BPE tokenization. Appendix E proposes heuristic decompositions and aggregations for one-to-many and many-to-one token-terminal mappings, but it gives no evaluation of how often these heuristics produce a correct parse on partial or incomplete code. If the PDA reduction is noisy, the repetition detector in Eq. (5) will misidentify patterns and the penalty will be applied to the wrong tokens. Please report parser success and failure rates on the generated prefixes and describe the fallback behavior when a parse fails.
minor comments (6)
  1. [§5.3] The TR-N formula is ambiguous: it should specify that G(x)' ranges over all n-grams of the reduced sequence and that the denominator is the total number of n-gram positions; the displayed expression is missing parentheses and clarity about the uniqueness operator.
  2. [§4.2, Appendix D] The condition in Eq. (5), Suf[i-1] = Suf[i] + LCP[i], is not aligned with the pseudo-code in Appendix D, which uses variables 'previous_start' and 'current_start' without defining them; please clarify the indexing and explicitly state what 'consecutive' means.
  3. [Table 1, Table 2] Table 1's header lists 'GenLen' without an arrow indicating whether lower is better, and Table 2's caption uses 'donate' where 'denote' is meant.
  4. [§5.2, §7.2] The Repetition Penalty baseline is attributed to Keskar et al. (2019) in §5.2 but to Holtzman et al. (2020) in §7.2; please reconcile these citations and use the correct primary source.
  5. [§5.3] EGP is described both as the 'frequency with which a model successfully interrupts repetitive sequences' and as the 'proportion of end-of-sentence tokens across all samples'; these are different quantities, and the metric should be defined precisely.
  6. [Appendix B] The λ-sweep figures in Appendix B are hard to read because each panel shows three scenarios without a legend; adding per-scenario plots or a clear legend would improve interpretability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the grammar-space evaluation mirrors the method's objective but is not definitionally forced, and the external HumanEval/MBPP results provide independent grounding.

full rationale

The central mechanism (Eqs. 6–8) is a decoding-time heuristic: it uses suffix-array/LCP repetition detection on the grammar-reduced sequence to scale token scores, and its reported gains on CodeRepetEval and on HumanEval/MBPP are empirical outcomes, not identities. TR-N and TR-S are defined in the same grammar-reduced space G(x) that RPG manipulates, so the in-dataset repetition metrics are a favorable but legitimate target; a penalty that frequently fires could still harm correctness or fail to reduce measured repetition, so improvement is not automatic. The external Pass@k results on HumanEval(-ET) and MBPP(-ET) are independent of the repetition metrics and show RPG above greedy, providing outside grounding. The only self-citations are to the authors' PDA-reduction work (Dong et al., 2023b) and extended benchmarks (Dong et al., 2024a); neither is invoked as a uniqueness result or fitted input, and the PDA reduction is a standard compiler technique. The ambiguity in Eqs. (6)–(7) about whether Count(Rep(X_{1:t})) is recomputed per candidate and the unused g(x_t) is a specification/reproducibility concern, not a circular reduction. No fitted parameter is relabeled as a prediction. Therefore no circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its load-bearing axioms are about grammar reduction fidelity and a normative prior on repetition frequency. The single free parameter lambda is hand-picked and admitted to be suboptimal in the paper itself.

free parameters (3)
  • decay factor lambda = 0.9 (default)
    Chosen intuitively in Section 5.4; Appendix B and Figure 5 show lambda=0.6 gives better EGP and TR metrics, and the paper admits further improvement is possible with other lambda values.
  • TR-N n-gram size n = 4
    Chosen for the structural repetition metric in Section 5.3; the value is arbitrary but only affects evaluation, not the method.
  • repetition extension count in Artificial Synthesis = 5 to 10 copies
    Dataset construction detail in Section 5.1 that determines the difficulty of the artificial scenario.
assumptions (4)
  • domain assumption Python and Go are context-free languages and their grammar rules represent the code structure relevant to repetition.
    The whole method maps generated code to grammar rules and treats repeated nonterminal patterns as defects; stated in Section 3 and used throughout.
  • domain assumption The pushdown automaton reduction g(x_t) = [q_t, z_t] is unique and correct for each BPE token during generation.
    Appendix E describes heuristic adaptations for BPE mismatches; if the reduction is ambiguous or wrong, the repetition detection is unreliable.
  • domain assumption Human-written code has decreasing probability of repeated grammar structures.
    This normative assumption, introduced in Section 2, justifies penalizing repetitions by exponential decay; legitimate repeated code may be over-penalized.
  • ad hoc to paper Suffix-array and LCP with condition Suf[i-1] = Suf[i] + LCP[i] captures all relevant repetitions.
    The condition only flags consecutive repeated segments that are adjacent in the reduced sequence, potentially missing interleaved or distant repetitions; this design choice appears in Eq. (5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Repetition Problems of LLMs in Code Generation." pith.science (2026). https://pith.science/paper/YGFGVR3K

@misc{pith2026250510402,
  author       = {Pith},
  title        = {Pith review of: Rethinking Repetition Problems of LLMs in Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGFGVR3K}},
  note         = {Machine review of arXiv:2505.10402}
}
read the original abstract

With the advent of neural language models, the performance of code generation has been significantly boosted. However, the problem of repetitions during the generation process continues to linger. Previous work has primarily focused on content repetition, which is merely a fraction of the broader repetition problem in code generation. A more prevalent and challenging problem is structural repetition. In structural repetition, the repeated code appears in various patterns but possesses a fixed structure, which can be inherently reflected in grammar. In this paper, we formally define structural repetition and propose an efficient decoding approach called RPG, which stands for Repetition Penalization based on Grammar, to alleviate the repetition problems in code generation for LLMs. Specifically, RPG first leverages grammar rules to identify repetition problems during code generation, and then strategically decays the likelihood of critical tokens that contribute to repetitions, thereby mitigating them in code generation. To facilitate this study, we construct a new dataset CodeRepetEval to comprehensively evaluate approaches for mitigating the repetition problems in code generation. Extensive experimental results demonstrate that RPG substantially outperforms the best-performing baselines on CodeRepetEval dataset as well as HumanEval and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code.

Figures

Figures reproduced from arXiv: 2505.10402 by the authors.

Figure 1
Figure 1. Examples of repetition problems in code generation, collected from the well-trained LLMs, e.g., CodeL [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A case of structural repetition generated by CodeLlama with temperature = 0, where the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The performance of RPG applied to LLMs of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Two cases of generating structural repetition and the effect of our approach on them. LLMs succumb [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: The influence of hyper-parameters λ on Artificial Synthesis, Code Generation Benchmarks, and Real￾world Repositories scenarios of CodeRepetEval dataset, as well as HumanEval and MBPP benchmarks. We use the gray dashed line to represent the employed hyper-parameters. C …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning

    cs.SE 2025-07 conditional novelty 6.0 of 10

    CodeReasoner combines a concise execution-focused dataset, instruction tuning, and GRPO RL to make 7B/14B models match or beat GPT-4o on code reasoning benchmarks.

Reference graph

Works this paper leans on

49 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    V Aho Alfred, S Lam Monica, and D Ullman Jeffrey. 2007. Compilers Principles, Techniques & Tools. pearson Education

    work page 2007

  2. [2]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR, abs/2108.07732

    arXiv 2021

  3. [3]

    Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2019. Language gans falling short. In (ICLR)

    work page 2019

  4. [4]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian - Guang Lou, and Weizhu Chen. 2023. CodeT : Code generation with generated tests. In ICLR

    work page 2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad B...

    arXiv 2021

  6. [6]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2024 a . Codescore: Evaluating code generation by learning code execution. ACM TOSEM

    work page 2024

  7. [7]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023 a . Self-collaboration code generation via chatgpt. CoRR, abs/2304.07590

    arXiv 2023

  8. [8]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.716 Generalization or memorization: Data contamination and trustworthy evaluation for large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039--12050, Bangkok, Thailand. Ass...

    doi:10.18653/v1/2024.findings-acl.716 2024

Show all 49 references
  1. [9]

    Yihong Dong, Ge Li, and Zhi Jin. 2023 b . CODEP: grammatical seq2seq model for general-purpose code generation. In ISSTA , pages 188--198. ACM

    2023

  2. [10]

    Aryaz Eghbali and Michael Pradel. 2022. Crystalbleu: Precisely and efficiently measuring the similarity of code. In ICSE-Companion, pages 341--342. ACM/IEEE

    2022

  3. [11]

    Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In ACL (1) , pages 889--898. Association for Computational Linguistics

    2018

  4. [12]

    Zihao Fu, Wai Lam, Anthony Man - Cho So, and Bei Shi. 2021 a . A theoretical analysis of the repetition problem in text generation. In AAAI , pages 12848--12856. AAAI Press

    2021

  5. [13]

    Zihao Fu, Wai Lam, Anthony Man - Cho So, and Bei Shi. 2021 b . A theoretical analysis of the repetition problem in text generation. In AAAI , pages 12848--12856. AAAI Press

    2021

  6. [14]

    Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017. Program synthesis. Foundations and Trends in Programming Languages , 4(1-2):1--119

    2017

  7. [15]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross-modal pre-training for code representation. In ACL (1) , pages 7212--7225. Association for Computational Linguistics

    2022

  8. [16]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming - the rise of code intelligence. CoRR, abs/2401.14196

    2024 arXiv

  9. [17]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR . OpenReview.net

    2020

  10. [18]

    Xue Jiang, Yihong Dong, Yongding Tao, Huanyu Liu, Zhi Jin, Wenpin Jiao, and Ge Li. 2024. ROCODE: integrating backtracking mechanism and program analysis in large language models for code generation. CoRR, abs/2411.07112

    2024 arXiv

  11. [19]

    Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. Self-planning code generation with large language model. CoRR, abs/2303.06689

    2023

  12. [20]

    Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021. Treebert: A tree-based pre-trained model for programming language. In UAI , volume 161 of Proceedings of Machine Learning Research, pages 54--63. AUAI Press

    2021

  13. [21]

    Varshney, Caiming Xiong, and Richard Socher

    Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858

    2019 arXiv

  14. [22]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu - Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In NeurIPS

    2022

  15. [23]

    Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. 2023 a . Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. In NeurIPS

    2023

  16. [24]

    Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. 2024 a . Evocodebench: An evolving code generation benchmark with domain-specific evaluations. In NeurIPS

    2024

  17. [25]

    Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. 2024 b . Deveval: A manually-annotated cod...

    2024

  18. [26]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy - Poirier, Jo \ a o Mont...

    2023 arXiv

  19. [27]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092--1097

    2022

  20. [28]

    Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tom \' a s Kocisk \' y , Fumin Wang, and Andrew W. Senior. 2016. Latent predictor networks for code generation. In ACL (1) . The Association for Computer Linguistics

    2016

  21. [29]

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, and Li Zhang. 2024. Exploring and evaluating hallucinations in llm-powered code generation. CoRR, abs/2404.00971

    2024

  22. [30]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In NeurIPS

    2023

  23. [31]

    Reps, Swarat Chaudhuri, and Christopher M

    Rohan Mukherjee, Yeming Wen, Dipak Chaudhari, Thomas W. Reps, Swarat Chaudhuri, and Christopher M. Jermaine. 2021. Neural program generation modulo static analysis. In NeurIPS, pages 18984--18996

    2021

  24. [32]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In ICLR . OpenReview.net

    2023

  25. [33]

    OpenAI. 2022. https://openai.com/blog/chatgpt/ ChatGPT

    2022

  26. [34]

    OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774

    2023 arXiv

  27. [35]

    Vechev, and Eran Yahav

    Veselin Raychev, Martin T. Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In PLDI , pages 419--428. ACM

    2014

  28. [36]

    Baptiste Rozi \` e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \' e r \' e my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton - Ferrer, Aaron Grattafiori, Wenhan Xiong,...

    2023 arXiv

  29. [37]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL (1) . The Association for Computer Linguistics

    2016

  30. [38]

    Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In NeurIPS

    2022

  31. [39]

    Zeyu Sun, Qihao Zhu, Lili Mou, Yingfei Xiong, Ge Li, and Lu Zhang. 2019. A grammar-based structural CNN decoder for code generation. In AAAI , pages 7055--7062. AAAI Press

    2019

  32. [40]

    Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang. 2020. Treegen: A tree-based transformer architecture for code generation. In AAAI , pages 8984--8991. AAAI Press

    2020

  33. [41]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In ACL (1) , pages 13484--13508. Association for Computational Linguistics

    2023

  34. [42]

    Joty, and teven C

    Yue Wang, Weishi Wang, Shafiq R. Joty, and teven C. H. Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In EMNLP (1) , pages 8696--8708

    2021

  35. [43]

    Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code generation as a dual task of code summarization. In NeurIPS, pages 6559--6569

    2019

  36. [44]

    Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In NeurIPS

    2022

  37. [45]

    Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In EMNLP (Demonstration) , pages 7--12. Association for Computational Linguistics

    2018

  38. [46]

    Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, and Zhi Jin. 2024. Codedpo: Aligning code models with self generated and verified source code. CoRR, abs/2410.05605

    2024 arXiv

  39. [47]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. CoRR, abs/2303.17568

    2023 arXiv

  40. [48]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.