{"id":"01f090ec-13d4-4470-a69c-43201ebcd4c1","arxiv_id":"2505.10402","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RPG penalizes tokens that continue repeated grammar structures in LLM-generated code, reducing structural repetition and improving Pass@k on HumanEval and MBPP.","lead":"A new decoding method for large language models detects when generated code is repeating a fixed grammatical pattern and lowers the odds of continuing that pattern. The authors report fewer repetitions and higher pass rates on code benchmarks, but the main evaluation metric is defined through the same grammar mechanism the method uses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core penalty formula (Eq. 6–7) does not depend on the candidate token as printed, so the formal definition of RPG cannot implement the claimed selective decay of repetition tokens; the reported gains must come from an unspecified token-dependent variant.","rationale":"I read the paper as making a two-part central claim: grammar-rule reduction detects structural repetition (Eq. 3–5), and a token-level penalty then discourages continuation of the detected patterns (Eq. 6–8), improving repetition metrics and functional correctness. For the second part to work, the penalty must discriminate between candidate tokens that would extend a repetition and candidate tokens that would not. The paper's prose in §4.3 says exactly this ('decays the likelihood of critical tokens that contribute to repetitions,' with the weight based on 'the frequency and recency of its associated grammar rule'), but the printed formalization does not implement it: Eq. 6's right-hand side is constant in xt at a fixed decoding step, so RPG as specified is identical to the base decoder under argmax and can at most flatten the distribution under sampling. Table 4 and Figure 5 therefore cannot be explained by the published equations; some token-dependent penalty, never formally defined, must be what produces the results. This is an internal inconsistency in the core mechanism, and it is more load-bearing than the reader's chosen weakest assumption: even a flawless BPE-adapted PDA reduction cannot make a token-independent penalty selective, whereas a noisy reduction can be checked against the execution-based Pass@k and compiler-based CCP results, which are grammar-agnostic.\n\nI keep the verdict at CONDITIONAL (unchanged from the reader) rather than moving it, for three reasons. First, the intended mechanism is stated clearly enough in prose that the missing dependence on g(xt) is identifiable and mechanically correctable. Second, the released code (URL given in the paper) can settle in minutes whether the implementation uses a token-dependent penalty; if it does, the flaw is a formal presentation error. Third, the paper has independent support not touched by this concern: Pass@k on HumanEval/MBPP is measured by executing test cases, and CodeRepetEval reports CCP, both grammar-agnostic, and both show RPG ahead of Greedy, confirming that some selective mechanism is at work even if the equations fail to define it. The condition I would attach to acceptance is precise: correct Eq. 6–7 so the penalty is an explicit function of the candidate token's grammar rule (or of the per-candidate extension of Rep(X1:t)), and confirm the released implementation matches the corrected formula; otherwise the empirical comparisons cannot be attributed to the claimed grammar-based penalization. The reader reached the same CONDITIONAL verdict and listed the penalty formula in the rationale but anchored the weakest assumption elsewhere, so my agreement is partial.","tokens_in":1334,"tokens_out":3057,"duration_ms":171060,"concrete_test":"Inspect the released implementation at github.com/LYC127/RPG and locate the scoring loop that applies the penalty. Determine whether the penalty factor attached to a candidate token is a function of that token's grammar reduction g(x) (e.g., λ^count/recency of g(x) in the detected repetition patterns), or the single per-step constant of Eq. 6. Then hard-code the literal constant version — no g(x) dependence, one multiplier for the whole vocabulary per step — and re-run the CodeRepetEval 'Code Generation Benchmarks' scenario; if the sampled outputs are statistically indistinguishable (up to effective temperature) from the base decoder's, the printed equations cannot explain Table 1 and Table 4, confirming that the mechanism is an unspecified token-dependent penalty rather than the published formula.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that RPG selectively lowers the score of tokens that would continue a detected structural repetition ('strategically decays the likelihood of critical tokens that contribute to repetitions', Abstract). The formal definition does not do this. In Eq. 6, Pn(xt|x<t) = λ^Count(Rep(X1:t)); at scoring time, the prefix is fixed and the RHS is the same number for every candidate token xt. Multiplying every score by one positive constant leaves the argmax unchanged, and in sampling it merely flattens the distribution; it cannot penalize tokens that extend a repetition more than tokens that break it. Eq. 7's caption even names g(xt) as the grammar rule associated with xt, but g(xt) never appears in either equation. The prose in §4.3 says the weight depends on 'the frequency and recency of its associated grammar rule', so the equations contradict the described mechanism. A charitable rescue — interpreting Count(Rep(X1:t)) as evaluated per candidate by extending the PDA reduction and repetition detection to X1:t for each vocabulary element — would make the penalty token-dependent, but the paper never states that the suffix-array/LCP detection is recomputed per candidate at every step, nor how count/recency of a rule is defined. This is load-bearing because it is the paper's own specification of its central mechanism: if the printed equation is meant literally, RPG cannot produce the reported effects (Table 4 shows RPG differing from Greedy on HumanEval, 0.325 vs 0.301, and Figure 5 shows λ shifting EGP and Pass@1, which a constant multiplier could only do by changing effective temperature, not by grammar-targeted penalization). The reported results therefore rest on a token-dependent penalty that the paper never formally defines. This is an internal inconsistency, independent of the BPE/PDA reliability question: even a perfect grammar reduction cannot make a token-independent penalty selective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a distinction between content repetition and structural repetition in LLM code generation, where structural repetition consists of repeated code patterns that share the same grammar-rule structure. It proposes RPG (Repetition Penalization based on Grammar), a decoding-time method that (i) reduces each generated token to a grammar-rule label via a pushdown automaton, (ii) detects repeated grammar-rule blocks using suffix arrays and LCP arrays, and (iii) applies an exponential decay weight to token scores to discourage continuations of detected repetitions. The authors construct a new dataset, CodeRepetEval, with three scenarios (artificial synthesis, code generation benchmarks, and real-world repositories), and report that RPG substantially outperforms baselines on repetition-mitigation metrics as well as on HumanEval(-ET) and MBPP(-ET) Pass@1.","tokens_in":21532,"tokens_out":6033,"duration_ms":57654,"significance":"The conceptual framing of structural repetition as a grammar-level phenomenon is a useful contribution, and the proposed CodeRepetEval dataset could benefit the community if it is released with clear construction details. The experiments cover multiple base models, model sizes, and an additional programming language (Go), which strengthens the generality claims. The paper also provides pseudo-code and a full Python grammar specification in the appendix, which aids reproducibility. However, the formal specification of the penalty mechanism in §4.3 is inconsistent with the prose and, as written, cannot produce the claimed token-selective effect. The primary evaluation also has a circularity concern because TR-N and TR-S are computed on the grammar-reduced representation that RPG directly manipulates. The independent Pass@k results provide some evidence of benefit, but the improvements over greedy are modest and the comparison mixes decoding policies. The central claim as stated is therefore not fully supported, though the underlying direction is promising.","major_comments":[{"comment":"Equation (6) defines Pn(xt|x<t) = λ^Count(Rep(X1:t)); at decoding time X1:t is fixed, so the weight is the same positive constant for every candidate token xt. Multiplying all candidate scores by a constant does not change the argmax in Eq. (8) and only uniformly scales the sampling distribution, so the printed equations cannot implement the claimed selective penalization of tokens that continue a repetition. The prose in §4.3 states that the weight depends on the frequency and recency of the token's associated grammar rule (g(xt)), but g(xt) appears nowhere in Eqs. (6)–(7). Please specify the token-dependent variant actually used, for example by recomputing the repetition detection for each candidate extended prefix, and give precise definitions of Count and recency.","section":"§4.3, Eq. (6)–(8)"},{"comment":"TR-N and TR-S are defined on G(x), the grammar-reduced sequence that RPG directly manipulates through the PDA reduction and repetition detection. Because RPG is designed to lower repetition counts in exactly this representation, large improvements on these two metrics are partly attributable to the evaluation being aligned with the mechanism, rather than to an external property of the generated code. The independent EGP and CCP gains on CodeRepetEval and the Pass@k results provide some supporting evidence, but the paper should acknowledge this circularity and supplement the CodeRepetEval analysis with repetition metrics computed directly on the raw token sequence.","section":"§5.3, §6.1, Tables 1–2"},{"comment":"The main comparison mixes decoding policies: RPG uses deterministic argmax (Eq. (8)) while the baselines in Table 4 are sampled with temperature 0.8 (as stated in §5.4). Since sampling at temperature 0.8 substantially lowers Pass@1 relative to greedy (e.g., 0.226 vs 0.301 on HumanEval), the reported 'substantial outperformance' over the best-performing baselines is confounded. The appropriate direct comparison is against Greedy, and the improvements there are modest (absolute Pass@1 increases of 0.024–0.031 in Table 4); please add greedy-matched baselines, such as Repetition Penalty applied with greedy decoding or with the same sampling policy as RPG, and report the variance across the five trials.","section":"§5.2, §5.4, §6.2, Table 4"},{"comment":"The claim that Eq. (3) reduces every generated token to a unique grammar rule [q_t, z_t] is not established for BPE tokenization. Appendix E proposes heuristic decompositions and aggregations for one-to-many and many-to-one token-terminal mappings, but it gives no evaluation of how often these heuristics produce a correct parse on partial or incomplete code. If the PDA reduction is noisy, the repetition detector in Eq. (5) will misidentify patterns and the penalty will be applied to the wrong tokens. Please report parser success and failure rates on the generated prefixes and describe the fallback behavior when a parse fails.","section":"§4.1, Appendix E"}],"minor_comments":[{"comment":"The TR-N formula is ambiguous: it should specify that G(x)' ranges over all n-grams of the reduced sequence and that the denominator is the total number of n-gram positions; the displayed expression is missing parentheses and clarity about the uniqueness operator.","section":"§5.3"},{"comment":"The condition in Eq. (5), Suf[i-1] = Suf[i] + LCP[i], is not aligned with the pseudo-code in Appendix D, which uses variables 'previous_start' and 'current_start' without defining them; please clarify the indexing and explicitly state what 'consecutive' means.","section":"§4.2, Appendix D"},{"comment":"Table 1's header lists 'GenLen' without an arrow indicating whether lower is better, and Table 2's caption uses 'donate' where 'denote' is meant.","section":"Table 1, Table 2"},{"comment":"The Repetition Penalty baseline is attributed to Keskar et al. (2019) in §5.2 but to Holtzman et al. (2020) in §7.2; please reconcile these citations and use the correct primary source.","section":"§5.2, §7.2"},{"comment":"EGP is described both as the 'frequency with which a model successfully interrupts repetitive sequences' and as the 'proportion of end-of-sentence tokens across all samples'; these are different quantities, and the metric should be defined precisely.","section":"§5.3"},{"comment":"The λ-sweep figures in Appendix B are hard to read because each panel shows three scenarios without a legend; adding per-scenario plots or a clear legend would improve interpretability.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The formal inconsistency in §4.3 is the most serious issue: if the authors cannot provide the token-dependent implementation, the central claim collapses. The reported results suggest an implementation exists, but the paper as written does not specify it. The evaluation also needs restructuring to remove the decoding-policy confound and to address the circularity between RPG and the TR-N/TR-S metrics. I recommend major revision rather than rejection because the dataset and the underlying grammar-based idea are valuable and potentially correct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper introduces a genuinely new way of thinking about repetition in code generation: instead of identical tokens, it formalizes structural repetition as repeated nonterminal patterns in the grammar-rule reduction, and detects them with suffix/LCP arrays. That framing is useful, and the new CodeRepetEval dataset is a real contribution. Second, the formal definition of the penalty is wrong as printed. Eq. 6 defines P_n(x_t|x_<t) = λ^Count(Rep(X_{1:t})), which does not depend on x_t at all. Multiplying every candidate's score by the same positive constant cannot selectively penalize tokens that continue a repetition, so the abstract's \"strategically decays the likelihood of critical tokens\" is not what the equations do. The prose says the weight uses the frequency and recency of the token's grammar rule, but g(x_t) never appears in the formula. Either the formula is a typo and the real implementation recomputes Count(Rep) per candidate, or the method as described could only change effective temperature and could not produce the reported gains. This is load-bearing, not cosmetic.\n\nThat said, the experiments are encouraging. RPG improves Pass@1 on HumanEval and MBPP by 6-11% over greedy, and since those benchmarks are independent of the repetition metrics, that support is real. The gains on CodeRepetEval are less convincing: TR-N and TR-S are defined on G(x), the exact sequence RPG manipulates, so big improvements there are partly built in, and the main comparison mixes RPG's greedy decoding with stochastic baselines. The PDA/BPE adaptation in Appendix E is heuristic and could be noisy on partial code, but that is a separate concern from the formula issue.\n\nThe paper deserves a serious referee. The idea is solid, the dataset is useful, and the flaw in Eq. 6 is probably fixable with a precise per-candidate statement. But the authors must clarify whether the implementation matches the prose, and redo the CodeRepetEval comparison against a matched greedy baseline.","headline":"Novel framing of structural repetition in code generation and a useful new dataset, but the core penalty formula as printed cannot implement the claimed selective decay; the experiments are encouraging, so the paper deserves a serious referee despite the formal bug.","tokens_in":22103,"tokens_out":2030,"would_cite":false,"duration_ms":20624,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that penalizing repetition at the level of grammar rules—not surface text—breaks the structural loops that trap LLMs during code generation, and that doing so also improves functional correctness on standard benchmarks.","keywords":["code generation","structural repetition","grammar-based decoding","pushdown automaton","repetition penalty","large language models","decoding strategy","CodeRepetEval"],"falsifier":"Feed generated code with known parse trees through the BPE-adapted PDA and compare the resulting grammar-rule sequence, token by token, against the sequence read off the true parse tree; if the two differ at any token that RPG then penalizes for repetition, the central reduction is unreliable and the reported gains rest on a noisy detector.","tokens_in":20979,"feed_emoji":"🔁","tokens_out":7094,"duration_ms":58534,"temperature":0.7,"pith_summary":"The paper sets out to solve a failure mode of large language models that write code: the model gets stuck generating endless variations of the same code shape, such as repeated elif branches or repeated import statements, even though the exact text differs each time. It calls this 'structural repetition' and claims it is far more common than the content-level repetition studied before. The proposed fix, RPG, is a decoding-time method that maps each generated token to the grammar rule it belongs to, detects when a sequence of grammar rules starts repeating, and applies an exponential decay to the likelihood of the tokens inside that repeated pattern. The paper argues that this realigns generation back to a correct path, and reports that RPG outperforms standard sampling methods and content-repetition baselines on its new CodeRepetEval dataset as well as on HumanEval and MBPP, with relative Pass@1 gains up to 11.3%.","feed_headline":"Grammar-level penalty cuts LLM code repetition and lifts pass@1","feed_subtitle":"RPG maps tokens to grammar rules, detects repeated patterns, and decays their likelihood, improving HumanEval and MBPP scores.","key_machinery":"The central object is the pushdown automaton (PDA) reduction of a token stream into grammar-rule symbols, $\\hat{x}_t = g(x_t) = [q_t, z_t]$, where $q_t$ is the PDA state and $z_t$ the stack symbol at step $t$. This mapping lets structurally identical but textually different code collapse to the same rule sequence after merging adjacent identical symbols. Repetition is then detected on that reduced sequence with suffix arrays and LCP arrays ($O(n\\log n)$ time, $O(n)$ space), and the penalty is applied as an exponential decay $\\lambda^{\\mathrm{Count}(\\mathrm{Rep}(X_{1:t}))}$ in the token scoring function. The PDA adaptation to BPE tokenization, where one token can correspond to several grammar terminals or several tokens to one terminal, is what makes the whole pipeline usable with LLM vocabularies.","core_discovery":"The paper's central discovery is that structural repetition in LLM code generation is a grammar-level phenomenon, not just a text-level one: repeated code fragments that look different on the surface share the same sequence of context-free grammar rules. RPG exploits this by using a pushdown automaton built on the programming language's grammar to reduce each generated token to a unique grammar-rule symbol, $\\hat{x}_t = g(x_t) = [q_t, z_t]$, merging adjacent identical symbols into a reduction sequence $\\hat{R}_{1:t}$. Suffix-array and longest-common-prefix structures then locate repeated sub-sequences in $\\hat{R}_{1:t}$, and a dynamic weight $P_n(x_t \\mid x_{<t}) = \\lambda^{\\mathrm{Count}(\\mathrm{Rep}(X_{1:t}))}$ decays the scores of tokens whose grammar rules participate in the repetition. The paper claims this both stops the endless repetition and restores correct generation, and presents experiments showing consistent gains across model families and sizes, across programming languages, and on functional correctness benchmarks.","pith_inferences":["The grammar-reduction view suggests a testable extension: repetition penalties could be applied at parse-tree node level rather than token level, which might be more robust when BPE tokens split grammar terminals.","If the mechanism is as effective as reported, it implies that self-reinforcement in repetition is driven by the model's own confidence in the repeated structure; a penalty that flattens that confidence may generalize to other structured outputs like JSON, SQL, or formal proofs, where grammar rules are also available.","The paper's own data show that the probability of the repetition start token rises with each repetition, so early intervention may matter more than penalty strength; adaptive decay schedules could outperform the fixed $\\lambda=0.9$."],"forward_implications":["Decoding-time grammar penalties can be dropped into existing LLMs without retraining or fine-tuning.","Repetition metrics defined on grammar rules, such as TR-N and TR-S, give a way to measure code quality beyond exact string matching.","The approach transfers to other programming languages simply by swapping in the target language's grammar, as demonstrated with Go.","RPG improves functional correctness, not just repetition avoidance: it reports relative Pass@1 gains of 8.0% on HumanEval, 11.3% on HumanEval-ET, 6.4% on MBPP, and 10.3% on MBPP-ET over greedy sampling.","Content repetition is a special case of structural repetition, so a grammar-level detector subsumes earlier content-repetition methods."],"supporting_citations":[{"why":"Supplies the pushdown-automaton grammar reduction that RPG adapts.","marker":"(Dong et al., 2023b)"},{"why":"Defines BPE tokenization, which the PDA adaptation in Appendix E must accommodate.","marker":"(Sennrich et al., 2016)"},{"why":"Provides the HumanEval benchmark and the unbiased Pass@k estimator used for functional correctness.","marker":"(Chen et al., 2021)"},{"why":"Provides the MBPP benchmark used for functional correctness evaluation.","marker":"(Austin et al., 2021)"},{"why":"Provides the HumanEval-ET and MBPP-ET extended-test benchmarks.","marker":"(Dong et al., 2024a)"},{"why":"Defines the Repetition Penalty baseline that RPG is compared against and outperforms.","marker":"(Keskar et al., 2019)"},{"why":"Defines the Repetition Dropout training-time baseline that RPG is contrasted with.","marker":"(Li et al., 2023a)"},{"why":"CodeLlama is the base LLM used in the main experiments.","marker":"(Rozière et al., 2023)"}],"fun_headline_variants":["Grammar-aware decoding curbs repetitive code generation","RPG: penalize tokens by grammar repetition for better code","Structural repetition in code gen fixed via grammar rules","Decay critical tokens: grammar-based fix for code repetition","Turn grammar rules into a repetition penalty for LLM code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pushdown automaton's reduction of each generated token to a unique grammar rule (Eq. 3) is correct even under BPE tokenization and on partial or incomplete code; if that mapping is noisy, the repetition detector misidentifies patterns and the penalty hits the wrong tokens.","fun_headline_variants_meta":{"raw":{"variants":["Grammar-aware decoding curbs repetitive code generation","RPG: penalize tokens by grammar repetition for better code","Structural repetition in code gen fixed via grammar rules","Decay critical tokens: grammar-based fix for code repetition","Turn grammar rules into a repetition penalty for LLM code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1283,"prompt_tokens":974,"completion_tokens":309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":232}},"tokens_in":590,"tokens_out":309,"duration_ms":3630,"temperature":1.0,"reasoning_tokens":232,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:08:55.510208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed generated code with known parse trees through the BPE-adapted PDA and compare the resulting grammar-rule sequence, token by token, against the sequence read off the true parse tree; if the two differ at any token that RPG then penalizes for repetition, the central reduction is unreliable and the reported gains rest on a noisy detector.","supporting_citations":[],"review_version":1}