{"id":"ed170ee8-84eb-421d-b39e-a45f30ed2e5e","arxiv_id":"2505.15039","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-tuned CodeT5 model translates competitive programming input specifications into Context-Free Grammars with Counters, which generate test cases that outperform direct LLM-generated tests on CodeContests.","lead":"This paper presents a system that turns natural language problem descriptions into formal grammars with counters, then uses those grammars to generate test inputs for competitive programming problems. It reports that these grammar-generated tests catch more wrong solutions than tests written directly by ChatGPT or Gemini.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth CCFG labels serve as both training targets and evaluation oracle; the check that they parse all CodeContests test cases does not rule out over-restrictive grammars, which would systematically penalize direct LLM baselines.","rationale":"The reader's weakest assumption is exactly the correctness and completeness of the manually created ground-truth CCFGs. I agree, and I have made the concern more concrete by articulating the mechanism that over-restrictive labels would penalize direct LLM baselines more than CCFGT5, because CCFGT5 is trained on the same labeling style and generates test cases from its generated grammars, which are therefore inside the ground-truth language by construction. The paper's existing validation step (parsing all existing CodeContests test cases) only checks one direction of correctness (accepting known valid inputs), not completeness or rejection of invalid inputs, so it does not address this concern. The proposed independent-annotation test would directly measure the objectivity of the labels and their coverage of the valid input space. Since the reader already assigned CONDITIONAL and this concern is the same one, I recommend leaving the verdict unchanged rather than escalating, because the empirical comparison might survive independent ground-truth validation. The lack of statistical testing and the exclusion of 29 problems are also relevant, but they are secondary to the ground-truth issue: even with perfect statistics, a biased oracle would invalidate the comparison.","tokens_in":11029,"tokens_out":6973,"duration_ms":59153,"concrete_test":"Select a random sample of 30 evaluation problems. Have two independent annotators, unfamiliar with the paper's grammar conventions, write CCFGs directly from the problem statements. For each problem, generate 100 test cases from each of the three grammars (the authors' ground truth and the two independent ones) and compute pairwise semantic equivalence using the paper's element-based validity and generality metrics (Section 4.4). If the authors' ground-truth grammar fails to parse more than 5% of the test cases generated by either independent grammar, the ground truth is over-restrictive and the reported comparison against direct LLM baselines is biased. In addition, hand-craft boundary test cases (e.g., n=1, n=10^5) from the problem statements and check whether the ground-truth grammar parses them; failures at expected valid boundaries provide direct evidence of incompleteness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CCFGT5 10 beats direct LLM generation (Table 2: 81.18% set-based validity, 67.73% set-based effectiveness vs. 78.97% and 63.54% for ChatGPT direct) is computed entirely against the authors' manually written ground-truth CCFGs (Section 4.1). The paper's only validation of these ground truths is that each grammar parses all public and private CodeContests test cases (Section 4.5). This is a necessary but insufficient check: a grammar can be over-restrictive—omitting valid inputs—and still pass, because CodeContests test cases are a sparse sample of the input space. Since CCFGT5 is fine-tuned on the same authors' grammar style (Section 3.2), its outputs are likely to be inside the ground-truth language, whereas direct LLM generation may produce valid but 'unconventional' test cases that an over-restrictive ground-truth grammar rejects, artificially depressing the baselines' validity and effectiveness. For example, if a ground-truth grammar ties a counter to the wrong minimum bound, boundary test cases from ChatGPT would be scored invalid while CCFG-generated tests, drawn from that grammar, would never hit the omitted region. The paper's Observation 1 (CCFGs most effective on hard specs) could partly reflect that complex specifications are harder to annotate correctly, increasing the chance of over-restrictive labels. This mechanism directly undermines the claimed superiority of the CCFG approach, so the central result is not firmly established until the ground-truth grammars are independently validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Context-Free Grammars with Counters (CCFGs), an extension of CFGs intended to capture counting and repetition constraints in competitive programming input specifications. It fine-tunes CodeT5 in two modules (grammar and constraints) to translate natural-language specifications into CCFGs, then samples test cases from the resulting grammars. On the CodeContests dataset, with ground-truth CCFGs manually written by the authors for 1,500 problems (1,200 training, 300 evaluation, reduced to 271 after filtering), the paper reports that CCFGT5 with beam size 10 achieves the highest set-based validity and set-based effectiveness among the compared methods, including direct ChatGPT generation: 81.18% vs 78.97% set-based validity and 67.73% vs 63.54% set-based effectiveness. The paper also reports observations about specification difficulty, test-case length, and grammar generality.","tokens_in":11335,"tokens_out":10656,"duration_ms":90118,"significance":"If the evaluation is trustworthy, the paper makes a useful contribution to automated test case generation: it provides a grammar-driven framework that avoids per-problem manual grammar writing at inference time, and the released code and data support reproduction and further study. The CCFG formalism, once precisely defined, could also inform grammar-based fuzzing and benchmark construction. However, the headline numerical gains are modest, and the evaluation's validity oracle is the authors' own manual CCFG annotations, so the central comparative claim is not yet firmly established. The paper is transparent about some failure cases in Section 4.6, which is a positive feature.","major_comments":[{"comment":"The paper never gives a formal definition of CCFG. Example 3 uses subscripted nonterminals such as T_i and T_{i-1} together with informal counter operations ('set the value of an internal counter to t', 'decrement the counter by 1'), but no formal syntax or semantics is provided. Because CCFG is the central contribution, and because the paper claims it is substantially different from the grammar model of Chistikov et al. [2018], the authors need to define the formalism precisely: how counters are declared, initialized, decremented, and how acceptance of a string is characterized. Without this, the implementation, the undecidability remark in Section 4.4, and the reproducibility of the evaluation cannot be assessed.","section":"Section 3.1"},{"comment":"All validity, generality, and equality metrics in Tables 2 and 5 are computed against the authors' manually written ground-truth CCFGs. The only validation reported for these labels is that each grammar parses all public and private CodeContests test cases (Section 4.5). This is a necessary but insufficient check: an over-restrictive grammar that omits valid inputs can still pass because CodeContests test cases are a sparse sample of the input space. Since CCFGT5 is fine-tuned on the same annotation style, its outputs are likely to match the labels, whereas direct LLM generation may produce valid but unconventional test cases that are marked invalid. This bias directly affects the headline comparison in Table 2 (81.18% vs 78.97% set-based validity; 67.73% vs 63.54% set-based effectiveness). The authors should mitigate this concern by, for example, having independent annotators re-label a sample of problems, manually auditing test cases rejected by the ground-truth grammars, or reporting how the headline gaps change under plausible label perturbations.","section":"Sections 4.1, 4.4, 4.5"},{"comment":"The paper reports no variance or significance information. Set-based validity is a per-problem binary outcome and set-based effectiveness is a per-problem proportion, so the observed differences of 2.21 and 4.19 percentage points over roughly 270 problems may be within sampling noise. In addition, the rows of Table 2 report different 'Well-defined(#)' counts (e.g., 271 for direct ChatGPT vs 264 for CCFGT5 10) without explaining how these counts enter the denominators of the reported percentages, which prevents a reader from reconstructing the paired per-problem comparison. Please provide a contingency table, confidence intervals, and a paired statistical test such as a Wilcoxon signed-rank test or a bootstrap over problems.","section":"Section 4.5, Table 2"},{"comment":"The claim that CCFGs are especially effective for complex input specifications rests on the Hard category, which contains only 11 evaluation problems; CCFGT5 10's reported 90.91% hard set-based validity corresponds to 10 of 11 problems. This sample is too small to support the qualitative conclusion, and it is also the category where annotation errors are most plausible. The authors should either collect substantially more hard problems or soften Observation 1 accordingly.","section":"Section 4.5, Table 3, Observation 1"},{"comment":"The evaluation excludes 29 of the 300 evaluation problems: 6 because CodeContests has no incorrect solutions for them and 23 because the authors' CCFG implementation cannot process their human-labeled grammars. If the excluded problems differ systematically from the included ones, the results in Table 2 may not generalize. The authors should report the difficulty distribution and specification features of the excluded problems and show that the main comparisons are stable under different inclusion criteria.","section":"Section 4.1"}],"minor_comments":[{"comment":"In the definition of set-based generality, the phrase 'if and only if the test case-based generality is also 1' should presumably read 'element-based generality'; the terminology is inconsistent.","section":"Section 4.4"},{"comment":"The table is hard to read because numbers are run together in the extracted text (e.g., '26482.3281.18'); use clear column separation and consistent decimal formatting.","section":"Table 2"},{"comment":"The paper does not report the exact prompt templates, the number of chain-of-thought examples, or the sampling parameters used for ChatGPT and Gemini, which are needed for reproducing the direct-generation baselines.","section":"Section 4.3"},{"comment":"The text says the authors select 'a pair of grammar and constraints among top-k grammars and constraints' but the value of k is never specified, making the CCFGT5 inference procedure underspecified.","section":"Section 3.2"},{"comment":"Element-based effectiveness is defined to be 0 whenever the test-case set is invalid, so it is not actually an average over valid individual test cases; the authors should clarify why this coupling was chosen.","section":"Section 4.4"},{"comment":"The sentence 'We will expand CCFGs to handle broader input domains and optimizing sampling strategies' is ungrammatical; 'and optimizing' should likely be 'and optimize'.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The main risk in this paper is benchmark circularity rather than technical fraud: the ground-truth grammars are authored by the same group that builds and tunes the model, and the evaluation oracle is exactly those grammars. The stress-test concern in the reader's report lands, and it should be the central request in the revision. The paper is otherwise within scope for cs.SE and the authors have provided code and data, which is a strength. If the authors can add an independent validity check and proper statistical comparison, the paper could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper proposes a genuinely new way to generate test cases for competitive programming—translate the natural language input specification into a context-free grammar with counters (CCFG) using a fine-tuned CodeT5, then sample test cases from the grammar. The reported gains over direct LLM generation are real but modest: 81.18% set-based validity and 67.73% set-based effectiveness for CCFGT5-10 versus 78.97% and 63.54% for ChatGPT direct. The strength is the idea and the resource: they hand-annotated 1,500 CodeContests problems with CCFGs, which is a useful dataset even if you don't buy the main result.\n\nWhat I liked: The CCFG formalism is a nice fit for competitive programming input specs—counters tied to input values capture structure like \"the next t lines each contain...\" that plain CFGs miss. The distinction from Chistikov et al.'s counter grammars is meaningful. The evaluation includes a breakdown by specification difficulty and test-case length, and they report both element- and set-based metrics. They also ship code and data. That's all to the good.\n\nWhere I'm skeptical: The ground-truth grammars are the authors' own hand-written labels, and those same labels are the oracle for validity and generality. The stress-test note gets this right: a grammar can be over-restrictive—missing valid inputs—and still parse all public/private CodeContests tests, because those tests are a sparse sample. Since CCFGT5 is fine-tuned on the same grammar style, its outputs are likely to sit inside the ground-truth language, while direct LLM generation might produce valid but \"unconventional\" test cases that a restrictive grammar rejects. That would artificially depress the baselines. The paper's check that each ground-truth grammar parses all official tests is necessary but not sufficient. I don't think this is fatal—the gains are small enough that the main claim might still hold—but it means the central comparison is not firmly established.\n\nOther soft spots: no variance or statistical testing across the 271 problems; they exclude 29 problems (6 with no incorrect algorithms, 23 their implementation can't handle), which could bias results; and the CCFG semantics are described informally, with examples doing the work. The case-study failures in Section 4.6 are honest and show the model's limits.\n\nWho this is for: anyone working on automated test generation, grammar-based fuzzing, or using LLMs for formal-spec translation. It deserves a serious referee. The right revision would add independent validation of a sample of ground-truth grammars (e.g., by a second annotator or by checking that the grammars generate a diverse set of valid inputs beyond the official tests), report confidence intervals, and tighten the formalism. I'd accept it for review with the expectation of a major revision.","headline":"A genuinely new method that likely works, but the evaluation leans on the authors' own grammars as ground truth, so the headline numbers are plausible yet not firmly established.","tokens_in":11914,"tokens_out":2596,"would_cite":true,"duration_ms":22199,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that translating competitive-programming input specifications into Context-Free Grammars with Counters, using a fine-tuned CodeT5 model, produces test cases that are more often fully valid as sets and catch more…","keywords":["test case generation","context-free grammars with counters","competitive programming","CodeT5","grammar-based fuzzing","input specification translation","CodeContests","LLM test generation"],"falsifier":"Take roughly 30 CodeContests problems where CCFGT5-10's grammar is scored as semantically equivalent to the ground truth, sample the ten generated test cases per problem, and have independent human annotators check each input against the original problem statement; if any sampled case violates a stated numerical constraint, such as an array longer than the declared bound, or an implied structural rule, the validity and effectiveness numbers are inflated.","tokens_in":1865,"feed_emoji":"🧪","tokens_out":2032,"duration_ms":83051,"temperature":0.7,"pith_summary":"This paper introduces a grammar-first pipeline for automated test-case generation in competitive programming. Instead of asking a large language model to write test cases directly, the authors fine-tune a CodeT5 model to translate each problem's natural-language input specification into a Context-Free Grammar with Counters (CCFG), and then sample test cases from that grammar. They claim this approach produces test suites whose entire set is valid more often and that expose incorrect algorithms more effectively than direct ChatGPT or Gemini generation: on the CodeContests evaluation set, CCFGT5-10 reaches 81.18% set-based validity and 67.73% set-based effectiveness, compared with 78.97% and 63.54% for direct ChatGPT generation. The practical stake is that a correct grammar, once learned, can generate any number of specification-compliant tests for free, without per-case validation or further model calls.","feed_headline":"Grammar-first test generation beats direct LLM prompts","feed_subtitle":"Test sets built from counted grammars hit 67.73% effectiveness on CodeContests, ahead of the 63.54% from direct LLM prompts.","key_machinery":"The central object is the Context-Free Grammar with Counters (CCFG), a context-free grammar in which some nonterminals carry an index that acts as a counter, for example $T_i$ with $T_t$ expanded $t$ times before reaching $T_1$; the counter is set by an integer already read from the input and decremented at each application. This lets the grammar enforce that a variable's value determines the number of following lines, array elements, or queries, so both syntax and the semantic constraints that depend on earlier values live inside the grammar. The translation is carried out by CCFGT5, two fine-tuned CodeT5 modules, one for grammar structure and one for numerical constraints, plus a CCFG tokenizer that labels symbols with descriptors such as 'variable' and 'nonterminal.' The machinery's job is to reduce test-case generation to grammar sampling: once the grammar is faithful, every generated test case is valid, and varying the sampling interval for the counter-driven variables produces short, medium, and long cases that catch different incorrect algorithms.","core_discovery":"The central claim is that CCFG-based test cases with the best CCFGT5 configuration exhibit the highest set-based validity and both types of effectiveness, achieving 81.18%, 42.26%, and 67.73% for set-based validity, element-based effectiveness, and set-based effectiveness, respectively. Ground-truth grammars, which the authors use as an upper bound, reach 100%, 52.51%, and 83.40% on the same metrics. The paper argues that the mechanism is a formal grammar with counters: the grammar records that an integer read earlier, such as $t$ or $n$, controls how many times a subsequent block repeats, so generation is valid by construction rather than by checking. On normal and hard specifications, CCFGT5-10's set-based validity remains 76.24% and 90.91%, while direct ChatGPT falls to 65.35% and 63.64%, which the authors cite as evidence that the formalism pays off precisely where specifications are complex.","pith_inferences":["A generated grammar that passes the paper's filters, parsing all public and private CodeContests tests for its problem and resembling training grammars syntactically, could be trusted to generate tests without a human-written ground truth; this suggests a fully automatic validation loop using only existing tests, though the paper itself keeps ground-truth grammars as labels.","The counter formalism and the translation model could transfer to other specification-driven generation tasks, such as synthesizing input data for program repair or property-based testing, because the CCFG is an executable model of the spec; whether transfer holds across domains is untested.","The failure analysis implies that normalizing constraint language, for instance rewriting 'the number of digits does not exceed 10^5' into $n \\le 10^5$, and standardizing variable naming between grammar and constraint modules would likely close most of the gap between CCFGT5-10 and ground-truth grammars.","The comparison of set-based versus element-based validity suggests a cheap quality signal: when a generated grammar is valid on every sampled test case but its set-based validity is lower than its element-based validity, the grammar likely misses a rare branching condition, so validity alone should not be used to certify completeness."],"forward_implications":["Once a CCFG is correct, no per-test-case validation is needed; validity is a property of the grammar, so arbitrary numbers of valid test cases can be sampled cheaply.","The approach's advantage concentrates where specifications are complex: on hard specifications CCFGT5-10 reaches 90.91% set-based validity, while direct ChatGPT generation reaches 63.64%.","Mixed-length suites are the most effective setting: the union of short and long test cases outperforms either length alone, and the paper's 4-short/3-medium/3-long mix gives the best 10-case set-based effectiveness.","Set-based generality tracks set-based validity closely for CCFGT5-10, meaning the generated grammars cover nearly the same test-case space as the ground truth when they are valid.","Because grammar generation is a one-time cost, scaling to 30 test cases is straightforward and improves set-based effectiveness, unlike direct generation whose cost grows with each additional LLM prompt."],"supporting_citations":[{"why":"Supplies the CodeContests benchmark: the problems, the correct and incorrect algorithms, and the public/private test cases that ground every reported evaluation.","marker":"[Li et al., 2022]"},{"why":"Provides CodeT5, the encoder-decoder model that CCFGT5 fine-tunes into separate grammar and constraint translators.","marker":"[Wang et al., 2021]"},{"why":"Introduces grammar prompting, the idea of guiding generation with a formal grammar that motivates translating descriptions into CCFGs.","marker":"[Wang et al., 2023]"},{"why":"Defines the prior counter-grammar formalism whose counters this paper repurposes and distinguishes from its own pre-bound rules.","marker":"[Chistikov et al., 2018]"},{"why":"Supplies the undecidability result that forces the paper to measure CCFG validity empirically by parsing against ground-truth grammars.","marker":"[Hopcroft et al., 2007]"},{"why":"Provides the Chain-of-Thought prompting style used to build the direct ChatGPT and Gemini test-generation baselines.","marker":"[Wei et al., 2022]"},{"why":"Describes MuTAP, a mutation-based LLM test-generation method whose mutation idea the paper's fuzzing baseline adapts.","marker":"[Dakhel et al., 2024]"},{"why":"Presents CodaMosa, the LLM-with-mutation test generator whose coverage-only objective the paper contrasts with fault-detection effectiveness.","marker":"[Lemieux et al., 2023]"}],"fun_headline_variants":["Counted grammars beat LLM prompts for test generation","Grammar-crafted tests find more bugs than LLM prompts","Formal grammars with counters outperform direct LLM prompts","CCFG test cases top LLM prompts on CodeContests"],"cache_read_input_tokens":13952,"weakest_assumption_plain":"The evaluation assumes the 1,500 manually written CCFGs are correct and complete formalizations of the intended input specifications, kept only when they parse all public and private CodeContests test cases, since every validity, generality, and effectiveness score compares generated grammars against these author-created labels.","fun_headline_variants_meta":{"raw":{"variants":["Counted grammars beat LLM prompts for test generation","Grammar-crafted tests find more bugs than LLM prompts","Formal grammars with counters outperform direct LLM prompts","CCFG test cases top LLM prompts on CodeContests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1682,"prompt_tokens":896,"completion_tokens":786,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":718}},"tokens_in":512,"tokens_out":786,"duration_ms":6544,"temperature":1.0,"reasoning_tokens":718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:24:29.161061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take roughly 30 CodeContests problems where CCFGT5-10's grammar is scored as semantically equivalent to the ground truth, sample the ten generated test cases per problem, and have independent human annotators check each input against the original problem statement; if any sampled case violates a stated numerical constraint, such as an array longer than the declared bound, or an implied structural rule, the validity and effectiveness numbers are inflated.","supporting_citations":[{"cited_title":"Context-free commutative grammars with integer counters and resets","cited_arxiv_id":null,"evidence_quote":"Defines the prior counter-grammar formalism whose counters this paper repurposes and distinguishes from its own pre-bound rules."},{"cited_title":"Desmarais","cited_arxiv_id":null,"evidence_quote":"Describes MuTAP, a mutation-based LLM test-generation method whose mutation idea the paper's fuzzing baseline adapts."},{"cited_title":"Lahiri, and Siddhartha Sen","cited_arxiv_id":null,"evidence_quote":"Presents CodaMosa, the LLM-with-mutation test generator whose coverage-only objective the paper contrasts with fault-detection effectiveness."}],"review_version":1}