{"id":"d86c245e-5822-4898-9aa9-87c61ce510aa","arxiv_id":"2412.01526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A template-and-combinatorial method produces HumanEval_T variants, and four LLMs score lower on them than on original HumanEval, suggesting likely data leakage.","lead":"This paper proposes generating fresh versions of programming benchmark tasks by swapping words and structure, so models cannot simply memorize answers. On a small HumanEval subset, four LLMs scored lower on these fresh versions, which the authors interpret as possible data leakage in the original benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing concern: HumanEval_T changes prompt format (no example I/O, altered signatures), so the performance drop is confounded with format mismatch rather than proving data leakage.","rationale":"The paper's central claim is that the performance drop indicates likely data leakage. That inference is only valid if the HumanEval_T variants are interchangeable with the original tasks. The reader identified semantic/difficulty equivalence as the weakest assumption. I agree, and sharpen it to a specific, observable confound: the variants are not prompt-format matched because example input–output pairs are excluded and the function signature is not part of the described template variation. Code-generation LLMs are highly sensitive to such format changes, so the observed drops may be fully explained by format mismatch. The proposed control directly isolates format from leakage by preserving signature and examples while varying only the template parameters. If the control restores performance, the original leakage claim fails; if not, it is corroborated. This is the single most load-bearing concern because the entire empirical argument rests on it. The method itself is promising and the paper is honest about preliminary scope, so the appropriate verdict remains CONDITIONAL: the construction approach deserves follow-up, but the leakage evidence is not yet conclusive.","tokens_in":7734,"tokens_out":4523,"duration_ms":38453,"concrete_test":"Run a format-matched control on the same 10 HumanEval tasks: create one variant per task that keeps the exact original function signature and the same number/type of example input–output pairs, changing only the semantic descriptors specified in the template. Evaluate GPT-3.5, GPT-4o, Claude 3.5, and Llama 3.1 on this control, on the original HumanEval subset, and on HumanEval_T, using identical decoding settings and the same test suites. If the format-matched control scores within a pre-registered tolerance (e.g., ≤2 points) of the original, the HumanEval_T drop is caused by prompt-format mismatch, not leakage; if it still drops by the original 4.75–13.75 points, the leakage interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.B.1 concludes that 'the consistent performance drop across all models indicates likely data leakage' from Table I. This inference requires the HumanEval_T variants to be interchangeable with the original tasks (Section IV.B: 'semantic equivalence and comparable complexity'). That premise is unvalidated and, as constructed, doubtful. The footnote in Section IV.A reports that HumanEval's example input–output pairs were excluded; the template instantiation varies only the natural-language description (e.g., 'numbers'→'measurements'), not the function signature or the presence of examples. Most LLM code-generation prompts built from HumanEval include a signature and examples, so the variants are not format-matched to HumanEval. For models trained on HumanEval-style prompts, the original format can trigger memorized solution patterns, while the reworded, example-free variant demands de-novo reasoning; the gap then reflects format sensitivity, not memorization of the exact task. No control condition—such as original tasks reworded with identical signature/examples, or unseen tasks in original format—is run, and no paired significance test is reported despite n=10 tasks. The claimed outliers in three of four models are asserted without a described outlier test. The drop is therefore confounded and does not establish leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a benchmark-construction method in which template tasks are instantiated into multiple concrete tasks using combinatorial test design, and instantiates this method on a 10-problem subset of HumanEval to create HumanEval_T with five variants. The authors compare four LLMs on the original HumanEval subset and on the HumanEval_T variants, report consistent performance drops on the variants, and interpret these drops as evidence of likely data leakage in HumanEval. They also examine cross-variant consistency and conclude that the concrete tasks are likely interchangeable. The paper explicitly states that the results are not conclusive and identifies future work.","tokens_in":7976,"tokens_out":5006,"duration_ms":44108,"significance":"If the equivalence of the HumanEval_T variants with the original tasks were validated, the proposed method would be a practical way to build evolving benchmark variants and to probe for data contamination. The manuscript is honest in reporting preliminary results and the data are presented in a readable form. However, the central inference from performance drops to data leakage rests on an unvalidated equivalence assumption and on a confounded comparison, so the empirical contribution, as it stands, does not support the headline claim.","major_comments":[{"comment":"The evidence for data leakage is confounded with a prompt-format mismatch. The original HumanEval prompts include a function signature and example input/output pairs, while the HumanEval_T variants, per the footnote in Section IV.A, exclude the example pairs. A performance drop under this comparison can be caused by the absence of examples or by reworded descriptions that LLMs handle less well, not by memorization of the original tasks. The paper needs a control condition, such as reworded versions of the original tasks that keep the signature and examples, original-format prompts on held-out tasks, or example-free versions of the original HumanEval prompts. Without such a control, the phrase 'likely data leakage' in Section V.B.1 is not warranted.","section":"Section IV.B and Section V.B.2"},{"comment":"The load-bearing premise that HumanEval_T variants are semantically equivalent and of comparable complexity to the original tasks is asserted but not validated. Section IV.A states the goal of maintaining semantic behavioural equivalence and Section IV.B claims 'semantic equivalence and comparable complexity,' but there is no calibration, human study, difficulty metric, or control condition. If the reworded tasks are slightly harder or phrased in a way that LLMs process less effectively, the observed drops are fully explained without invoking data leakage. The authors need to demonstrate equivalence, for example through human ratings, difficulty measurements on uncontaminated models, or per-template pass-rate comparisons on a larger task set.","section":"Section V.B.1 and Table I"},{"comment":"The reported numbers are internally inconsistent and lack uncertainty quantification. The GPT-3.5 row lists AVG=76.7 and HE=80.0, but the text reports a 4.75-point drop from 80.0 to 75.25; neither the printed AVG nor the row average of V1–V5 (75.92) equals 75.25. The paper says all results are averages of five runs, but it does not report per-run or per-task data, making it impossible to verify the averages, the 'consistent drop' claim, or the outlier assertion. With n=10 tasks, paired significance tests (e.g., Wilcoxon signed-rank) and confidence intervals are needed for each model before a leakage conclusion can be drawn.","section":"Section V.B.2"},{"comment":"The consistency analysis in RQ2 does not test interchangeability. Standard deviations of whole-benchmark averages (sigma from 1.8% to 8.9%) describe variation across five 10-task variants, not whether concrete tasks generated from the same template are interchangeable. For example, a model could pass exactly the same 8 of 10 tasks in every variant and still show a small standard deviation, which would not demonstrate that the hard tasks are equivalent across variants. The authors should report per-template results across variants and use a formal equivalence criterion, such as paired comparisons per task or an item-difficulty model.","section":"Section V.B.2"}],"minor_comments":[{"comment":"Please report which 10 HumanEval tasks were randomly sampled and provide the sampling procedure or seed, since the random subset strongly influences all reported gaps.","section":"Section IV.A"},{"comment":"The footnote stating that example input/output pairs were excluded from the analysis is easy to miss; this design decision should be moved to the experimental setup and discussed as a limitation, because it is central to interpreting Table I.","section":"Section IV.A"},{"comment":"The 'AVG' column should be defined explicitly as the mean of V1–V5, and the GPT-3.5 row should be corrected to match the stated drop of 4.75 points.","section":"Table I"},{"comment":"The combinatorial test design is described only by example; please specify the parameters, value sets, and coverage criterion used for the 10 tasks, or provide them in an appendix for reproducibility.","section":"Section IV.B"},{"comment":"The boxplots would be more informative if per-task or per-variant points were overlaid, and the caption should clarify whether the boxes are computed across problems or across variants.","section":"Figure 3"},{"comment":"The text uses 'GPT-4' in the RQ2 discussion while Table I and Section V.A use 'GPT-4o'; please standardize the model name.","section":"Section V.B.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is more a preliminary proposal than a full benchmark study, and the main empirical claim is not yet supported. The combinatorial-construction idea is worth pursuing, but the current evidence is confounded by prompt-format differences and lacks statistical grounding. A revised version with a control condition, equivalence validation, and paired significance tests could make the contribution publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is sensible: build benchmarks from template tasks instantiated with combinatorial test design, so each evaluation can use fresh concrete variants and contamination of any one variant matters less. The paper describes that construction clearly and honestly — it is a pilot on 10 HumanEval tasks, 5 variants, 4 models, and the authors say the results are 'not conclusive' rather than overclaiming. Related work is cited appropriately, including combinatorial testing for LLM evaluation and dynamic benchmarks, and the incremental positioning is fair.\n\nThe soft spot is the experimental inference. The paper reads the consistent performance drop across all models as 'likely data leakage' in HumanEval. That inference depends on the variants being interchangeable with the original tasks: same difficulty, same semantics, same prompt format. The paper asserts the goal but never validates it. The footnote in Section IV.A says the HumanEval example input–output pairs were excluded from the analysis; the original HumanEval prompt includes those examples, the variants do not. That is a prompt-format confound that can explain a systematic drop without any memorization. The stress-test note also mentions altered signatures; I don't see evidence of that in the text — the template seems to vary wording and maybe types, not the function signature — but the example exclusion alone is enough. There is no control condition, no paired significance test despite n=10, no confidence intervals, and the 'outlier' claim for three of four models is asserted without a described outlier test. No artifact is released either.\n\nWhat the paper does well: the template construction method is concrete, the combinatorial design step is standard but sensibly applied, and the authors are candid about the pilot's limits. What it does not do is establish contamination.\n\nRecommendation: this deserves a serious referee — the method is legitimate and the contamination problem is real — but the expectation should be major revision. A control condition with format-matched rewrites, a difficulty calibration for the variants, and an artifact release would be needed before the leakage claim can stand. For benchmark builders and LLM evaluation folks, it is a worthwhile read as a proposal, but I would not cite it as evidence about contamination in the near term.\n\nBest,","headline":"Sensible template-based benchmark idea, but the data-leakage claim is not supported because the HumanEval_T variants differ from HumanEval in prompt format and task equivalence is never validated.","tokens_in":8447,"tokens_out":3318,"would_cite":false,"duration_ms":28149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that all four tested LLMs perform consistently worse on reworded HumanEval tasks, indicating likely data leakage in the original benchmark.","keywords":["data leakage","data contamination","HumanEval","benchmark construction","combinatorial test design","template tasks","LLM evaluation","code generation"],"falsifier":"Run the same models on template variants of tasks that were released after the models' training cutoffs and thus could not have leaked; if those variants also score lower, the gap is about wording or difficulty, while if they score the same, the original gap is consistent with memorization.","tokens_in":7565,"feed_emoji":"📉","tokens_out":4776,"duration_ms":40080,"temperature":0.7,"pith_summary":"This paper proposes a way to build LLM benchmarks that are harder to game by memorization: turn each task into a template, use combinatorial test design to generate several concrete wordings, and assemble evaluation variants from those wordings. The authors build HumanEval_T from 10 HumanEval problems and test four LLMs on five variants. Every model scores lower on the variants than on the original tasks, with drops from 4.75 to 13.75 percentage points. The paper reads the consistency of the drop as likely evidence that HumanEval has leaked into training data, and argues that template-variant benchmarks can separate real capability from memorized performance.","feed_headline":"Four LLMs score lower on reworded HumanEval, signaling data leakage","feed_subtitle":"If the gap is memorization, original scores overstate code-generation ability by up to 13.75 points.","key_machinery":"The central mechanism is the template task: a HumanEval problem generalized by replacing specific words, such as input type, threshold descriptor, and value descriptor, with variables. Combinatorial test design then systematically instantiates those variables into multiple concrete tasks, and benchmark variant assembly randomly selects one concrete task per template into each evaluation variant. This produces textually distinct but purportedly equivalent tasks, so any performance gap between original and variant benchmarks can be attributed to memorization of the original wording. That difference is what the comparison measures.","core_discovery":"The paper's central claim is stated in Section V.B.1: the consistent performance drop across all models indicates likely data leakage in the original HumanEval benchmark. Since the concrete variants were designed to preserve semantic behaviour and difficulty, the authors attribute the lower pass rates to memorized exposure to the original tasks rather than to the variants being harder. They also claim that the five concrete versions of each template task are interchangeable enough for evaluation, based on consistency analysis across variants, while acknowledging that the results are preliminary and not conclusive.","pith_inferences":["A natural extension the paper does not run is to apply the same template process to tasks that postdate the models' training cutoffs; if the original-versus-variant gap disappears there, the leakage explanation is confirmed rather than inferred.","A control using non-leaked tasks with similar wording difficulty would distinguish leakage from general sensitivity to rewording, since the current design has no such control.","The same construction method could transfer to other static benchmarks in program repair or clone detection, where contamination is equally hard to detect."],"forward_implications":["If HumanEval scores are inflated by leakage, published comparisons of code-generation LLMs on HumanEval need to be re-read as partly measuring memorization rather than capability.","A template-variant benchmark can be refreshed over time by generating new concrete tasks, allowing longitudinal model comparisons without freezing one static test.","The method gives a practical contamination check: a large gap between original and reworded variants flags possible leakage even when training data is black-box.","If all variants are truly interchangeable, smaller benchmark subsets can be reused across evaluation rounds, since new variants prevent direct memorization."],"supporting_citations":[{"why":"Supplies the HumanEval tasks, the ten-problem subset, and the pass@1 evaluation metric used in the experiments.","marker":"[6]"},{"why":"Documents that contamination occurs when benchmark data appears in an LLM's training set, motivating the need for the proposed method.","marker":"[14]"},{"why":"Demonstrates that task contamination significantly inflates model performance, supporting the leakage interpretation of the observed gap.","marker":"[15]"},{"why":"Provides the pairwise testing strategy that underlies the combinatorial test design used to generate concrete task instances.","marker":"[16]"},{"why":"Defines combinatorial test design in practice and is cited as the source of the generation principle for varied concrete tasks.","marker":"[17]"},{"why":"Extends pairwise combinatorial testing to LLM evaluation and is the technique used to generate the five concrete versions per template task.","marker":"[20]"}],"fun_headline_variants":["Reworded HumanEval exposes LLM memorization","New benchmark flags data leakage in HumanEval","Combinatorial test design counters benchmark contamination","HumanEval variants suggest training data leakage","Template tasks unmask LLM test memorization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reworded concrete tasks are semantically and difficulty-equivalent to the original HumanEval tasks, so a lower score can only be explained by memorization of the original wording.","fun_headline_variants_meta":{"raw":{"variants":["Reworded HumanEval exposes LLM memorization","New benchmark flags data leakage in HumanEval","Combinatorial test design counters benchmark contamination","HumanEval variants suggest training data leakage","Template tasks unmask LLM test memorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1888,"prompt_tokens":912,"completion_tokens":976,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":905}},"tokens_in":528,"tokens_out":976,"duration_ms":6675,"temperature":1.0,"reasoning_tokens":905,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:09.625071+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same models on template variants of tasks that were released after the models' training cutoffs and thus could not have leaked; if those variants also score lower, the gap is about wording or difficulty, while if they score the same, the original gap is consistent with memorization.","supporting_citations":[{"cited_title":"NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark,","cited_arxiv_id":null,"evidence_quote":"Documents that contamination occurs when benchmark data appears in an LLM's training set, motivating the need for the proposed method."},{"cited_title":"Task Contamination: Language Models May Not Be Few-Shot Anymore,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that task contamination significantly inflates model performance, supporting the leakage interpretation of the observed gap."},{"cited_title":"A test generation strategy for pairwise testing,","cited_arxiv_id":null,"evidence_quote":"Provides the pairwise testing strategy that underlies the combinatorial test design used to generate concrete task instances."},{"cited_title":"Combinatorial test design in practice,","cited_arxiv_id":null,"evidence_quote":"Defines combinatorial test design in practice and is cited as the source of the generation principle for varied concrete tasks."},{"cited_title":"Applying pairwise combinatorial testing to large language model testing,","cited_arxiv_id":null,"evidence_quote":"Extends pairwise combinatorial testing to LLM evaluation and is the technique used to generate the five concrete versions per template task."}],"review_version":1}