Pith. sign in

REVIEW 8 cited by

LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.06215 v1 pith:LKYQ57X3 submitted 2025-02-10 cs.SE cs.AIcs.CL

Xin Zhou , Martin Weyssow , Ratnadira Widyasari , Ting Zhang , Junda He , Yunbo Lyu , Jianming Chang , Beiqi Zhang
show 2 more authors
Dan Huang David Lo
This is my paper
classification cs.SEcs.AIcs.CL
keywords leakagedatabenchmarksllmsbenchmarkevaluationratiosresearch
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) are widely utilized in software engineering (SE) tasks, such as code generation and automated program repair. However, their reliance on extensive and often undisclosed pre-training datasets raises significant concerns about data leakage, where the evaluation benchmark data is unintentionally ``seen'' by LLMs during the model's construction phase. The data leakage issue could largely undermine the validity of LLM-based research and evaluations. Despite the increasing use of LLMs in the SE community, there is no comprehensive study that assesses the extent of data leakage in SE benchmarks for LLMs yet. To address this gap, this paper presents the first large-scale analysis of data leakage in 83 SE benchmarks concerning LLMs. Our results show that in general, data leakage in SE benchmarks is minimal, with average leakage ratios of only 4.8\%, 2.8\%, and 0.7\% for Python, Java, and C/C++ benchmarks, respectively. However, some benchmarks exhibit relatively higher leakage ratios, which raises concerns about their bias in evaluation. For instance, QuixBugs and BigCloneBench have leakage ratios of 100.0\% and 55.7\%, respectively. Furthermore, we observe that data leakage has a substantial impact on LLM evaluation. We also identify key causes of high data leakage, such as the direct inclusion of benchmark data in pre-training datasets and the use of coding platforms like LeetCode for benchmark construction. To address the data leakage, we introduce \textbf{LessLeak-Bench}, a new benchmark that removes leaked samples from the 83 SE benchmarks, enabling more reliable LLM evaluations in future research. Our study enhances the understanding of data leakage in SE benchmarks and provides valuable insights for future research involving LLMs in SE.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection

    cs.SE 2026-08 conditional novelty 7.0 of 10

    A repository-level benchmark with line-level entry-point, critical-operation, and trace annotations shows today's coding agents detect under a quarter of hard vulnerabilities.

  2. ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new open-source benchmark evaluates LLM-generated end-to-end ML pipelines from Kaggle competition descriptions translated into 13 languages, with 6 private tasks to limit data leakage.

  3. Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning

    cs.AI 2025-08 unverdicted novelty 6.0 of 10

    A new retinal-imaging multimodal dataset and an ophthalmology-specific reasoning model claim state-of-the-art gains of 15 to 25 percent over existing medical and general multimodal LLMs.

  4. Benchmarking LLMs for Unit Test Generation from Real-World Functions

    cs.SE 2025-08 conditional novelty 6.0 of 10

    A new decontaminated benchmark of complex Python functions shows LLMs generate far fewer correct, covering, and bug-killing unit tests than on older benchmarks.

  5. ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities

    cs.AI 2025-06 conditional novelty 6.0 of 10

    ConsistencyChecker ranks LLMs by how well they survive chains of reversible transformations, and those scores track WMT 2024 translation quality rankings (r > 0.7) without using WMT paired data.

  6. Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering

    cs.CR 2025-10 conditional novelty 5.0 of 10

    A Bayesian model that groups similar LLM test prompts into clusters gives better predictive scores than a no-clustering baseline but does not prove that it truly corrects prompt dependence.

  7. ISACL: Internal State Analyzer for Copyrighted Training Data Leakage

    cs.CL 2025-08 conditional novelty 5.0 of 10

    An MLP trained on LLM internal states predicts Rouge-L-defined literal copying leakage with high accuracy, but not paraphrase-level leakage.

  8. A Conceptual Framework for AI Capability Evaluations

    cs.AI 2025-06 conditional novelty 5.0 of 10

    A descriptive conceptual framework with seven elements (target, task, subject, inputs, instance, measurement, result analysis) for systematizing analysis of AI capability evaluations.

Pith tools