Pith. sign in

REVIEW 3 major objections 5 minor 61 references

OSS-Bench: Benchmark Generator for Coding LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A benchmark generator, OSS-Bench, replaces functions in real open-source projects with LLM output and scores the result on compilability, tests, and memory safety.

desk verdict Useful new live benchmark for coding LLMs; the 10x memory-safety claim outruns the evidence. read the letter →

arxiv 2505.12331 v2 pith:654EYS7E submitted 2025-05-18 cs.SE cs.LG

Yuancheng Jiang , Roland Yap , Zhenkai Liang This is my paper · ORCID
classification cs.SEcs.LG
keywords benchmarkgeneratorcodeLLMevaluationmemorysafetysanitizeralertsopen-sourcesoftwarefunction-leveleditinglivecompilability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OSS-Bench is a method for building coding-LLM benchmarks automatically from real open-source projects. The paper claims that by deleting functions from large C codebases such as the PHP interpreter and SQLite3, asking an LLM to improve each function, and then rebuilding and testing the project, you get three natural ground-truth signals without human-written test cases or LLM-generated oracles: does it compile, do the existing tests still pass, and do sanitizers complain. The paper's central result is that this pipeline ranks 17 LLMs in a way that exposes behavior that simpler benchmarks miss: model size does not predict performance, memorizing OSS code can lead to counterproductive edits, and even the best models' edits introduce roughly ten times more memory-safety violations than the original code under fuzzing. A sympathetic reader would take away that evaluation of coding assistants should be anchored in real, evolving, low-level code rather than static hand-made tasks.

What carries the argument

The load-bearing mechanism is the replacement-and-rebuild loop: extract a function, swap in the LLM's output, recompile the whole project, run the project's own test suite, and collect sanitizer alerts from compilation and test logs. The scoring chain makes the dependencies explicit: $c_1=s_1$, $c_2=s_2\times c_1/100$, and $c_3=s_3\times c_2/100$, where $s_1$, $s_2$, and $s_3$ are raw compilability, test, and sanitizer scores, so a model can only earn later-metric credit on code that passed earlier stages. A dissimilarity bonus $d$, computed with difflib as the average number of changed lines per function, is added to discourage conservative no-edit responses.

What would settle it

A control experiment in which each LLM's output is replaced by the original function verbatim, running the same compile, test, and fuzzing pipeline on that null edit: if the null-edit condition produces sanitizer alert counts comparable to those attributed to LLM edits, then the memory-safety metric is not measuring the edits themselves.

Watch

Extended reading notes

Core claim

The central claim is that OSS-Bench can profile LLM coding ability by measuring the degradation caused when an LLM's rewrite of a single function is dropped into a real, unmodified project. Using libclang, it extracts 10,534 functions from the PHP interpreter and 7,321 from SQLite3; it prompts each model to optimize a snippet while preserving memory safety; then it scores the results with three chained metrics: compilability, the average test-pass rate over 1,000 sampled replacement batches, and the number of unique sanitizer alerts. The paper reports that closed instruction-tuned models lead, that Qwen3 models beat larger Qwen2.5-Coder models despite smaller size, and that extended fuzzing of 500 test iterations finds 336 unique sanitizer bugs for GPT-O1 and 736 for Qwen3-30B-A3B against 56 for unmodified PHP, supporting the claim that LLM edits introduce about ten times more memory-safety violations.

Load-bearing premise

The benchmark assumes that sanitizer alerts and test failures appearing after a function replacement are caused by the LLM's edit, rather than by pre-existing bugs in the unmodified project or by interactions with unmodified components.

Editorial extensions

If this is right

  • Benchmarks built this way can be refreshed automatically whenever the upstream project changes, so leaderboard tasks no longer need to be frozen.
  • A model that passes compilation and tests can still degrade memory safety, so test-pass rate alone is not a sufficient safety signal; the sanitizer layer is necessary.
  • The chained scoring means a model with a low compilability score is capped in functional and memory-safety scores, so the overall ranking rewards edits that survive all downstream stages.
  • Model size is not a reliable predictor on these tasks; architecture and training matter more, as shown by Qwen3-8B outperforming the larger Qwen2.5-Coder-32B.
  • Memorized patterns from training data can actively cause compile errors when applied in a slightly different context, as seen in the variadic-function case where top models injected an extra argument.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If OSS-Bench were extended to per-function differential attribution of sanitizer alerts, the 'ten times more violations' claim would become a statement about individual edits rather than whole-batch replacement, making the ranking more interpretable.
  • A live benchmark introduces version drift: scores will change as the underlying OSS evolves, so comparable leaderboard standings require pinning commits and re-running baselines on the same commit.
  • The same replacement-and-rebuild loop could be turned into a CI regression check for LLM code suggestions, flagging edits that compile and pass tests but introduce sanitizer alerts.
  • The dissimilarity bonus punishes conservative edits, but no metric directly verifies the prompt's second goal of improving efficiency; an oracle that checks whether the edit actually speeds up the code would sharpen the ranking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents OSS-Bench, an automated benchmark generator that extracts functions from open-source projects (PHP and SQLite3), prompts LLMs to produce improved versions of those functions, and scores the results with three metrics: compilability, functional test pass rate, and memory-safety via sanitizer alerts, plus a dissimilarity bonus intended to discourage trivial edits. The evaluation profiles 17 LLMs, reports rankings on OSS-Bench(php) and OSS-Bench(sql), and presents an extended fuzzing experiment that claims LLM-generated edits introduce roughly ten times more memory-safety violations than the original code.

Significance. If the claims are supported, OSS-Bench is a valuable contribution: it uses external, natural ground truth (compiler results, official OSS test suites, and sanitizer tools), avoids LLM-generated oracles, and provides a live, continuously updatable benchmark that mitigates static-dataset overfitting. The contamination analysis and the focus on low-level memory safety are also useful and relatively novel for coding-LLM benchmarks. The main caveat is that the headline fuzzing result and several ranking conclusions currently depend on aggregate alert counts and hand-set scoring parameters, so the quantitative claims need stronger support.

major comments (3)
  1. [Section 3.4 and Section 2 (Metric III)] The central claim that LLM edits introduce roughly ten times more memory-safety violations is not supported by the reported data. The fuzzing table compares aggregated deduplicated sanitizer-alert counts from 500 iterations over modified PHP (336 for GPT-O1, 736 for Qwen3-A3B-30B) to a single baseline count for unmodified PHP (56), without stack-trace-level matching or baseline subtraction. Because each iteration replaces about 100 functions at once, alerts in modified runs can originate from any replaced function, from interactions between replacements, or from unmodified code whose paths become reachable only after an edit; the baseline itself already exhibits 56 sanitizer bugs. The paper should provide per-alert attribution, such as stack-trace matching against the pristine baseline under identical seeds and inputs, or report baseline-subtracted counts before asserting the 10x ratio.
  2. [Section 2 (Scoring) and Section 3.2 (Table 1)] The final scores and rankings depend on hand-set parameters: 0.69 penalty points per sanitizer alert for PHP, 0.19 for SQLite, a 0.5 factor for memory-leak alerts, a dissimilarity weight w_d = 0.1, and an unspecified minimum dissimilarity threshold. No sensitivity analysis is reported, and these parameters materially affect the memory-safety subscore and thus the final ranking. The paper should demonstrate that model rankings are robust to reasonable variations of these parameters, or provide a principled justification for the chosen values.
  3. [Section 3.4 (fuzzing table)] The fuzzing table aggregates the 'Undefined' category (e.g., integer overflows) together with SEGV, spatial, and temporal memory-safety errors, but undefined behavior is not necessarily a memory-safety violation. Excluding the 'Undefined' category changes the GPT-O1 total from 336 to 173 and the Qwen3-A3B-30B total from 736 to 386, substantially reducing the reported ratio. The categories should be reported separately, and the wording should distinguish memory-safety bugs from undefined behavior in general.
minor comments (5)
  1. [Section 3.2, Table 1] The 'Delta (10%)' column appears to report the weighted bonus w_d * d rather than the raw dissimilarity d defined in Section 2; the notation should be clarified to avoid confusion with the scoring formula.
  2. [Section 3.1] There is a typo in the model list: 'Claude-3.7-Sonnect' should be 'Claude-3.7-Sonnet'. Model names such as 'Qwen3-A3B-30B-Instruct' and 'Qwen3-A3B' are also used inconsistently.
  3. [Section 3.4] The sentence 'even the original PHP code shows exhibits some safety bugs' contains a grammatical error and should be rewritten.
  4. [References] Reference [16] for CodeArena points to a 2019 IEEE TechDebt paper about Minecraft, which does not match the description 'adaptive scoring based on collective model performance' in Section 5; this citation appears to be incorrect.
  5. [Section 2 (Metric II) and Section 3.2] The use of 'pass@k' terminology with k=1 is unusual because the evaluation appears to record a single generated completion per function rather than sampling k completions; please clarify the relationship to the standard pass@k estimator.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OSS-Bench's metrics rest on external compiler, test-suite, and sanitizer signals, and no result is defined in terms of its own inputs.

full rationale

The derivation chain is self-contained. Metric I (Section 2) scores whether a replacement function compiles in the real project, using the compiler itself as ground truth; Metric II runs the project's existing official test suite and compares average pass rates against a measured unmodified baseline; Metric III counts sanitizer alerts from toolchain logs. None of these signals is defined by the LLM output being scored, and the raw scores s1, s2, and s3 are aggregate measurements rather than fitted predictions. The chained-score formula is an explicit weighting scheme, not an equivalence that manufactures its own evidence. The only same-author reference is FlowFusion [61], used in Section 3.4 as the fuzzing instrument; it is applied uniformly to the pristine PHP baseline and to modified runs, and it reports 56 bugs in the baseline, so the LLM-versus-baseline comparison is not forced by the citation. The concern that alerts in modified runs may be pre-existing or caused outside the replaced function is a validity and attribution threat to the 'roughly ten times more violations' quantitative claim, not a circularity: the claim could be incorrect without being definitionally identical to its inputs. No equation in the paper reduces a predicted quantity to a fitted parameter, to a self-citation chain, or to the benchmark's own scoring choices.

Assumptions & free parameters 8 free parameters · 3 assumptions · 0 invented entities

All scoring constants (sanitizer penalties, dissimilarity weight and threshold, sampling fractions) are hand-selected and are not derived from data or an optimization criterion; the benchmark's rankings are therefore conditional on these choices. The foundational premises are that OSS tests and sanitizer alarms are attributable to LLM edits, and that difflib line counts capture edit substance.

free parameters (8)
  • sanitizer_alert_penalty_php = 0.69 points per alert
    Hand-set constant for Metric III scoring in OSS-Bench php; affects the raw sanitizer score and the final chained score.
  • sanitizer_alert_penalty_sqlite = 0.19 points per alert
    Hand-set constant for Metric III scoring in OSS-Bench sql; affects the final ranking.
  • memory_leak_penalty_factor = 0.5 of standard deduction
    Memory-leak alerts incur half the standard deduction; the factor is chosen without stated justification.
  • dissimilarity_weight = w_d = 0.1
    Weight of the Delta bonus in the final score; chosen by default and not sensitivity-tested.
  • minimum_dissimilarity_threshold = unspecified
    The threshold below which models are excluded from evaluation is referenced in Section 2 but never given a numeric value.
  • function_token_filter = 10 to 256 tokens
    Functions outside this range are discarded; this changes the task set and the difficulty profile.
  • test_iterations = 1000
    Number of random sampling iterations for Metric II; chosen by the authors.
  • sampling_fraction = ~1% (100 functions for PHP, 73 for SQLite)
    Number of functions replaced per test iteration; affects the measured pass-rate degradation.
assumptions (3)
  • domain assumption The OSS test suites of PHP and SQLite are comprehensive and deterministic enough that test pass-rate degradation is a valid proxy for functional correctness.
    Metric II assumes test suites cover behavior affected by edited functions; coverage is not measured in the paper but inferred from project maturity.
  • domain assumption Sanitizer alerts detected in the modified project's compilation and test logs are attributable to the LLM-edited functions.
    Metric III aggregates alerts without differential analysis; the fuzzing section shows the pristine project already contains bugs, so attribution is not established.
  • ad hoc to paper Line-level differences computed by difflib are a valid measure of edit substantiveness for the Delta metric.
    The dissimilarity bonus assumes more changed lines means more substantive improvement, which is not established and is easily gamed by cosmetic edits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OSS-Bench: Benchmark Generator for Coding LLMs." pith.science (2026). https://pith.science/paper/654EYS7E

@misc{pith2026250512331,
  author       = {Pith},
  title        = {Pith review of: OSS-Bench: Benchmark Generator for Coding LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/654EYS7E}},
  note         = {Machine review of arXiv:2505.12331}
}
read the original abstract

In light of the rapid adoption of AI coding assistants, LLM-assisted development has become increasingly prevalent, creating an urgent need for robust evaluation of generated code quality. Existing benchmarks often require extensive manual effort to create static datasets, rely on indirect or insufficiently challenging tasks, depend on non-scalable ground truth, or neglect critical low-level security evaluations, particularly memory-safety issues. In this work, we introduce OSS-Bench, a benchmark generator that automatically constructs large-scale, live evaluation tasks from real-world open-source software. OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth. In our evaluation, the benchmark, instantiated as OSS-Bench(php) and OSS-Bench(sql), profiles 17 diverse LLMs, revealing insights such as intra-family behavioral patterns and inconsistencies between model size and performance. Our results demonstrate that OSS-Bench mitigates overfitting by leveraging the evolving complexity of OSS and highlights LLMs' limited understanding of low-level code security via extended fuzzing experiments. Overall, OSS-Bench offers a practical and scalable framework for benchmarking the real-world coding capabilities of LLMs.

Figures

Figures reproduced from arXiv: 2505.12331 by the authors.

Figure 1
Figure 1. Benchmark Workflow in OSS-BENCH - Large codebase. A large codebase increases the inherent complexity of evaluation tasks on realistic development scenarios. Although the LLM edits a small code snippet, that snippet must integrate seamlessly into the broader, complex codebase. - Comprehensive test suite. The selected OSS must include an actively maintained, well-designed test suite with broad coverage (e.g., 60% code… view at source ↗
Figure 2
Figure 2. OSS-BENCH uses daily code prompts to simulate common LLM-assisted coding workflows. re-compiles the entire project and check its compilability. The compilation result can be success, failure and unrelated (i.e., changing such functions do not affect the compilation). OSS-BENCH records each function’s compilation result, and the overall compilation pass rate (unrelated functions are excluded) is used as the final sco… view at source ↗
Figure 3
Figure 3. Visualized Test Pass Rates of 1,000 Test Iterations in OSS-B [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: More memorization adds incorrect statements in advanced models [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Models need caution when generating code for familiar content, such cases may not require [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 48 canonical work pages

  1. [1]

    Github copilot.https://copilot.github.com, 2021

    GitHub, Inc. Github copilot.https://copilot.github.com, 2021

    work page 2021

  2. [2]

    Cursor: The ai-powered code editor.https://cursor.so, 2023

    Cursor Team. Cursor: The ai-powered code editor.https://cursor.so, 2023

    work page 2023

  3. [3]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems, 36:21558–21572, 2023

    2023

  4. [4]

    Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models

    Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models. In2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 684–709. IEEE, 2024

    work page 2024

  5. [5]

    Dynacode: A dynamic complexity-aware code benchmark for evaluating large language models in code generation.arXiv preprint arXiv:2503.10452, 2025

    Wenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang, Yue Zhang, and Kaidi Xu. Dynacode: A dynamic complexity-aware code benchmark for evaluating large language models in code generation.arXiv preprint arXiv:2503.10452, 2025

    arXiv 2025

  6. [6]

    Humanevo: An evolution-aware benchmark for more realistic evaluation of repository-level code generation

    Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. Humanevo: An evolution-aware benchmark for more realistic evaluation of repository-level code generation. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 764–764. IEEE Computer Society, 2025

    work page 2025

  7. [7]

    Codeif-bench: Evaluating instruction-following capabilities of large language models in interactive code generation.arXiv preprint arXiv:2503.22688, 2025

    Peiding Wang, Li Zhang, Fang Liu, Lin Shi, Minxiao Li, Bo Shen, and An Fu. Codeif-bench: Evaluating instruction-following capabilities of large language models in interactive code generation.arXiv preprint arXiv:2503.22688, 2025

    arXiv 2025

  8. [8]

    Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code.arXiv preprint arXiv:2311.09835, 2023

    Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al. Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code.arXiv preprint arXiv:2311.09835, 2023

    arXiv 2023

Show all 61 references
  1. [9]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  2. [10]

    Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida I. Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code, 2024

    2024

  3. [11]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam N. B. Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel R. Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, N...

    2024

  4. [12]

    Multi-swe-bench: A multilingual benchmark for issue resolving, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-swe-bench: A multilingual benchmark fo...

    2025 arXiv

  5. [13]

    SecRepoBench: Benchmarking LLMs for secure code generation in real-world repositories, 2025

    Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. SecRepoBench: Benchmarking LLMs for secure code generation in real-world repositories, 2025

    2025

  6. [14]

    CWEval: Outcome-driven evaluation on functionality and security of LLM code generation, 2025

    Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. CWEval: Outcome-driven evaluation on functionality and security of LLM code generation, 2025

    2025

  7. [15]

    Beyond correctness: Benchmarking multi-dimensional code generation for large language models

    Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. Beyond correctness: Benchmarking multi-dimensional code generation for large language models. arXiv preprint arXiv:2407.11470, 2024

    2024 arXiv

  8. [16]

    Codearena: Inspecting and improving code quality metrics using minecraft

    Simon Baars and Sander Meester. Codearena: Inspecting and improving code quality metrics using minecraft. In2019 IEEE/ACM International Conference on Technical Debt (TechDebt), pages 68–70. IEEE, 2019

    2019

  9. [17]

    CodeElo: Benchmarking competition-level code generation of LLMs with human-comparable elo ratings, 2025

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking competition-level code generation of LLMs w...

    2025

  10. [18]

    ComplexCodeEval: A benchmark for evaluating large code models on more complex code

    Jia Feng, Jiachen Liu, Cuiyun Gao, Chun Yong Chong, Chaozheng Wang, Shan Gao, and Xin Xia. ComplexCodeEval: A benchmark for evaluating large code models on more complex code. In39th IEEE/ACM International Conference on Automated Software Engineering (ASE), Sacramento, CA, USA, 2024

    2024

  11. [19]

    PythonSaga: Redefining the benchmark for code generating LLMs, 2024

    Ankit Yadav and Mayank Singh. PythonSaga: Redefining the benchmark for code generating LLMs, 2024

    2024

  12. [20]

    How well do llms generate code for different application domains? benchmark and evaluation.arXiv preprint arXiv:2412.18573, 2024

    Dewu Zheng, Yanlin Wang, Ensheng Shi, Hongyu Zhang, and Zibin Zheng. How well do llms generate code for different application domains? benchmark and evaluation.arXiv preprint arXiv:2412.18573, 2024

    2024 arXiv

  13. [21]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?, 2023

    2023

  14. [22]

    ClassEval: A manually-crafted benchmark for evaluating LLMs on class-level code generation, 2023

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. ClassEval: A manually-crafted benchmark for evaluating LLMs on class-level code generation, 2023

    2023

  15. [23]

    HumanEval-XL: A multilingual code generation benchmark for cross-lingual natural language generalization, 2024

    Qiwei Peng, Yekun Chai, and Xuhong Li. HumanEval-XL: A multilingual code generation benchmark for cross-lingual natural language generalization, 2024

    2024

  16. [24]

    Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. InProceedings of the 29th ACM SIGKD...

    2023 doi

  17. [25]

    {AddressSanitizer}: A fast address sanity checker

    Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. {AddressSanitizer}: A fast address sanity checker. In2012 USENIX annual technical conference (USENIX ATC 12), pages 309–318, 2012

    2012

  18. [26]

    php-src: The php interpreter.https://github.com/php/php-src, 2025

    The PHP Group. php-src: The php interpreter.https://github.com/php/php-src, 2025

    2025

  19. [27]

    Richard Hipp

    D. Richard Hipp. Sqlite version 3.49.1.https://sqlite.org/index.html, 2025

    2025

  20. [28]

    https://testing.googleblog.com/2020/08/ code-coverage-best-practices.html, 2020

    Code coverage best practices. https://testing.googleblog.com/2020/08/ code-coverage-best-practices.html, 2020

    2020

  21. [29]

    libclang: C interface to the clang library

    The LLVM Project. libclang: C interface to the clang library. https://clang.llvm.org/doxygen/ group__CINDEX.html, 2025

    2025

  22. [30]

    Kostya Serebryany.{OSS-Fuzz}-google’s continuous fuzzing service for open source software. 2017

    2017

  23. [31]

    difflib — helpers for computing deltas between objects

    Python Software Foundation. difflib — helpers for computing deltas between objects. https://docs. python.org/3/library/difflib.html, 2025

    2025

  24. [32]

    gpt-o1 model.https://platform.openai.com/docs/models/o1, 2025

    OpenAI. gpt-o1 model.https://platform.openai.com/docs/models/o1, 2025

    2025

  25. [33]

    o3-mini model.https://platform.openai.com/docs/models/o3-mini, 2025

    OpenAI. o3-mini model.https://platform.openai.com/docs/models/o3-mini, 2025

    2025

  26. [34]

    Claude 3.7 sonnet.https://www.anthropic.com/claude/sonnet, 2025

    Anthropic, Inc. Claude 3.7 sonnet.https://www.anthropic.com/claude/sonnet, 2025

    2025

  27. [35]

    Claude 3.5 haiku.https://www.anthropic.com/claude/haiku, 2025

    Anthropic, Inc. Claude 3.5 haiku.https://www.anthropic.com/claude/haiku, 2025

    2025

  28. [36]

    Gemini 2.5 flash.https://developers.generativelanguage.google/, 2025

    Google LLC. Gemini 2.5 flash.https://developers.generativelanguage.google/, 2025

    2025

  29. [37]

    Llama 3.3 70b instruct (fp16)

    Meta Platforms, Inc. Llama 3.3 70b instruct (fp16). https://ollama.com/library/llama3.3: 70b-instruct-fp16, 2025

    2025

  30. [38]

    Codellama 70b instruct (fp16)

    Code Llama. Codellama 70b instruct (fp16). https://ollama.com/library/codellama: 70b-instruct-fp16, 2025

    2025

  31. [39]

    Qwen 2.5 coder 32b instruct (fp16)

    Qwen AI. Qwen 2.5 coder 32b instruct (fp16). https://ollama.com/library/qwen2.5-coder: 32b-instruct-fp16, 2025

    2025

  32. [40]

    Qwen 3.0 30b-a3b fp16.https://ollama.com/library/qwen3:30b-a3b-fp16, 2025

    Qwen AI. Qwen 3.0 30b-a3b fp16.https://ollama.com/library/qwen3:30b-a3b-fp16, 2025

    2025

  33. [41]

    Qwen 3 8b fp16.https://ollama.com/library/qwen3:8b-fp16, 2025

    Qwen AI. Qwen 3 8b fp16.https://ollama.com/library/qwen3:8b-fp16, 2025

    2025

  34. [42]

    Gemma 3 27b-it fp16.https://ollama.com/library/gemma3:27b-it-fp16, 2025

    Gemma AI. Gemma 3 27b-it fp16.https://ollama.com/library/gemma3:27b-it-fp16, 2025. 11

    2025

  35. [43]

    Qwen 2.5 coder 14b instruct (fp16)

    Qwen AI. Qwen 2.5 coder 14b instruct (fp16). https://ollama.com/library/qwen2.5-coder: 14b-instruct-fp16, 2025

    2025

  36. [44]

    Deepseek coder v2 16b lite instruct (fp16)

    DeepSeek AI. Deepseek coder v2 16b lite instruct (fp16). https://ollama.com/library/ deepseek-coder-v2:16b-lite-instruct-fp16, 2025

    2025

  37. [45]

    Starcoder2-15b-instruct-v0.1 (fp16)

    BigCode Project. Starcoder2-15b-instruct-v0.1 (fp16). https://ollama.com/library/starcoder2: 15b-instruct-v0.1-fp16, 2024

    2024

  38. [46]

    Phi-4 14b fp16.https://ollama.com/library/phi4:14b-fp16, 2025

    Phi Labs. Phi-4 14b fp16.https://ollama.com/library/phi4:14b-fp16, 2025

    2025

  39. [47]

    Mistral 7b instruct (fp16)

    Mistral AI. Mistral 7b instruct (fp16). https://ollama.com/library/mistral:7b-instruct-fp16 , 2025

    2025

  40. [48]

    Codegemma 7b instruct (fp16)

    Gemma AI. Codegemma 7b instruct (fp16). https://ollama.com/library/codegemma: 7b-instruct-fp16, 2025

    2025

  41. [49]

    Openai: Advances in safe and beneficial ai.https://openai.com, 2025

    OpenAI. Openai: Advances in safe and beneficial ai.https://openai.com, 2025

    2025

  42. [50]

    Anthropic: Building reliable, steerable ai systems

    Anthropic, Inc. Anthropic: Building reliable, steerable ai systems. https://www.anthropic.com, 2025

    2025

  43. [51]

    Google: Organizing the world’s information.https://www.google.com, 2025

    Google LLC. Google: Organizing the world’s information.https://www.google.com, 2025

    2025

  44. [52]

    Deepseek: Developer of high-performance open-source llms

    DeepSeek AI. Deepseek: Developer of high-performance open-source llms. https://deepseek.ai, 2023

    2023

  45. [53]

    Alibaba group: Global trade and technology.https://www.alibaba.com, 2025

    Alibaba Group. Alibaba group: Global trade and technology.https://www.alibaba.com, 2025

    2025

  46. [54]

    Meta: Bringing the metaverse and social technology together

    Meta Platforms, Inc. Meta: Bringing the metaverse and social technology together. https://about. meta.com, 2025

    2025

  47. [55]

    Microsoft: Empowering every person and organization

    Microsoft Corporation. Microsoft: Empowering every person and organization. https://www. microsoft.com, 2025

    2025

  48. [56]

    Bigcode: Open and responsible development of code llms

    BigCode Project. Bigcode: Open and responsible development of code llms. https://www. bigcode-project.org, 2025

    2025

  49. [57]

    Mistral ai: Frontier ai in your hands.https://mistral.ai, 2025

    Mistral AI. Mistral ai: Frontier ai in your hands.https://mistral.ai, 2025

    2025

  50. [58]

    Ollama: Get up and running with large language models locally

    Ollama, Inc. Ollama: Get up and running with large language models locally. https://ollama.com, 2025

    2025

  51. [59]

    Spoc: Search-based pseudocode to code

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pseudocode to code. https://arxiv.org/abs/1906.04908, 2019. arXiv preprint arXiv:1906.04908

    1906 arXiv

  52. [60]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  53. [61]

    You are an expert on open-source software

    Yuancheng Jiang, Chuqi Zhang, Bonan Ruan, Jiahao Liu, Manuel Rigger, Roland Yap, and Zhenkai Liang. Fuzzing the php interpreter via dataflow fusion.arXiv preprint arXiv:2410.21713, 2024. 12 Appendix Contamination Prompt and Example Function.The system prompt is“You are an expe...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.