Pith. sign in

REVIEW 3 major objections 5 minor 51 references

SciCodePile's 128GB scientific code corpus and 200 executable tasks show current LLMs are unreliable scientific coders, with best Pass@1 of 12.30% and completion CodeBLEU near 38.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:24 UTC pith:GGXWJGXD

load-bearing objection A genuinely large and potentially useful scientific-code corpus and executable benchmark, but the training-utility claims are invalidated by direct train/eval leakage. the 3 major comments →

arxiv 2607.19104 v1 pith:GGXWJGXD submitted 2026-07-21 cs.SE cs.AI

SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

Weifeng Sun , Ye Fan , Yuchen Chen , Gou Tan , Jieke Shi , Yuan Yidi , Swee Liang Wong , Jonathan Pan
show 1 more author
David Lo
This is my paper
classification cs.SE cs.AI
keywords scientific code generationcode corpusLLM evaluationexecutable benchmarkpass@kcontinued pretraininginstruction tuningcomputational science
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to answer a specific question: how well can large language models write genuinely correct scientific code, not just plausible code? To make the question answerable at scale, the authors build SciCodePile, a 128GB corpus of code from 37,737 public repositories across computational science, and turn a slice of it into 200 executable tasks with sandboxed tests. Their measurements show that every model they test—15 open and closed systems—falls far short: the best reaches 12.30% Pass@1 and completion CodeBLEU reaches only about 38. They then argue the corpus still has training value, reporting that continued pretraining improves CodeBLEU by 2.84× on completion and instruction tuning improves Pass@1 by 4.79× on the benchmark. If the results are right, the field has both a large shared resource and a hard target to beat.

Core claim

This paper's central claim is that scientific code generation is a distinct and largely unsolved problem, and that prior resources were too small or too narrow to expose the gap. SciCodePile is presented as the largest scientific code corpus to date, assembled by retrieving candidate repositories with an expanded scientific vocabulary, filtering them by quality and LLM-judged relevance, and organizing the result into four aligned formats: raw files, README summaries, function-instruction pairs, and problem-solution pairs. From the function-level format, the authors construct an executable benchmark of 200 pure-Python tasks, each with a synthesized test suite and sandboxed runner; a model suc

What carries the argument

The load-bearing machinery is a retrieve-then-filter pipeline ending in a scoring identity used across the corpus. Repositories are found from 198 hand-curated seed keywords expanded by an LLM into 213 queries, then filtered to 37,737 by star count, non-code content, and README-based relevance judgment. Each candidate function is ranked by Finalscore = mdscore × codescore, where mdscore is the embedding similarity between the scientific keyword set and the repository README, and codescore is the similarity to the function body; this product selects functions that are scientifically relevant at both repository and implementation level. The executable benchmark then restates the top-ranked pur

Load-bearing premise

The load-bearing premise is that the evaluation data and training data are disjoint. Section 5.1 samples 1,000 evaluation files from F1, Section 3.3 derives the 200 benchmark tasks from F3, and Section 5.4 trains on F1 and F3/F4; no deduplication, decontamination, or held-out split is described. If those sets overlap, the reported ×2.84 and ×4.79 gains do not show improved scientific code generation.

What would settle it

Retrain both models on a version of F1/F3/F4 from which every file, function, and snippet that overlaps the 200 benchmark tasks has been removed by content hash and exact match, then re-evaluate. If the gains persist, the training-signal claim survives; if they disappear, the results are contamination artifacts. A cheaper first check is a hash-based overlap report between the 200 task source functions and F1/F3/F4, which the released corpus makes possible.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Scientific code generation is now measurable, and current models are far from reliable: on the released benchmark, no tested model reaches 13% Pass@1.
  • Text-similarity metrics can be misleading: CodeBLEU scores near 38 coexist with Pass@1 below 12.3%, so executable verification is necessary for scientific code claims.
  • The corpus can support continued pretraining and instruction tuning; the reported small-model gains suggest scaling these experiments may yield further improvement.
  • The multi-granularity formats link repository intent to function-level instructions, giving a path toward training models that use README-level context rather than only code tokens.
  • Benchmark difficulty is higher than prior scientific data-science benchmarks: for example, DeepSeek-Coder-6.7B drops from 16.30% on DSCodeBench to 6.10% Pass@1 here.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark tasks are pure Python with stubbed dependencies, the next natural stress test is a container-based variant that preserves real external packages; the current numbers likely underestimate the difficulty of end-to-end scientific workflows.
  • The multi-granularity formats invite a concrete experiment the paper does not run: feed the README summary to the model together with the function prompt and measure whether Pass@1 rises, since the failure analysis implicates missing project-level context.
  • The 12.3% ceiling gives the community a simple yardstick: hold the 200 tasks fixed and report Pass@1 before and after any new training method; moving it materially while keeping tests identical would be the first clear sign of progress on scientific code.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SciCodePile introduces a 128GB corpus of scientific code collected from 37,737 GitHub repositories, organized into four aligned formats (F1 raw files, F2 README summaries, F3 function-instruction pairs, F4 problem-solution pairs), plus a 200-task executable benchmark with sandboxed tests. The paper evaluates 15 LLMs on prefix-to-suffix completion, fill-in-the-middle, and executable code generation, reporting that the best model reaches only 12.30% Pass@1. It also claims that continued pretraining on F1 improves CodeBLEU by 2.84x on completion and instruction tuning on F3/F4 improves Pass@1 by 4.79x on the executable benchmark.

Significance. If the corpus and benchmark are clean, this is a valuable community resource. The executable benchmark addresses a real gap: most scientific code benchmarks do not verify functional correctness, and the manual audit of 40 tasks (85% full alignment, Cohen's kappa 0.92) is a strength. The multi-granularity F1-F4 alignment and the public release are also positive. However, the training-utility claims are central to the paper's contribution and are currently confounded by data overlap; they need to be re-established with proper held-out evaluation before the paper's conclusions can be accepted.

major comments (3)
  1. [§5.1 and §5.4, Table 4 (left)] The PTS evaluation set is randomly sampled from F1 (Sec. 5.1), and the continued pretraining of GPT-2 is performed on F1 (Sec. 5.4) with no removal of those 1,000 evaluated files. The evaluation files are therefore a subset of the training data by construction. The reported gains (CodeBLEU x2.84, Exact Match x20) can be explained by memorization of exact files rather than by improved scientific code-completion ability. Please re-run with a held-out split (e.g., held-out repositories or files) and report overlap statistics, or remove/reframe the utility claim.
  2. [§3.3 and §5.4, Table 4 (right)] The executable benchmark tasks are constructed from F3 functions (Sec. 3.3), and the instruction-tuning experiment uses F3/F4 (Sec. 5.4). Since F3 contains the reference functions and F4 is generated from F3, the fine-tuning set includes the reference implementations of the very tasks used for evaluation. The Pass@1 improvement from 1.90% to 9.10% is not evidence of generalization to unseen tasks. The paper must either construct the benchmark from data excluded from F3/F4, apply explicit decontamination, or report overlap statistics between training and evaluation instances.
  3. [Table 1 and Reference [10]] The paper claims that SciCodePile is the largest scientific code corpus to date, but Table 1 lists ChemPile as 15.6GB while the cited reference [10] has the title 'Chempile: A 250gb diverse and curated dataset for chemical foundation models.' If the 15.6GB in Table 1 is only the code subset of ChemPile, this needs to be stated and justified; otherwise the claim that 128GB is the largest is internally inconsistent with the paper's own reference. This is load-bearing for the primary contribution.
minor comments (5)
  1. [Abstract, §3.2, §3.4] The corpus size is given as 128GB in the abstract and Table 1, but 125GB in §3.2 and §3.4. Please unify the numbers.
  2. [§5.1 vs §5.4] The PTS evaluation in §5.1 uses a 1,024-token prefix and 256-token suffix, while §5.4 says the setting was adjusted to a 768-token prefix for the GPT-2 experiment. Please clarify why the setting changes and whether this affects comparability.
  3. [§5.2] The Pass@k estimation in Eq. (2) does not specify the number of samples n generated per task, the sampling temperature, or whether n is the same for Pass@1 and Pass@5. Please provide this information in the experimental setup.
  4. [References] References [29] and [31] are the same HumanEval paper and should be merged.
  5. [§3.3] Since the benchmark tasks are derived from public GitHub repositories, the authors should discuss whether the evaluated models may have seen these repositories during pretraining, and whether any decontamination was applied to the target model evaluation (distinct from the utility experiments). This is relevant to interpreting the absolute Pass@1 numbers.

Circularity Check

2 steps flagged

Training-utility gains are in-sample by construction: PTS evaluation files are sampled from F1 and then GPT-2 is pretrained on all of F1; benchmark tasks are derived from F3 while instruction tuning uses F3/F4.

specific steps
  1. self definitional [Section 5.1 and Section 5.4 (Table 4, left)]
    "We randomly sample 1,000 class-level code files from the collected corpus F1 to construct evaluation instances. ... We perform 5-epoch continued pretraining of GPT-2 (124M) on F1, using a maximum sequence length of 1024 and adjusting the prefix-to-suffix evaluation to a 768-token prefix with a 256-token suffix."

    The PTS evaluation set is defined as a random sample of F1, and the continued-pretraining corpus is F1 with no exclusion of those sampled files. Hence every evaluation instance is inside the training set by construction. The reported CodeBLEU gain (×2.84) and Exact Match gain (×20) can be produced by memorizing exact training files and do not demonstrate transfer to held-out scientific code. The paper describes no held-out split, no decontamination, and no overlap statistics between the evaluation sample and the pretraining corpus.

  2. fitted input called prediction [Section 3.3, Section 5.4 (Table 4, right)]
    "We construct candidate tasks from F3, which provides function-level code–description alignments grounded in real-world repositories. ... We perform instruction tuning of Qwen2.5-Coder-0.5B on our scientific instruction data and evaluate on the executable benchmark. ... fine-tuning improves Pass@1 from 1.90% to 9.10% (×4.79)."

    The executable benchmark tasks are selected from F3 functions, and F3 records include the raw reference code behind those functions. F4 is itself built by sampling 20,000 functions from F3. Instruction tuning on F3/F4 therefore includes the very reference implementations used to construct the benchmark prompts and test harnesses. The Pass@1 improvement is an in-sample fit to the task source pool, not evidence of generalization to unseen scientific tasks. No exclusion of benchmark-derived functions from the tuning data is reported.

full rationale

The corpus construction and the 15-model benchmark evaluation are not circular: benchmark correctness is determined by executing generated code against test harnesses, and the model comparisons are external assessments of existing systems. The circularity is confined to the two training-utility claims advertised in the abstract and contributions. Concretely, Section 5.1 samples the PTS evaluation files from F1, and Section 5.4 then pretrains GPT-2 on all of F1, so the evaluation set is a subset of the training set by construction. Likewise, Section 3.3 derives benchmark tasks from F3 functions, while Section 5.4 instruction-tunes on F3/F4, so the tuning data contains the function pool from which the benchmark is built. In both cases the reported improvements (×2.84 CodeBLEU and ×4.79 Pass@1) can be explained by memorization of in-sample instances; they do not establish transferable scientific code-generation ability. The paper reports no held-out split, decontamination step, or overlap statistics. Because the main benchmark results remain independent, the paper is partially but not wholly circular, warranting a score of 7 rather than a higher score.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. The free parameters are construction choices (star threshold, truncation, cutoff values, dedup threshold) that affect the resource but are not fitted to external data. The key unstated assumption is the disjointness of training and evaluation data in the utility experiments.

free parameters (4)
  • repository star threshold = >=10
    Used to filter low-quality repositories; affects the composition and scale of the corpus. Chosen following prior work, not empirically optimized for this task.
  • README truncation length = 8,000 tokens
    F2 summaries are generated from the first 8,000 tokens of each README, which constrains the project-level context available to downstream formats.
  • finalscore selection cutoff = not specified
    Functions are ranked by mdscore × codescore, but the numerical threshold used to select F3 instances, F4 samples, and the 200 benchmark candidates is never stated, making the selection process hard to reproduce.
  • SimHash near-duplicate threshold = not specified
    F1 deduplication removes near-duplicates using SimHash, but the similarity threshold is not reported, affecting the exact file count and corpus size.
axioms (6)
  • domain assumption GitHub search APIs and star counts provide a representative and quality-controlled sample of scientific repositories.
    Central to the corpus claim; the >=10 star threshold may bias toward popular projects and exclude valid scientific code.
  • domain assumption LLM relevance judgments (Qwen3-32B) and NatureLM verification are accurate enough to filter the corpus and F3 descriptions at scale.
    Used without full human validation; only 50 F3 descriptions and 40 benchmark tasks were manually audited.
  • domain assumption Embedding similarity (Finalscore, Eq. 1) correlates with scientific relevance.
    The entire function selection and benchmark task ranking depend on this correlation, which is not independently validated.
  • domain assumption GPT-5.3-Codex-synthesized test harnesses correctly capture functional correctness.
    Only 85% of 40 manually reviewed tasks were perfectly aligned; 15% had minor issues, and the remaining 160 tasks were not manually audited.
  • ad hoc to paper The evaluation data is disjoint from the training data in the utility experiments.
    No decontamination or train/eval split is described when using F1 for pretraining and F3/F4 for instruction tuning while evaluating on samples from F1/F3.
  • domain assumption Benchmark tasks are not contaminated by the pretraining corpora of the evaluated LLMs.
    No contamination analysis is provided, despite the benchmark being built from public GitHub repositories that may appear in LLM training data.

pith-pipeline@v1.3.0-alltime-deepseek · 21914 in / 10833 out tokens · 91402 ms · 2026-08-01T13:24:47.569529+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) excel at general-purpose code generation, yet how well they handle scientific code remains an open question. Existing datasets and benchmarks are limited in scale, domain coverage, or executable verification, leaving the true gap between current LLMs and reliable scientific code generators inadequately assessed. To address these limitations, we present SciCodePile, the largest scientific code corpus to date, constructed from 37,737 public repositories and collectively comprising 128GB of code that spans multiple computational science disciplines. From this corpus, we further curate an executable benchmark of 200 tasks, each equipped with a sandboxed execution environment and an automated test harness for functional verification. We evaluate 15 LLMs from both open-source and closed-source families on three tasks: prefix-to-suffix completion, fill-in-the-middle infilling, and executable code generation. Results show that scientific code generation remains highly challenging: The best CodeBLEU reaches only 38.13 and 38.37 on the two completion tasks, while the strongest model achieves just 12.30\% Pass@1 on the executable benchmark, underscoring how far current models remain from reliable scientific code generation. To demonstrate the training utility of SciCodePile, we further show that continued pretraining on our corpus improves CodeBLEU by $\times$2.84 on scientific code completion, and instruction tuning on our data improves Pass@1 by $\times$4.79 on the executable benchmark. All code and data are available at https://huggingface.co/SciCodePile.

Figures

Figures reproduced from arXiv: 2607.19104 by David Lo, Gou Tan, Jieke Shi, Jonathan Pan, Swee Liang Wong, Weifeng Sun, Ye Fan, Yuan Yidi, Yuchen Chen.

Figure 1
Figure 1. Figure 1: Overall workflow of dataset construction. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of domain-language distribution and benchmark prompt embedding space. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Programming-language characteristics in the collected scientific code corpus: (a) distribu [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Failure breakdown of different models on the executable benchmark. Each bar shows the [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Word cloud of scientific keywords extracted from the collected repositories. The size of [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Programming-language distribution in the collected scientific code corpus. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An simplified example in Function Instruction dataset. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An simplified example in executable benchmark. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt Template for Expanding Scientific Keywords. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt Template for Judging Repo. Relevance. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt Template for Generating README Summaries. K.3 Prompt Template for F3 To construct the F3 function-instruction dataset, we design a prompt to generate structured natural￾language descriptions for individual functions. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt Template for Generating Function Description. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt Template for Generating Problem Solution Pairs. [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt Template for Generating Test Cases. [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Deepseek-coder: when the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196, 2024

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, et al. Deepseek-coder: when the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology, 33(8):1–79, 2024

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology, 33(8):1–79, 2024

    2024

  4. [4]

    Rlcoder: Reinforcement learning for repository-level code completion

    Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. Rlcoder: Reinforcement learning for repository-level code completion. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1140–1152. IEEE, 2025

    2025

  5. [5]

    A survey on large language models for code generation.ACM Transactions on Software Engineering and Methodology, 35 (2):1–72, 2026

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation.ACM Transactions on Software Engineering and Methodology, 35 (2):1–72, 2026

    2026

  6. [6]

    Calibration and correctness of language models for code

    Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. Calibration and correctness of language models for code. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 540–552. IEEE, 2025

    2025

  7. [7]

    Quantum mechanics in drug discovery: a comprehensive review of methods, applications, and future directions.International Journal of Molecular Sciences, 26(13):6325, 2025

    Sarfaraz K Niazi. Quantum mechanics in drug discovery: a comprehensive review of methods, applications, and future directions.International Journal of Molecular Sciences, 26(13):6325, 2025

    2025

  8. [8]

    Structure-based drug discovery.Molecular Modeling and Docking Techniques for Drug Discovery and Design, pages 435–472, 2025

    Shreya Satyanarayan Bhat, Vidya Niranjan, Spoorti Anil Bandikatte, Soumya Govindappa, and Adarsh Vishal. Structure-based drug discovery.Molecular Modeling and Docking Techniques for Drug Discovery and Design, pages 435–472, 2025

    2025

  9. [9]

    Abubakar Sadiq Bello, Adamu Uzairu, Shallangwa G Aadmu, AA Razzak Mahmood, Muham- mad Tukur Ibrahim, et al. Molecular dynamics simulation, molecular docking, admet prediction, qsar modeling and density functional theory computation for the identification of possible hep- atitis c virus ns5b protease inhibitors through in silico study of fluorine-2, 7-diam...

    2025

  10. [10]

    Chempile: A 250gb diverse and curated dataset for chemical foundation models.arXiv preprint arXiv:2505.12534, 2025

    Adrian Mirza, Nawaf Alampara, Martiño Ríos-García, Mohamed Abdelalim, Jack Butler, Bethany Connolly, Tunca Dogan, Marianna Nezhurina, Bünyamin ¸ Sen, Santosh Tirunagari, et al. Chempile: A 250gb diverse and curated dataset for chemical foundation models.arXiv preprint arXiv:2505.12534, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Scicode: A research coding benchmark curated by scientists.Advances in Neural Information Processing Systems, 37:30624–30650, 2024

    Minyang Tian, Luyu Gao, Shizhuo D Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists.Advances in Neural Information Processing Systems, 37:30624–30650, 2024

    2024

  12. [12]

    Mattools: Bench- marking large language models for materials science tools.arXiv preprint arXiv:2505.10852, 2025

    Siyu Liu, Bo Hu, Beilin Ye, Jiamin Xu, David J Srolovitz, and Tongqi Wen. Mattools: Bench- marking large language models for materials science tools.arXiv preprint arXiv:2505.10852, 2025. 10

    arXiv 2025

  13. [13]

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080, 2024

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Discoverybench: Towards data-driven discovery with large language models.arXiv preprint arXiv:2407.01725, 2024

    Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth V ora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models.arXiv preprint arXiv:2407.01725, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Large language models are poor medical coders—benchmarking of medical code querying.Nejm Ai, 1(5):AIdbp2300040, 2024

    Ali Soroush, Benjamin S Glicksberg, Eyal Zimlichman, Yiftach Barash, Robert Freeman, Alexander W Charney, Girish N Nadkarni, and Eyal Klang. Large language models are poor medical coders—benchmarking of medical code querying.Nejm Ai, 1(5):AIdbp2300040, 2024

    2024

  16. [16]

    Dscodebench: A realistic benchmark for data science code generation.arXiv preprint arXiv:2505.15621, 2025

    Shuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun, Qihao Zhu, and Jie M Zhang. Dscodebench: A realistic benchmark for data science code generation.arXiv preprint arXiv:2505.15621, 2025

    arXiv 2025

  17. [17]

    Biocoder: a benchmark for bioinformatics code generation with large language models.Bioinformatics, 40(Supplement_1):i266–i276, 2024

    Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark B Gerstein. Biocoder: a benchmark for bioinformatics code generation with large language models.Bioinformatics, 40(Supplement_1):i266–i276, 2024

    2024

  18. [18]

    Hpc-coder: Modeling parallel programs using large language models

    Daniel Nichols, Aniruddha Marathe, Harshitha Menon, Todd Gamblin, and Abhinav Bhatele. Hpc-coder: Modeling parallel programs using large language models. InISC High Performance 2024 Research Paper Proceedings (39th International Conference), pages 1–12. Prometeus GmbH, 2024

    2024

  19. [19]

    Revisiting the identification of the co-evolution of production and test code.ACM Transactions on Software Engineering and Methodology, 32(6):1–37, 2023

    Weifeng Sun, Meng Yan, Zhongxin Liu, Xin Xia, Yan Lei, and David Lo. Revisiting the identification of the co-evolution of production and test code.ACM Transactions on Software Engineering and Methodology, 32(6):1–37, 2023

    2023

  20. [20]

    A large-scale empirical study on code-comment inconsistencies

    Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. A large-scale empirical study on code-comment inconsistencies. In2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), pages 53–64. IEEE, 2019

    2019

  21. [21]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Similarity estimation techniques from rounding algorithms

    Moses S Charikar. Similarity estimation techniques from rounding algorithms. InProceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 380–388, 2002

    2002

  23. [23]

    Tree-sitter: An incremental parsing system for programming tools

    Tree-sitter. Tree-sitter: An incremental parsing system for programming tools. https: //tree-sitter.github.io/, 2024. Accessed: 2026-03-03

    2024

  24. [24]

    Nature language model: deciphering the language of nature for scientific discovery.arXiv preprint arXiv:2502.07527, 2025

    Yingce Xia, Peiran Jin, Shufang Xie, Liang He, Chuan Cao, Renqian Luo, Guoqing Liu, Yue Wang, Zequn Liu, Yuan-Jyue Chen, et al. Nature language model: deciphering the language of nature for scientific discovery.arXiv preprint arXiv:2502.07527, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Magicoder: Empow- ering code generation with oss-instruct.arXiv preprint arXiv:2312.02120, 2023

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with oss-instruct.arXiv preprint arXiv:2312.02120, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    LM-CPPF: Paraphrasing- guided data augmentation for contrastive prompt-based few-shot fine-tuning

    Amirhossein Abaskohi, Sascha Rothe, and Yadollah Yaghoobzadeh. LM-CPPF: Paraphrasing- guided data augmentation for contrastive prompt-based few-shot fine-tuning. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 670–681, Toronto, Canada, July 2023. Association for Computational Linguistics

    2023

  27. [27]

    Out of style: Misadventures with LLMs and code style transfer

    Karl Munson, Chih-Kai Ting, Serenity Wade, Anish Savla, Julian Dolby, Kiran Kate, and Kavitha Srinivas. Out of style: Misadventures with llms and code style transfer.CoRR, abs/2406.10320, 2024. doi: 10.48550/ARXIV .2406.10320. URL https://doi.org/10. 48550/arXiv.2406.10320. 11

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.10320 2024

  28. [28]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview. net/forum?...

    2023

  29. [30]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    2008

  30. [31]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  31. [32]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  32. [33]

    Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297, 2020

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297, 2020

    Pith/arXiv arXiv 2009

  33. [34]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  34. [35]

    Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173, 2024

    Pith/arXiv arXiv 2024

  35. [36]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  36. [37]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  37. [38]

    ""Runs the web interface version of chemprop, allowing for 2training and predicting in a web browser

    Jacob Cohen. A coefficient of agreement for nominal scales.Educational and psychological measurement, 20(1):37–46, 1960. 12 A Models Used for Data Construction We use a fixed, role-based assignment of models throughout the construction pipeline. The choice is task-driven: rather than switching models arbitrarily across stages, we assign one model family t...

    1960

  38. [39]

    project context

    The summary should be suitable for use as "project context", with a length between 300-800 words

  39. [40]

    Focus on the core features, dependencies, and usage methods of the project

  40. [41]

    If the project involves scientific computing, chemistry, biology, physics, or related fields, please specifically note this

  41. [42]

    Extract key information: installation steps, runtime commands, input/output formats, etc

  42. [43]

    project_overview

    Keep it objective and accurate, avoiding adding information not present in the original documentation Below is the README file content to be summarized: --- {readme_content} --- Please output a JSON-formatted summary following this structure (must use JSON, do not include any other text): { "project_overview": "One-sentence overview of the project", "main...

  43. [44]

    A Python function or code snippet extracted from a larger project

  44. [45]

    setup_code

    The project's README file Your task is to write a **precise and detailed function description**, suitable as a docstring or a HumanEval-style task description for code generation. Requirements: - Clearly explain the function's purpose and what task it accomplishes - List all input parameters with their names, types, and meaning - Describe the return value...

  45. [46]

    test_code must define exactly one function signature: def check(candidate):

  46. [47]

    test_code must contain at least 3 asserts

  47. [48]

    No network

    Use only Python standard library. No network. No heavy external dependencies

  48. [49]

    If the function depends on unavailable globals/modules, put lightweight stubs/mocks in setup_code

  49. [50]

    setup_code may be empty string when not needed

  50. [51]

    In test_code, avoid importing non-standard modules

  51. [52]

    Figure 14: Prompt Template for Generating Test Cases

    Do NOT include markdown fences or explanations. Figure 14: Prompt Template for Generating Test Cases. 27