Pith. sign in

REVIEW 3 major objections 6 minor 53 references

From Discussion to Execution: Replicating Buggy and Correct Data Science Code

T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM-based generator–reviewer pipeline reconstructs executable buggy and patched data-science programs from informal Q&A posts.

desk verdict Useful pipeline and benchmark for turning Q&A threads into executable buggy/patched pairs, but the headline semantic numbers rest on an LLM judge that agrees with humans only half the time on patches — the same judge that drove generation. read the letter →

arxiv 2607.16569 v1 pith:FKFAU4OD submitted 2026-07-18 cs.SE

Ragib Shahariar Ayon , Mohammad Wardat , Shibbir Ahmed This is my paper
classification cs.SE
keywords bugreplicationlargelanguagemodelscodegenerationdatascienceStackOverflowGitHubissuesprogramrepairsemanticfaithfulness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reprodgen is an LLM-based pipeline that takes an informal Q&A thread—say, a Stack Overflow question about slow pandas code and its accepted answer—and produces two runnable Python programs: one that reproduces the buggy behavior described in the question, and one that applies the intended fix from the answer. It fills in what the discussion omits: mock data, imports, dependency versions, and random seeds, guided by three structured artifacts distilled from the post (code intent, functional requirements, structured chain-of-thought) and refined through a generator–reviewer loop that actually executes the code and checks the behavior. The paper argues this closes the gap between informal discussions and executable debugging benchmarks, and supports the claim with a human-vetted benchmark of 176 real cases plus experiments on 203 Stack Overflow posts and 50 GitHub issues across seven data-science libraries. The reported success rates, however, depend on an LLM judge whose agreement with human labels is modest, especially for patched code, which is the load-bearing assumption underneath the headline numbers.

What carries the argument

The load-bearing mechanism is the iterative Generator–Reviewer loop built around three staged artifacts: Code Intent, Functional Requirements, and Structured Chain-of-Thought. Each is generated separately for the buggy and patched versions and reviewed by an independent LLM against the original post before feeding the code generator. The generator outputs a full script with inferred dependencies and mock data, which is actually executed in an isolated container; execution errors and reviewer rationales feed back for refinement, capped at three iterations. This split—one model writes, another checks—carries the claim that the final pair is runnable and faithful.

What would settle it

Run the 203-post benchmark with human labeling on a held-out subset and recompute success rates; if human-labeled SRs are substantially lower than the LLM-judged SRs, or the model ranking flips, the reported numbers overstate true faithfulness. A cheaper probe: collect the judge's false-positive patch cases (judge 'correct', human 'incorrect') and check for a systematic pattern, such as accepting code that runs but changes the output semantics.

Watch

Extended reading notes

Core claim

The central claim is that a structured Generator–Reviewer workflow can reliably reconstruct executable buggy–patched program pairs directly from Q&A discussions, without the original code or data. The framework derives buggy and patched versions of three guidance artifacts—Code Intent, Functional Requirements, Structured Chain-of-Thought—and uses them to synthesize code with inferred dependencies and mock data. A reviewer LLM checks the generated code and its actual execution output against the post, repeating up to three iterations. In the evaluation, the best configuration achieved execution rates of 60% (buggy) and 52% (patched) on 203 Stack Overflow posts, with success rates of 58% and 5

Load-bearing premise

The success-rate metric assumes the LLM judge is a valid scorer of semantic faithfulness, yet the paper measures that judge's agreement with humans at only F1 = 0.78 on buggy code and 0.50 on patched code, so any leniency or miscalibration in the judge directly inflates or distorts the reported success rates.

Editorial extensions

If this is right

  • Executable buggy–patched pairs can be produced automatically from forum discussions, providing a cheaper path to debugging and repair benchmarks than manual curation.
  • Buggy-code replication is systematically more reliable than patched-code replication across models, so repair benchmarks should expect and account for that gap.
  • LLM reviewers are reliable for textual intermediate artifacts but degrade sharply on code-level judgment, especially for patches, meaning any LLM-judged code correctness inherits that weakness.
  • Structured guidance substantially improves executability (buggy-code execution rate rose from 0.70 to 0.96 in the ablation) without harming semantic alignment.
  • Generated programs are reproducible by construction — fixed seeds, pinned dependencies, synthesized data — which is necessary for downstream debugging and verification tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LLM judge were replaced or calibrated, absolute success rates and even the ranking of generator models could shift; the paper's own reviewer F1 numbers (0.78 buggy / 0.50 patched) imply the reported SRs are upper bounds modulo judge leniency.
  • The same generator–reviewer architecture could transfer to other informal bug-report sources (e.g., Jupyter notebooks, non-Python issue trackers) provided mock-data synthesis and dependency inference generalize.
  • A hybrid verification scheme—using LLMs for guidance artifacts but differential or property-based execution checks between the buggy and patched outputs—would likely give more trustworthy code-level success labels than an LLM judge alone.
  • A natural stress test: generate multiple candidate buggy–patched pairs for the same post with different seeds and measure diversity; if the framework collapses to one canonical pair, its usefulness for training repair models is limited.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Reprodgen, an LLM-based framework that automatically reconstructs executable buggy and patched Python data-science programs from informal Q&A discussions (Stack Overflow and GitHub Issues). The framework extracts Code Intent (CI), Functional Requirements (FR), and Structured Chain of Thought (SCoT) from the posts, then uses a Generator–Reviewer pair with iterative refinement and actual execution in isolated Docker containers to produce runnable code pairs. The authors construct two benchmarks (ReprodgenBench-B with 203 Stack Overflow posts and ReproIgenBench-GI with 50 GitHub Issues), validate them with human experts, and evaluate five open-source LLMs across Execution Rate (ER), Success Rate (SR), CodeBERTScore, Levenshtein distance, and efficiency. The central claim is that the framework "reliably replicates" buggy–patched pairs, with reported buggy SR up to 58.12% and patched SR up to 51.72%.

Significance. The problem is well-motivated: converting unstructured Q&A discussions into executable regression tests would be valuable for debugging, repair, and benchmarking. The strengths of the work are genuine: executability is objectively measured in Docker, the benchmarks and code are released, and the evaluation spans multiple models and datasets. The structured guidance approach is reasonable and the ablation study is a useful contribution. However, the headline semantic metric (SR) relies on an LLM judge whose code-level reliability is weak—particularly for patched code (F1=0.50 for Qwen3 in Table III)—and the same judge is used during generation. This makes the claimed semantic-success rates unreliable and undermines the central claim. If the SR metric is fixed (e.g., with a stronger judge or human evaluation), the benchmark and framework would be a solid foundation for future work.

major comments (3)
  1. [§IV-A4, §IV-B2, §IV-B3 (Eq. 1, Table III)] The Success Rate (SR) metric is computed as SR = (1/N)Σ 1[JUDGE(C_i,B_i)=correct], where JUDGE is an LLM. For all open-source generators, the paper standardizes on Qwen3 as the judge (§IV-B2). Table III shows that Qwen3's coverage-weighted F1 on the code-review subtask is only 0.78 for buggy code and 0.50 for patched code. A patched-code F1 of 0.50 means the judge's 'correct' label is barely more reliable than a coin flip for exactly the patched-code success rates that the paper headlines (e.g., 51.72% in the abstract). Moreover, Qwen3 is also the reviewer inside the generate-refine loop (§III-E), so generated code is iteratively refined until this same judge approves it. The SR therefore conflates 'semantically faithful to the original post' with 'satisfies Qwen3's review preferences.' The paper itself acknowledges this degradation in Finding 2, yet still uses Qwen3 for all headline SR
  2. [§IV-A2, §IV-A4 (B_i definition)] The SR metric requires a reference behavior B_i, but the paper does not precisely define B_i. For ReproIgenBench-B, the ground truth is described as 'LLM-synthesized artifacts' (Claude Sonnet 4.5 + GPT-4o) that are 'not treated as ground truth' but then 'assessed by two expert judges' to produce final labels. If B_i is a single human-validated candidate implementation, then SR measures agreement with one particular reference, not the space of valid behaviors described in the post. This is especially concerning because the reference candidates come from a different model family than the open-source generators being tested, potentially biasing SR and CodeBERTScore comparisons. Please clarify how B_i is constructed (e.g., is it the human-labeled candidate code, a natural-language specification, or a set of oracle behaviors?) and whether human judges validated multiple alternative correct im
  3. [§IV-B2 and Abstract] The paper concludes 'Results show reliable replication' in the abstract and conclusion, but the reported SR numbers are moderate: the best patched SR is 51.72% on ReproIgenBench-B, and several models are much lower. Given that ER only measures absence of runtime exceptions (Section IV-A4) and SR is currently unreliable as argued above, the claim of 'reliable' replication is not supported by the evidence presented. Please temper the central claim or provide additional evidence (e.g., a human evaluation on a random sample) to justify 'reliable.'
minor comments (6)
  1. [Abstract and §I] The abstract says 'Results show reliable replication,' but the patched-code SR is ~50% even for the best model. Suggest rewording to avoid overclaiming, e.g., 'moderate success with clear model differences.' Also, §I introduces ReproIgenBench-V as a 176-program benchmark, but the evaluation uses ReproIgenBench-B (203 posts) and ReproIgenBench-GI (50 issues). Clarify the relationship among V, B, and GI.
  2. [§IV-A2] The text alternates between 'two annotators' and 'two expert judges' when describing dataset curation. Are these the same people? Please make the annotation process explicit: who filtered the 578 candidates, who labeled the benchmark instances, and who mediated disagreements.
  3. [§IV-A3] Typo: 'choosen' should be 'chosen'.
  4. [Table III] The note says 'best, worst values' but the table does not visibly mark best/worst. Also, the 'Average' column appears to be an unweighted mean of eight F1 values, but this is not stated. Please specify the averaging method.
  5. [§VI] The stability check mentions 'rerunning a random subset' but does not report the subset size or the variance. Please provide concrete numbers.
  6. [References] References [1] and [30] appear to be the same paper (Islam et al., ESEC/FSE 2019). Please deduplicate.

Circularity Check

1 steps flagged · score 6.0 of 10

SR is computed by the same Qwen3 judge used inside the generate-refine loop; with patched-code F1 of 0.50, headline success rates largely measure judge self-consistency rather than independent semantic fidelity.

  1. self definitional [Section III-E; Section IV-A4; Section IV-B2/B3; Table III; Finding 2]
    "Success Rate (SR): SR measures the proportion of generated programs that successfully reproduce the target behavior, as determined by a judge function ... SR = 1/N ... 1[JUDGE(C i, Bi) = correct] ... In this paper, we use the terms Reviewer LLM and Judge LLM interchangeably. ... We therefore select Qwen 3 as the standardized reviewer LLM for computing SR across all generator models."

    The same JUDGE function both defines the success event in SR and acts as the Reviewer that iteratively refines generated code until it approves (Section III-E). For all open-source models, Qwen 3 is that judge (Section IV-B2), so the event counted by SR is exactly the event the pipeline was optimized to produce: the judge's 'correct' label. This is not an independent check of semantic faithfulness to the original Q&A post. The paper's own Table III/Finding 2 reports that this judge is near chance on patched code (Qwen 3 F1 = 0.50), so the headline patched SR of 51.72% largely measures self-consistency with an unreliable oracle rather than externally grounded correctness. Executability is verified independently, but semantic success is not.

full rationale

Most of the paper's derivation chain is independent and non-circular. The framework's artifacts (CI, FR, SCoT) are derived from the Q&A posts, code is executed in isolated Docker containers to measure ER, and the benchmark itself is human-validated by expert annotators with reported inter-annotator agreement. The self-citations to the authors' prior work (e.g., refs. [3], [31]) are used only for bug-characterization context and annotation-protocol precedent, not as load-bearing evidence for the central claim. The one significant circularity is the evaluation metric: SR is defined as agreement with an LLM judge, and the same LLM judge (Qwen 3) is also the reviewer inside the generate-refine loop, with the paper explicitly saying 'Reviewer LLM' and 'Judge LLM' are interchangeable. Thus the reported semantic success rates are, by construction, measures of how well the generator satisfies the reviewer it was already optimized against. The paper's own Table III and Finding 2 acknowledge the judge's poor patched-code reliability (F1 0.50), which confirms that this is not a merely hypothetical concern. Because executability and human benchmark validation provide independent grounding, the circularity is partial rather than total, warranting a score of 6 rather than higher.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The main design choices are the refinement cap and the LLM judge setup; the benchmark is a constructed dataset rather than a postulated entity. The three structured artifacts (CI, FR, SCoT) are prompt constructs, not independently evidenced entities.

free parameters (1)
  • Max refinement iterations per sub-task = 3
    A pilot study on 20 programs was used to set the cap at three iterations for both execution-driven and review-driven loops. This directly bounds ER/SR and efficiency numbers reported later.
assumptions (5)
  • domain assumption A Q&A post's question and selected answer contain enough information to reconstruct both the original buggy behavior and the intended fix.
    The entire pipeline depends on recovering missing context from informal posts; stated in Section III-A and III-C.
  • domain assumption The LLM reviewer's Correct/Incorrect labels are a valid proxy for semantic fidelity of generated code.
    SR uses JUDGE(C_i, B_i) implemented with LLM labels; Section IV-A4. Table III shows code-level F1 as low as 0.50, so this assumption is measurable and fragile.
  • domain assumption Execution without a runtime exception is a meaningful executability signal, even if the program's output is wrong.
    ER counts any program that runs without exceptions as executed; Section IV-A4.
  • domain assumption Human expert labels on LLM-synthesized code constitute reliable ground truth for benchmark instances.
    The SO benchmark starts from LLM-synthesized candidates that two human judges then label; Section IV-A2. The initial buggy-label kappa of 0.32 shows the difficulty of this judgment.
  • domain assumption The selected answer (accepted or highest-scored) reflects the intended fix described by the post.
    The Q&A parser selects one answer as the source of patched behavior; Section III-B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Discussion to Execution: Replicating Buggy and Correct Data Science Code." pith.science (2026). https://pith.science/paper/FKFAU4OD

@misc{pith2026260716569,
  author       = {Pith},
  title        = {Pith review of: From Discussion to Execution: Replicating Buggy and Correct Data Science Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FKFAU4OD}},
  note         = {Machine review of arXiv:2607.16569}
}
read the original abstract

Reproducing reliable data science code from informal sources is challenging due to ambiguous problem specifications, missing dependencies, and performance bottlenecks. Although developer Q&A forums provide rich discussions on diagnosing and fixing real-world issues, the information is often incomplete and unstructured, limiting its use for automated debugging and verification. In this paper, we introduce Reprodgen, a large language model (LLM) based framework for automatically replicating executable buggy and patched data science programs from Q&A forum posts. Given a question and its corresponding answer, Reprodgen reconstructs the buggy behavior described in the question and the intended fix described in the answer, producing executable buggy and patched code pairs that reflect the original discussion. The framework builds structured representations of code intent (CI), functional requirements (FR), and Structured Chain of Thought (SCoT), and iteratively refines code using an LLM-based reviewer until it is executable and semantically consistent. We evaluate Reprodgen on Stack Overflow (SO) and GitHub Issues (GI) across seven data science libraries, including pandas, numpy, and scikit-learn, and construct a benchmark of runnable buggy and patched programs validated by human experts. Our pipeline uses LLMs for semantic assessment, while executability is verified through actual execution. Results show reliable replication with clear differences in model performance.

Figures

Figures reproduced from arXiv: 2607.16569 by the authors.

Figure 1
Figure 1. Motivating example illustrating how ⃝D Reprodgen generates fully executable ⃝E buggy and ⃝F patched code, along with ⃝G metrics and ⃝H inferred requirements.txt, from ⃝A Question and Answer forum post #20076195. In contrast, prior state-of-the-art (SOTA) approaches such as ⃝B AutoCodeRover and ⃝C ArchCode typically produce partial, non-executable snippets that fail to replicate the complete functionality of the orig… view at source ↗
Figure 2
Figure 2. Overview of Reprodgen framework LLM-generated code mimics inefficient logic but fails to capture intended optimizations or necessary context. To evaluate whether existing state-of-the-art approaches can reproduce both the reproducible buggy code and its corresponding patched version, we experimented with Au￾toCodeRover and ArchCode, investigating how they performed for some posts [19]. AutoCodeRover (ACR) combines L… view at source ↗
Figure 3
Figure 3. Sampling methodology for ReprodgenBench-E. a) ReprodgenBench-B: Following the sampling method￾ology of Misu et al. [29] and to stay within budget constraints, we randomly sampled 203 posts from this curated set using a 95% confidence level and a 5% margin of error, yield￾ing ReprodgenBench-B, which is used for all experimental evaluations for Stack Overflow in this study. All artifacts in ReprodgenBench-B are first … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Execution and success ratios for buggy and patched code generation on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 7 linked inside Pith

  1. [2]

    Repairing deep neural networks: Fix patterns and challenges,

    M. J. Islam, R. Pan, G. Nguyen, and H. Rajan, “Repairing deep neural networks: Fix patterns and challenges,” inProceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE ’20. New York, NY , USA: Association for Computing Machinery, 2020, p. 1135–1146. [Online]. Available: https://doi.org/10.1145/3377811.3380378

    arXiv 2020

  2. [3]

    Charac- terizing bugs in python and r data analytics programs,

    S. Ahmed, M. Wardat, H. Bagheri, B. D. Cruz, and H. Rajan, “Charac- terizing bugs in python and r data analytics programs,”arXiv preprint arXiv:2306.08632, 2023

    arXiv 2023

  3. [4]

    Towards understanding performance bugs in popular data science libraries,

    H. Yang, Z. Li, Z. Zhong, X. Tang, and P. He, “Towards understanding performance bugs in popular data science libraries,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 2335–2358, 2025

    2025

  4. [5]

    Towards understanding fine-grained programming mistakes and fixing patterns in data science,

    W.-H. Chen, J. L. Cheoh, M. Keim, S. Brunswicker, and T. Zhang, “Towards understanding fine-grained programming mistakes and fixing patterns in data science,”Proceedings of the ACM on Software Engi- neering, vol. 2, no. FSE, pp. 1824–1846, 2025

    2025

  5. [6]

    Bug analysis in jupyter notebook projects: An empirical study,

    T. L. De Santana, P. A. D. M. S. Neto, E. S. De Almeida, and I. Ahmed, “Bug analysis in jupyter notebook projects: An empirical study,”ACM Trans. Softw. Eng. Methodol., vol. 33, no. 4, Apr. 2024. [Online]. Available: https://doi.org/10.1145/3641539

    doi:10.1145/3641539 2024

  6. [7]

    Why do machine learning notebooks crash? an empirical study on public python jupyter notebooks,

    Y . Wang, W. Meijer, J. A. H. Lopez, U. Nilsson, and D. Varro, “Why do machine learning notebooks crash? an empirical study on public python jupyter notebooks,”IEEE Transactions on Software Engineering, 2025

    2025

  7. [8]

    Studying vulnerable code entities in r,

    Z. Zhao, M. M. Das, and F. Fard, “Studying vulnerable code entities in r,” inProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension (ICPC), 2024, pp. 328–332

    2024

  8. [9]

    Knowledge- enhanced program repair for data science code,

    S. Ouyang, J. M. Zhang, Z. Sun, and A. M. Penuela, “Knowledge- enhanced program repair for data science code,”arXiv preprint arXiv:2502.09771, 2025

    arXiv 2025

Show all 53 references
  1. [10]

    Specrover: Code intent extraction via llms,

    H. Ruan, Y . Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 963–974

    2025

  2. [11]

    How effective are llms for data science coding? a controlled experiment,

    N. Nascimento, E. Guimaraes, S. S. Chintakunta, and S. A. Boomi- nathan, “How effective are llms for data science coding? a controlled experiment,” in2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 2025, pp. 211–222

    2025

  3. [12]

    Improving patch correctness analysis via random testing and large language models,

    F. Molina, J. M. Copia, and A. Gorla, “Improving patch correctness analysis via random testing and large language models,” in2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2024, pp. 317–328

    2024

  4. [13]

    Do current language models support code intelligence for r programming language?

    Z. Zhao and F. Fard, “Do current language models support code intelligence for r programming language?” inACM Transactions on Software Engineering and Methodology, 2024

    2024

  5. [14]

    Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering,

    R. Wang, J. Guo, C. Gao, G. Fan, C. Y . Chong, and X. Xia, “Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering,”Proc. ACM Softw. Eng., vol. 2, no. ISSTA, Jun

  6. [15]

    Patch correctness assessment: A survey,

    Z. Fei, J. Ge, C. Li, T. Wang, Y . Li, H. Zhang, L. Huang, and B. Luo, “Patch correctness assessment: A survey,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1–50, 2025

    2025

  7. [16]

    Patchzero: Zero-shot automatic patch correctness assessment,

    X. Zhou, B. Xu, K. Kim, D. Han, T. Le-Cong, J. He, B. Le, and D. Lo, “Patchzero: Zero-shot automatic patch correctness assessment,”arXiv preprint arXiv:2303.00202, 2023

    2023 arXiv

  8. [17]

    Tag Trends Data Science Libraries,

    “Tag Trends Data Science Libraries,” https://trends.stackoverflow.co/?ta gs=pandas%2Cscipy%2Cscikit-learn%2Cnumpy%2Cmatplotlib%2Cda taframe%2Cpyqt, 2025, [Online; accessed Apr-2026]

    2025

  9. [18]

    What is the most efficient way of counting occurrences in pandas?

    Stack Overflow user, “What is the most efficient way of counting occurrences in pandas?” https://stackoverflow.com/questions/200761 95/what-is-the-most-efficient-way-of-counting-occurrences-in-pandas, 2013, accessed: 2025-07-19. [Online]. Available: https://stackoverflow. com/...

    2013

  10. [19]

    Output logs from AutoCodeRover and ArchCode ,

    “Output logs from AutoCodeRover and ArchCode ,” https://github.c om/reprodgen/reprodgen/tree/main/sota-comparison, 2026, [Online; accessed Apr-2026]

    2026

  11. [20]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. ...

    2024

  12. [21]

    Archcode: Incor- porating software requirements in code generation with large language 11 models,

    H. Han, J. Kim, J. Yoo, Y . Lee, and S.-w. Hwang, “Archcode: Incor- porating software requirements in code generation with large language 11 models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 13...

    2024

  13. [22]

    ReprodGen Prompts,

    “ReprodGen Prompts,” https://github.com/reprodgen/reprodgen/tree/ma in/src/reprodbench/llm/prompts, 2026, [Online; accessed Apr-2026]

    2026

  14. [23]

    Intermediate artifacts,

    “Intermediate artifacts,” https://github.com/reprodgen/reprodgen/tree/ma in/results/figures, 2026, [Online; accessed Apr-2026]

    2026

  15. [24]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1–23, 2025

    2025

  16. [25]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  17. [26]

    On reliability of patch correctness assessment,

    X.-B. D. Le, L. Bao, D. Lo, X. Xia, S. Li, and C. Pasareanu, “On reliability of patch correctness assessment,” in2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 524–535

    2019

  18. [27]

    Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation,

    Z. Zhang, C. Wang, Y . Wang, E. Shi, Y . Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng, “Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation,”Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 481–503, 2025

    2025

  19. [28]

    Ds patch gen query,

    S. Exchange, “Ds patch gen query,” https://data.stackexchange.com /stackoverflow/query/1907457/ds-patch-gen, 2025, [Online; accessed July-2025]

    2025

  20. [29]

    Towards ai-assisted synthesis of verified dafny methods,

    M. R. H. Misu, C. V . Lopes, I. Ma, and J. Noble, “Towards ai-assisted synthesis of verified dafny methods,”Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643763

    2024 doi

  21. [30]

    A comprehensive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” inProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2019...

    2019

  22. [31]

    The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory, in-the-small, and in-the-large,

    S. Biswas, M. Wardat, and H. Rajan, “The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory, in-the-small, and in-the-large,” inProceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York,...

    2022

  23. [32]

    Bigcode models leaderboard,

    BigCode, “Bigcode models leaderboard,” https://huggingface.co/spaces/ bigcode/bigcode-models-leaderboard, 2026, accessed: 2026-04-20

    2026

  24. [33]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  25. [34]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram ´e, M. Rivi `ereet al., “Gemma 3 technical report,”arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  26. [35]

    Llama 3: Open and efficient foundation language models,

    M. AI, “Llama 3: Open and efficient foundation language models,” 2024, accessed: 2025-07-19. [Online]. Available: https://ai.meta.com/llama/

    2024

  27. [36]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,

    Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y . Wu, Y . Li, H. Gao, S. Maet al., “Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,”arXiv preprint arXiv:2406.11931, 2024

    2024 arXiv

  28. [37]

    Phi- 4 technical report,

    M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmannet al., “Phi- 4 technical report,”arXiv preprint arXiv:2412.08905, 2024

    2024 arXiv

  29. [38]

    Evaluating and improving chatgpt for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1703–1726, 2024

    2024

  30. [39]

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models,

    P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V . Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Trameret al., “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,”Advances in Neural Information Processing Systems, vol. ...

    2024

  31. [40]

    CodeBERTScore: Evaluating code generation with pretrained models of code,

    S. Zhou, U. Alon, S. Agarwal, and G. Neubig, “CodeBERTScore: Evaluating code generation with pretrained models of code,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for...

    2023

  32. [41]

    Accurate and efficient refactoring detection in commit history,

    N. Tsantalis, M. Mansouri, L. M. Eshkevari, D. Mazinanian, and D. Dig, “Accurate and efficient refactoring detection in commit history,” inProceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY , USA: Association for Computing Mac...

    2018

  33. [42]

    Binary codes capable of correcting deletions, inser- tions, and reversals,

    V . I. Levenshtein, “Binary codes capable of correcting deletions, inser- tions, and reversals,”Soviet Physics Doklady, vol. 10, no. 8, pp. 707–710, 1966

    1966

  34. [43]

    Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models ,

    Z. Wang, Z. Zhou, D. Song, Y . Huang, S. Chen, L. Ma, and T. Zhang, “ Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models ,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE ...

    2025

  35. [44]

    Can LLMs reason about program semantics? a comprehensive evaluation of LLMs on formal specification inference,

    T. Le-Cong, B. Le, and T. Murray, “Can LLMs reason about program semantics? a comprehensive evaluation of LLMs on formal specification inference,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabe...

    2025

  36. [45]

    Hints help finding and fixing bugs differently in python and text-based program representations,

    R. Rawal, V .-A. P ˘adurean, S. Apel, A. Singla, and M. Toneva, “Hints help finding and fixing bugs differently in python and text-based program representations,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2025, pp. 729– 729

    2025

  37. [46]

    Imitation game: Reproduc- ing deep learning bugs leveraging an intelligent agent,

    M. B. Shah, M. M. Rahman, and F. Khomh, “Imitation game: Reproduc- ing deep learning bugs leveraging an intelligent agent,”arXiv preprint arXiv:2512.14990, 2025

    2025

  38. [47]

    Zs4c: Zero-shot synthesis of compilable code for incomplete code snippets using llms,

    A. Kabir, S. Wang, Y . Tian, T.-H. Chen, M. Asaduzzaman, and W. Zhang, “Zs4c: Zero-shot synthesis of compilable code for incomplete code snippets using llms,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 4, pp. 1–30, 2025

    2025

  39. [48]

    Selfpico: Self-guided partial code execution with llms,

    Z. Xue, Z. Gao, S. Wang, X. Hu, X. Xia, and S. Li, “Selfpico: Self-guided partial code execution with llms,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1389–1401

    2024

  40. [49]

    Evaluating large language models in class-level code generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Evaluating large language models in class-level code generation,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  41. [50]

    Identifying patch correctness in test-based program repair,

    Y . Xiong, X. Liu, M. Zeng, L. Zhang, and G. Huang, “Identifying patch correctness in test-based program repair,” inProceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 789–79...

    2018

  42. [51]

    ReprodGen Repository,

    “ReprodGen Repository,” https://github.com/reprodgen/reprodgen, 2026, [Online; accessed Apr-2026]

    2026

  43. [52]

    ReprodGen Bench,

    “ReprodGen Bench,” https://github.com/reprodgen/reprodgen/tree/main /data, 2026, [Online; accessed Apr-2026]

    2026

  44. [53]

    ReprodGen Leaderboard,

    “ReprodGen Leaderboard,” https://github.com/reprodgen/reprodgen-lea derboard, 2026, [Online; accessed Apr-2026]. 12

    2026

  45. [2025]

    Available: https://doi.org/10.1145/3728963

    [Online]. Available: https://doi.org/10.1145/3728963

    doi

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.